Go Back   Hardware Canucks > HARDWARE CANUCKS COMMUNITY > HardwareCanucks F@H Team

    
Reply
 
LinkBack Thread Tools Display Modes
  #1 (permalink)  
Old December 9, 2010, 10:21 AM
10e's Avatar
10e 10e is offline
Allstar
F@H
 
Join Date: Jan 2008
Location: Markham, Ontario
Posts: 571

My System Specs

Default 48 cores of Canadian Magny-Cours Motherfolding

A while back around July or August I had bought the following stuffs:

1) 2 x AMD Opteron 6168 1.9Ghz, 12-core Magny-Cours CPUs
2) 1 Asus KGPE-D16 Dual Socket G34 motherboard
3) 8 1GB Kingston KVR1333DE9S (ECC) DDR 1333 DIMMs
4) Lian Li PC-P80 case

I bought it because I was curious about folding performance of the Magny-Cours chips and how it compared with the current Xeon line-up of chips.

Outside of Bit-Tech.Net which compared dual Xeon x5650s and dual Opteron 6174, there wasn't really much out there, and this comparison was done using BigAdv A2 units, which were de-commissioned by this time.

Anyway, I found the performance was very close to a well-overclocked Core I7 980x running at 4.1 Ghz or higher, except on BigAdv 2684 units, where the dual Opteron could out-fold even a 4.4Ghz Core I7 980x.

Generally I was getting between 55 - 60K PPD on 2685, 2686, and 2692 units and around 42K PPD on 2684 BigAdv units. Not bad, but I wanted more!

Power usage at the time was around 303 - 309 watts, which really was in line with a Core I7 980x system running a modern day graphics adapter. The Asus KGPE-D16 board comes with a ultra-low end graphics adapter, but it also comes with an IPMI/KVM adapter allowing console access without keyboard and mouse, which works great. It even allows BIOS access and the ability to force power off/on the server. Very nice.

In the meantime I also decided to get an SR-2 with twin Xeon x5650s. I then remembered why I hated overclocking Core I7 based stuff and it was even worse with dual Xeon chips. I got the SR-2 to 3.9Ghz and it went between 88K and 147K PPD on BigAdv units. Impressive no doubt, but going 4.0Ghz and higher actually resulted in a loss of folding speed so I left it at that.

A while after that, after corresponding with a fellow KGPE-D16 and Opteron 6168 duallie owning NCIX folder who was looking to get rid of some Opteron 6168 "Black Edition" chips, we worked out a deal for me to get this second set of Opteron 6168s. He wanted to go SR-2. I had already done so, and I had been interested in those chips since seeing them listed on EBay a few months earlier.

Fortuitous luck one might say.

After playing around with multiplier overclocking I settled on a 2.6Ghz overclock and got between 60K PPD and 92K PPD with BigAdv units.

Not bad either, and I could go even higher to 2.8, maybe 2.9Ghz, but at that point I changed my mind.

QUAD box!!!!!!!!!!!!!!!!!!!

I already had my original Opteron 6168 retail chips, safely tucked away back in their boxes, and I now had two more 6168s. So why not? The big cost had already been absorbed and either I could re-sell my original 6168s at a loss, or keep them and see what I could do.

If worse came to worst I would sell them and keep the "Black Editions".

So I needed to get 8 more 1GB DIMMS, which luckily enough had dropped in price like a rock to nearly half of what I had bought them at just a few months earlier. I originally bought them for about $40.00 per DIMM back in August, but now in November they were $23 - $24.00 a DIMM!

The Opterons are unique because to see full performance you need quad-channel DDR3 RAM. Each chip "package" holds two 6-core CPUs, each with a dual-channel RAM IMC, so four channels in total.

I also needed a motherboard (of course), so I settled on this:

Supermicro H8QGI-F quad opteron G34 motherboard with 32 DIMM slots:

Super Micro Computer, Inc. - Aplus Products | Motherboards | H8QGi-F

This monstrous SWTX board is wider and nearly as deep as the SR-2 which is one of the biggest consumer motherboards (as many of us know) and hard to find a case for. It's 13" deep but over 16" wide!

So I special ordered it in November and last Friday the big monster board came in. Huge, heavy and as rigid as they come, but a thing of beauty.

Here's a shot with the CPUs installed:





So I hooked it up to the Enermax Revolution 85+ 1050W, stuck in the CPUs (as pictured) and it was off to the races. Right now I'd figured I would run 'er naked and figure out the case situation once the excitement wore off and my head was screwed back on

I had bought some SuperMicro SNK-P0043P and mounted a couple of CM BladeMaster 92mm 2500 RPM PWM fans on them and things were kept cool, and I bought two more TT ITBU CL-P0314 heat sinks which were Socket F 1207 HSF units, but using the 4.1" spacing I required to screw them into the G34 compatible screwholes (also pictured).

I then decided to stick the 16 1 GB DIMMs in and hook up all THREE EPS 12V power connectors. I needed to get a dual molex to EPS 12V connector as the board has THREE whereas the Enermax 1050W Revolution85+ has two stock.

The result!




I recently ordered two Noctua NH-U9DO A3 G34 compatible CPU coolers from Team 37412's sponsor to change out the two smaller coolers. Initially they were pretty buzzy but with the AMD fans on them they've quieted down.

So far at 100% folding load (6900 BigAdv) units the hottest CPUs are running in the low 53 - 54 celsius range and the larger SuperMicro SNK-P0043P equipped CPUs are only at 39 celsius.

I then installed Windows 2008 R2 x64 from my Technet sub using the IPMI by mapping the Windows ISO from my main PC as a "virtual DVD". This took about 40 minutes from start to finish over a gigabit connection, and I then proceeded to install and run Folding@home with the BigAdv switch.


Power Usage:

540 - 545 watts for the whole shebang at 100% load. This is around the same as my SR-2 @ 3.9Ghz with dual x5650s and lowly ATI Radeon 3670 GPU (no GPU folding).

Folding Performance:

With SMP A3 unit 6024 I was able to get a time-per-frame (TPF) of between 57 and 58 seconds. This results in around 100K PPD.

With BigAdv unit P6900 this box is ripping through it at a TPF of 10:51 and a PPD of 172K PPD!

Both times are almost exactly HALF of what I was getting with the dual, stock-clocked 1.9Ghz Opterons so if the question is asked, YES, folding@home SMP units seem to scale almost linearly in terms of doubling cores from 24 to 48.

If this trend holds up, I should be getting 2684 units to fold in under a day. The twin 6168s with 24 1.9Ghz cores can do a TPF of 27:50 (average) with 2684s and if I halve that, I will get a TPF of around 14 minutes. A TPF of 14:24 (average) on a 2684 will get one done in a 24 hour period.

The Opterons seem to really outstrip the Xeons/Core I7s at 2684s. Other units seem to be close. I have a benchmark that can simulate a 2686 which I will try later and post more numbers for that.

Even my SR-2 at 3.9Ghz and 24 threads can only do a TPF of 16:50 on 2684 units, meaning an average of 28 hours per unit.

If this baby can pull 14 minute TPFs on 2684, it will result in a PPD of over 115K !

So why did I do this?

1) I was really curious how folding@home scales.
2) I wanted a multi-CPU box "for the ages"! I plan to keep this box for a number of years and I really wanted it to kick some rear-end.
3) General hardware curiosity. I was able to get those two extra "Black Edition" Opteron 6168s for such a great price that I wanted to put them to good use.

As the build progresses with a case, and as more folding@home unit types get crunched, I will update this thread.

Feel free to ask any question you have! I will check back regularly on a daily basis to answer anything I can.

Thanks for reading! Stay Hard Canucks!

PS. This box, plus my SR-2 and associated other components will allow us to have close to 500,000 PPD extra for next year's chimp challenge.

Go Canada


__________________

Xeven: How about 10^8.450980400142567e-001 -as a possible replacement for "10e"

http://www.heatware.com/eval.php?id=71732
Reply With Quote
  #2 (permalink)  
Old December 9, 2010, 10:30 AM
matsta31's Avatar
Hall Of Fame
F@H
 
Join Date: May 2008
Location: Montreal
Posts: 2,204
Default

Thanks for posting here. This is amazing... To bad theses opterons are so expensives vs the xeon 5639 or othe Intel of the same type.
__________________

Sold hardware again.....
Currently using WHS hardware....
Reply With Quote
  #3 (permalink)  
Old December 9, 2010, 10:35 AM
lowfat's Avatar
Moderator
 
Join Date: Feb 2007
Location: Grande Prairie, AB
Posts: 8,105

My System Specs

Default

How was the cost compared to an SR-2 w/ X5650's?
__________________
Mael
Big Lian Li
Forever Alone
Reply With Quote
  #4 (permalink)  
Old December 9, 2010, 10:48 AM
10e's Avatar
10e 10e is offline
Allstar
F@H
 
Join Date: Jan 2008
Location: Markham, Ontario
Posts: 571

My System Specs

Default

Quote:
Originally Posted by matsta31 View Post
Thanks for posting here. This is amazing... To bad theses opterons are so expensives vs the xeon 5639 or othe Intel of the same type.
New vs. New the Opterons are nearly $300.00 cheaper each vs. Xeon x5650s, and the second set of "Black Edition" processors cost me about the same as the 5639 units you guys got from Ebay.

I got my SR-2 from Newegg earlier this year for $599.99 plus $20.00 shipping and the Xeons cost me about $900.00 each before taxes. I had 6 2GB DIMMs already, and the Lian Li PC-P80 case was transferred from the original KGPE-D16 dual 6168 setup to the SR-2.

Quote:
Originally Posted by lowfat View Post
How was the cost compared to an SR-2 w/ X5650's?
2 x Opteron 6168 Retail CPUs - $750.00 each
2 x Opteron 6168 "Black Edition" overclock-able CPUs - $500.00 each
8 x 1 GB Dimms - $40.00 each
8 x 1 GB Dimms - $23.00 each (second set)
1 x SuperMicro H8QGI-F quad socket motherboard - $850.00

Enermax Revo 85+ 1050W was $250.00 when I bought it new
WD5000AAKS - 500GB drive was less than $100.00

No need for a DVD-RW because I can map ISOs through IPMI (Integrated platform management interface) over the network.

Total: $3854 + taxes

Value? Priceless. Sorry had to add that. Like I said earlier, this was all bought over time to soften the blow on my wallet.

Needless to say, there will be NO more folding (or otherwise) upgrades for an extended period of time regardless of Sandy Bridge. Maybe BullDozer though
__________________

Xeven: How about 10^8.450980400142567e-001 -as a possible replacement for "10e"

http://www.heatware.com/eval.php?id=71732
Reply With Quote
  #5 (permalink)  
Old December 9, 2010, 10:53 AM
Alpine's Avatar
Allstar
 
Join Date: Dec 2008
Location: Boucherville / South shore of MTL
Posts: 870

My System Specs

Default




wish that i all this $$$ to build a monster like that
__________________
• ► ›Main system
• ► › |I7 3930K|Asus P9X79 WS|Evga GTX 690 Signature|Samsung 256G Pro|Corsair H100|Dominator Platinium 16GB 2133Mhz|
Corsair AX1200|Corsair 540 AIR|Corsair SP2500|Corsair Vengeance 2000|
• ► ›Server system
• ► › |I7 3820|Asus P9X79|12G G.Skill 1600Mhz|Noctua NH-D14|Corsair Force 3 120GB|LSI megaRAID 9261 8|RAID5 5X 3TB|Corsair AX650|Fractal Design Arc Midi|

• ► ›Twitch
Reply With Quote
  #6 (permalink)  
Old December 9, 2010, 10:56 AM
matsta31's Avatar
Hall Of Fame
F@H
 
Join Date: May 2008
Location: Montreal
Posts: 2,204
Default

My 5639 Build :

1 050 $ Shipped for two CPU
600 $ SR-2
200 $ = Ram 12 Gigs ( I should have gone for 6 gigs in retrospect)
60 $ WDD 640G black (I already bought it for prior Build - backup)
I added some GTX480's but this is for my 3 LCD gaming, so cost should not matter here.
Lets put a 50 $ value on some basic GPU
250 $ PSU, I got the AX 1200W, but a lower graded one Like any 850 W would have been enough.

total : 2 200 $ Taxe in, shipping in.

Your quad socket cool factor is no match, but I think the Sr-2 is better for now, because of overclocking. If the opteron do go down in price, then the equation differ. you also got a great deal on the ''black edition'' Opteron

I did also promise myself no more upgrade before LGA2011 and/or Buldozer. No more GPU buys before the next generation. All this should take me to Q3 2011. Hope I can supress the upgrade bug till then. Anyway, I dont see any single CPU solution beeting a Sr-2 before that time. (I hope !)
__________________

Sold hardware again.....
Currently using WHS hardware....
Reply With Quote
  #7 (permalink)  
Old December 9, 2010, 11:08 AM
Xeven's Avatar
MVP
F@H
 
Join Date: Aug 2010
Location: Vancouver, BC
Posts: 491
Default

172K PPD. Holy crap!



I want one, but I only have only have 40 bucks left in my wallet.
I'll get back with you after I sell one of my kidneys -.-

I'd say good luck, but you already have it running well and good so great job!
__________________

PSXeven. Batards of the World Unite!
The gap between happiness and utter misery is only but a few inches
Reply With Quote
  #8 (permalink)  
Old December 9, 2010, 11:23 AM
Top Prospect
F@H
 
Join Date: Feb 2010
Location: Waterloo, Ontario
Posts: 133

My System Specs

Default

Looks like I need more GTX 460s. I would need 15 to match that ppd and it would still cost $3k to build that farm.
Reply With Quote
  #9 (permalink)  
Old December 9, 2010, 11:39 AM
lowfat's Avatar
Moderator
 
Join Date: Feb 2007
Location: Grande Prairie, AB
Posts: 8,105

My System Specs

Default

Quote:
Originally Posted by Zink View Post
Looks like I need more GTX 460s. I would need 15 to match that ppd and it would still cost $3k to build that farm.
And you would no longer need any heat in the house.
__________________
Mael
Big Lian Li
Forever Alone
Reply With Quote
  #10 (permalink)  
Old December 9, 2010, 01:07 PM
Banned
F@H
 
Join Date: Aug 2007
Location: mtl
Posts: 12,694
Default

epic, epic thread title!!!!!



motherfolding hahahahha


go dude!!!! always believed in multi cpu amd setups.
Reply With Quote
Reply


Tags
48-cores , 6168 , h8qgi-f , opteron

Thread Tools
Display Modes

Similar Threads
Thread Thread Starter Forum Replies Last Post
2 to 4 to 3 cores Wurmer CPU's and Motherboards 2 April 15, 2010 07:15 PM
F@H only uses 2 out of 16 cores whited HardwareCanucks F@H Team 20 March 12, 2010 03:54 PM
AMD "Magny Cours" Processors Starting Production CMetaphor Press Releases & Tech News 18 February 23, 2010 06:29 PM
AMD Demos 48-core ''Magny-Cours'' System, Details Architecture Unknownm Press Releases & Tech News 13 August 27, 2009 06:19 AM
8 Intel cores faster than 16 AMD cores? Babrbarossa Rumor Mill 0 May 24, 2007 07:54 AM