Go Back   Hardware Canucks > HARDWARE CANUCKS COMMUNITY > HardwareCanucks F@H Team

    
Reply
 
LinkBack Thread Tools Display Modes
  #11 (permalink)  
Old December 9, 2010, 12:12 PM
Sagath's Avatar
Hall Of Fame
F@H
 
Join Date: Feb 2009
Location: Petawawa, ON
Posts: 2,454

My System Specs

Default

Hey 10e, you live in Markham eh? Thats only about 25-30 min from Chris & me.

I think we should meet up. It would be cool to meet you and see this in person!
__________________
Quote:
Originally Posted by Lpfan4ever View Post
AKG shot the hamsters, and then Perineum drove his Mustang with summer tires into the server.
My Disclaimer to any advice or comment I make;
Quote:
Originally Posted by CroSsFiRe2009 View Post
I'm a self certified whizbang repair technician with 20 years of professional bullshit so I don't know what I'm talking about
Reply With Quote
  #12 (permalink)  
Old December 9, 2010, 01:09 PM
10e's Avatar
10e 10e is offline
Allstar
F@H
 
Join Date: Jan 2008
Location: Markham, Ontario
Posts: 514

My System Specs

Default

Quote:
Originally Posted by Sagath View Post
Hey 10e, you live in Markham eh? Thats only about 25-30 min from Chris & me.

I think we should meet up. It would be cool to meet you and see this in person!
I was in your neck of the woods yesterday at Simcoe and Rosslands where there is "that hospital" :)

That's always a possibiliteeeeeeeeeeee!!!!!
__________________

Xeven: How about 10^8.450980400142567e-001 -as a possible replacement for "10e"

http://www.heatware.com/eval.php?id=71732
Reply With Quote
  #13 (permalink)  
Old December 9, 2010, 02:56 PM
Hall Of Fame
 
Join Date: Oct 2007
Posts: 1,561
Default

why were you at that hospital? nothing bad I hope? O.O

(I go to UOIT / Durham)
Reply With Quote
  #14 (permalink)  
Old December 9, 2010, 03:15 PM
Trial Member
 
Join Date: Dec 2010
Location: Small Town, Ontario
Posts: 1
Default

Quote:
Originally Posted by 10e View Post
A while back around July or August I had bought the following stuffs:

1) 2 x AMD Opteron 6168 1.9Ghz, 12-core Magny-Cours CPUs
2) 1 Asus KGPE-D16 Dual Socket G34 motherboard
3) 8 1GB Kingston KVR1333DE9S (ECC) DDR 1333 DIMMs
4) Lian Li PC-P80 case

I bought it because I was curious about folding performance of the Magny-Cours chips and how it compared with the current Xeon line-up of chips.

Outside of Bit-Tech.Net which compared dual Xeon x5650s and dual Opteron 6174, there wasn't really much out there, and this comparison was done using BigAdv A2 units, which were de-commissioned by this time.

Anyway, I found the performance was very close to a well-overclocked Core I7 980x running at 4.1 Ghz or higher, except on BigAdv 2684 units, where the dual Opteron could out-fold even a 4.4Ghz Core I7 980x.

Generally I was getting between 55 - 60K PPD on 2685, 2686, and 2692 units and around 42K PPD on 2684 BigAdv units. Not bad, but I wanted more!

Power usage at the time was around 303 - 309 watts, which really was in line with a Core I7 980x system running a modern day graphics adapter. The Asus KGPE-D16 board comes with a ultra-low end graphics adapter, but it also comes with an IPMI/KVM adapter allowing console access without keyboard and mouse, which works great. It even allows BIOS access and the ability to force power off/on the server. Very nice.

In the meantime I also decided to get an SR-2 with twin Xeon x5650s. I then remembered why I hated overclocking Core I7 based stuff and it was even worse with dual Xeon chips. I got the SR-2 to 3.9Ghz and it went between 88K and 147K PPD on BigAdv units. Impressive no doubt, but going 4.0Ghz and higher actually resulted in a loss of folding speed so I left it at that.

A while after that, after corresponding with a fellow KGPE-D16 and Opteron 6168 duallie owning NCIX folder who was looking to get rid of some Opteron 6168 "Black Edition" chips, we worked out a deal for me to get this second set of Opteron 6168s. He wanted to go SR-2. I had already done so, and I had been interested in those chips since seeing them listed on EBay a few months earlier.

Fortuitous luck one might say.

After playing around with multiplier overclocking I settled on a 2.6Ghz overclock and got between 60K PPD and 92K PPD with BigAdv units.

Not bad either, and I could go even higher to 2.8, maybe 2.9Ghz, but at that point I changed my mind.

QUAD box!!!!!!!!!!!!!!!!!!!

I already had my original Opteron 6168 retail chips, safely tucked away back in their boxes, and I now had two more 6168s. So why not? The big cost had already been absorbed and either I could re-sell my original 6168s at a loss, or keep them and see what I could do.

If worse came to worst I would sell them and keep the "Black Editions".

So I needed to get 8 more 1GB DIMMS, which luckily enough had dropped in price like a rock to nearly half of what I had bought them at just a few months earlier. I originally bought them for about $40.00 per DIMM back in August, but now in November they were $23 - $24.00 a DIMM!

The Opterons are unique because to see full performance you need quad-channel DDR3 RAM. Each chip "package" holds two 6-core CPUs, each with a dual-channel RAM IMC, so four channels in total.

I also needed a motherboard (of course), so I settled on this:

Supermicro H8QGI-F quad opteron G34 motherboard with 32 DIMM slots:

Super Micro Computer, Inc. - Aplus Products | Motherboards | H8QGi-F

This monstrous SWTX board is wider and nearly as deep as the SR-2 which is one of the biggest consumer motherboards (as many of us know) and hard to find a case for. It's 13" deep but over 16" wide!

So I special ordered it in November and last Friday the big monster board came in. Huge, heavy and as rigid as they come, but a thing of beauty.

Here's a shot with the CPUs installed:





So I hooked it up to the Enermax Revolution 85+ 1050W, stuck in the CPUs (as pictured) and it was off to the races. Right now I'd figured I would run 'er naked and figure out the case situation once the excitement wore off and my head was screwed back on

I had bought some SuperMicro SNK-P0043P and mounted a couple of CM BladeMaster 92mm 2500 RPM PWM fans on them and things were kept cool, and I bought two more TT ITBU CL-P0314 heat sinks which were Socket F 1207 HSF units, but using the 4.1" spacing I required to screw them into the G34 compatible screwholes (also pictured).

I then decided to stick the 16 1 GB DIMMs in and hook up all THREE EPS 12V power connectors. I needed to get a dual molex to EPS 12V connector as the board has THREE whereas the Enermax 1050W Revolution85+ has two stock.

The result!




I recently ordered two Noctua NH-U9DO A3 G34 compatible CPU coolers from Team 37412's sponsor to change out the two smaller coolers. Initially they were pretty buzzy but with the AMD fans on them they've quieted down.

So far at 100% folding load (6900 BigAdv) units the hottest CPUs are running in the low 53 - 54 celsius range and the larger SuperMicro SNK-P0043P equipped CPUs are only at 39 celsius.

I then installed Windows 2008 R2 x64 from my Technet sub using the IPMI by mapping the Windows ISO from my main PC as a "virtual DVD". This took about 40 minutes from start to finish over a gigabit connection, and I then proceeded to install and run Folding@home with the BigAdv switch.


Power Usage:

540 - 545 watts for the whole shebang at 100% load. This is around the same as my SR-2 @ 3.9Ghz with dual x5650s and lowly ATI Radeon 3670 GPU (no GPU folding).

Folding Performance:

With SMP A3 unit 6024 I was able to get a time-per-frame (TPF) of between 57 and 58 seconds. This results in around 100K PPD.

With BigAdv unit P6900 this box is ripping through it at a TPF of 10:51 and a PPD of 172K PPD!

Both times are almost exactly HALF of what I was getting with the dual, stock-clocked 1.9Ghz Opterons so if the question is asked, YES, folding@home SMP units seem to scale almost linearly in terms of doubling cores from 24 to 48.

If this trend holds up, I should be getting 2684 units to fold in under a day. The twin 6168s with 24 1.9Ghz cores can do a TPF of 27:50 (average) with 2684s and if I halve that, I will get a TPF of around 14 minutes. A TPF of 14:24 (average) on a 2684 will get one done in a 24 hour period.

The Opterons seem to really outstrip the Xeons/Core I7s at 2684s. Other units seem to be close. I have a benchmark that can simulate a 2686 which I will try later and post more numbers for that.

Even my SR-2 at 3.9Ghz and 24 threads can only do a TPF of 16:50 on 2684 units, meaning an average of 28 hours per unit.

If this baby can pull 14 minute TPFs on 2684, it will result in a PPD of over 115K !

So why did I do this?

1) I was really curious how folding@home scales.
2) I wanted a multi-CPU box "for the ages"! I plan to keep this box for a number of years and I really wanted it to kick some rear-end.
3) General hardware curiosity. I was able to get those two extra "Black Edition" Opteron 6168s for such a great price that I wanted to put them to good use.

As the build progresses with a case, and as more folding@home unit types get crunched, I will update this thread.

Feel free to ask any question you have! I will check back regularly on a daily basis to answer anything I can.

Thanks for reading! Stay Hard Canucks!

PS. This box, plus my SR-2 and associated other components will allow us to have close to 500,000 PPD extra for next year's chimp challenge.

Go Canada

I don't personally own a system like that. My work has a 48-core, built with AMD Opteron 6174s, same board though. And a Cooler Master Stacker 810 case which is now being replaced (already) with a Supermicro SC748TQ-R1400B because at load, I'm tripping the overcurrent protection on the 1000 W Corsair power supply. So, we're going to 1.4 kW instead. New for the company and for x86/x64-based systems. But probably not new if they've ever ran any of the IBM POWER UNIX workstations.

We also have 32x 4 GB Kingston 4 GB DDR3-1333 (128 GB total) of RAM and an OCZ Agility 2 240 GB SATA2 SSD.

(It slated to run FEAs.) I've currently got Windows HPC Server 2008 x64 on it, minus the HPC add-ons (haven't installed Active Directory Domain Services on it yet/don't know how, but I'm pretty sure that I can probably google it).

The original tests that I ran also with P6900 WUs, but with only 4 DIMMS (16 GB), I was only getting 78913 PPD. That should change though because we're going to be pulling half the RAM out so that hopefully it'll keep the overcurrent protection in check, and drop that down to just 64 GB.

Interestingly enough though, according to IntelBurnTest, my Intel 980X CAD/FEA workstation gets I think like 75 GFLOPS, while the server only got 66 GFLOPS (at least in the original tests with just 16 GB of RAM).

I'll have to do more testing tomorrow once the system is back up and running.

Quote:
Originally Posted by Sagath View Post
Hey 10e, you live in Markham eh? Thats only about 25-30 min from Chris & me.

I think we should meet up. It would be cool to meet you and see this in person!
@10e: Whereabouts in Markham are you?

I'm going to be in Toronto next weekend (Dec 18) so maybe you and I can meet up too?
Reply With Quote
  #15 (permalink)  
Old December 9, 2010, 03:30 PM
Rison's Avatar
Allstar
F@H
 
Join Date: Sep 2009
Location: Truro, NS
Posts: 936

My System Specs

Default

Quote:
Originally Posted by matsta31 View Post
My 5639 Build :

1 050 $ Shipped for two CPU
600 $ SR-2
200 $ = Ram 12 Gigs ( I should have gone for 6 gigs in retrospect)
60 $ WDD 640G black (I already bought it for prior Build - backup)
I added some GTX480's but this is for my 3 LCD gaming, so cost should not matter here.
Lets put a 50 $ value on some basic GPU
250 $ PSU, I got the AX 1200W, but a lower graded one Like any 850 W would have been enough.

total : 2 200 $ Taxe in, shipping in.
I don't know how you got an SR-2 for $600 tax in and shipped, unless it was used.

Anywho, to buy new vs new right now - you can't buy the 5639's anymore.. so the 5650's are in the $1050 range +tax and shipping. Overall buying all new (which i'm looking at) the opteron would be better in the long run, with higher PPD per watt (and we all know.. there is huge savings there over time)

The SR2 setup new would be:
$610 m/b
$2100 2x 5650s
$200 ram (6x2Gb)
= $2910 +tx + ship.
we'll say 145k PPD = 49.8 points per dollar.

G34 setup:
$850 m/b
$3000 4x 6168's (~$750/e)
$384 ram (4x 16 @ $24/e)
= $4234 +tx +ship
we'll say 170k PPD = 40.15 points per dollar.

That's kind of retail, round figures. But from a power standpoint, with each being in the ~500-550W range.. over time, the quad G34 would come out ahead when you factor in PPD/watt.
__________________

Quote:
I'll learn to manage my anger, when people learn to manage their stupid.
Reply With Quote
  #16 (permalink)  
Old December 9, 2010, 03:48 PM
G777's Avatar
Allstar
 
Join Date: Mar 2010
Location: Coquitlam,BC
Posts: 539

My System Specs

Default

Rison: just fyi: Sr-2 is $584.99 from Newegg Newegg.ca - EVGA Classified SR-2 270-WS-W555-A2 LGA 1366 Intel 5520 SATA 6Gb/s USB 3.0 HPTX Intel Motherboard
__________________
_______________________________________________

HWC Overclocking Team, breaking hardware for fun!

Team Thread
Reply With Quote
  #17 (permalink)  
Old December 9, 2010, 03:50 PM
Xeven's Avatar
MVP
F@H
 
Join Date: Aug 2010
Location: Vancouver, BC
Posts: 483
Default

Quote:
Originally Posted by Rison View Post
The SR2 setup new would be:
$610 m/b
$2100 2x 5650s
$200 ram (6x2Gb)
= $2910 +tx + ship.
we'll say 145k PPD = 49.8 points per dollar.
The SR-2 is $584.99 from Newegg before tax and shipping. You can also get a pair of E5645's for $1100 shipped, they are better than L5639s because of the higher multiplier. So by your computation it's actually ~$1885 without chassis, hdd/ssd, hsf/cooling and other associated parts. Really, if you're smart about the CPUs it shouldn't cost that much more than a brand new 980x build.

EDIT: Didn't see G777's reply before I posted.
__________________

PSXeven. Batards of the World Unite!
The gap between happiness and utter misery is only but a few inches

Last edited by Xeven; December 9, 2010 at 03:56 PM.
Reply With Quote
  #18 (permalink)  
Old December 9, 2010, 03:54 PM
Rison's Avatar
Allstar
F@H
 
Join Date: Sep 2009
Location: Truro, NS
Posts: 936

My System Specs

Default

Quote:
Originally Posted by Xeven View Post
The SR-2 is $584.99 from Newegg before tax and shipping. You can get also get a pair of E5645s for $1100 shipped, they are better than L5639s because of the higher multiplier. So by your computation it's actually $1885 without chassis, hdd/ssd, hsf/cooling and other associated parts.
And would be less PPD. I don't know what the 5645's would put out compared to the 5650's.. I was looking at a pair of 5650's myself.
$584.99 + tax is $672 (15%HST here) + shipping.

edit: and where can you get 5645s for $1100 shipped?
__________________

Quote:
I'll learn to manage my anger, when people learn to manage their stupid.
Reply With Quote
  #19 (permalink)  
Old December 9, 2010, 04:00 PM
Xeven's Avatar
MVP
F@H
 
Join Date: Aug 2010
Location: Vancouver, BC
Posts: 483
Default

They should be good to at least 3.85Ghz (disregarding the SR-2s craziness if you're unlucky). I guestimate 145K-147K for a P6900/P2686 which is decent enough I guess. Less PPD of course, but the initial cash outlay is also < 50% compared to the quad Opterons. This is probably helpful to most people.

I'm not undervaluing 10e's system of course. But I think his is more of an exercise in curiosity and circumstance. The more "practical" system IMO is the SR-2. But then let's face it, neither of these systems will make your wallet any happier.
__________________

PSXeven. Batards of the World Unite!
The gap between happiness and utter misery is only but a few inches

Last edited by Xeven; December 9, 2010 at 04:08 PM.
Reply With Quote
  #20 (permalink)  
Old December 9, 2010, 04:08 PM
matsta31's Avatar
Hall Of Fame
F@H
 
Join Date: May 2008
Location: Montreal
Posts: 2,204
Default

I looked at my credit card bill : 634 $ for the sr2, shipped. Sorry for the 34 $. It is true I dont get the 15 % in Qc, hurray one good point for us at last ! Also, you will find Xeon for cheaper, but on ebay and used. And 6 GB is enough for folding. so you can save another extra 100 $.
__________________

Sold hardware again.....
Currently using WHS hardware....
Reply With Quote
Reply


Tags
48-cores , 6168 , h8qgi-f , opteron

Thread Tools
Display Modes

Similar Threads
Thread Thread Starter Forum Replies Last Post
2 to 4 to 3 cores Wurmer CPU's and Motherboards 2 April 15, 2010 06:15 PM
F@H only uses 2 out of 16 cores whited HardwareCanucks F@H Team 20 March 12, 2010 02:54 PM
AMD "Magny Cours" Processors Starting Production CMetaphor Press Releases & Tech News 18 February 23, 2010 05:29 PM
AMD Demos 48-core ''Magny-Cours'' System, Details Architecture Unknownm Press Releases & Tech News 13 August 27, 2009 05:19 AM
8 Intel cores faster than 16 AMD cores? Babrbarossa Rumor Mill 0 May 24, 2007 06:54 AM