Go Back   Hardware Canucks > HARDWARE > Video Cards

    
Reply
 
LinkBack Thread Tools Display Modes
  #11 (permalink)  
Old June 16, 2010, 04:11 AM
sswilson's Avatar
Moderator
F@H
 
Join Date: Dec 2006
Location: Moncton NB
Posts: 14,227

My System Specs

Default

Great job SKYMTL!!!! I can only imagine how much work went into putting this together.

Personally what I'm taking away from it is that there is always going to be a need for a subjective opinion (doesn't have to be from reviewers, could be from trusted forum members)... no single test (or groups of tests) is ever going to be able to tell the whole story.

As to the companion article on our current benchmark files goes.... three thumbs up!!!! Can't be much more transparent than allowing individuals to dl the files and run them themselves. The only question I'd ask is if there is going to be a sticky on the forums with the latest version(s).
__________________
EVGA X58 3X SLI / i7 980X / 3X 4G Mushkin Blackline Frostbite / XFX Pro 1000W / EVGA GTX 680 SC+ 2GB / Intel DC S3700 200G / WD 6401AALS / TT Lvl 10 GT
Swiftech MCP655 WC Pump / EK XT 240 Rad / 2X Scythe Ultra Kaze / EK Supreme HF / Primochill Myriad Dual Bay Res / Dell UltraSharp U2412M

Gigabyte C1007UN-D / 2X4G Gskill PC3-10700 / Pico PSU / 2.5" 750G Toshiba HDD / Mini-Box M350 / 1X 19" BenQ / 1X 17" HP
Reply With Quote
  #12 (permalink)  
Old June 16, 2010, 05:12 AM
SKYMTL's Avatar
HardwareCanuck Review Editor
 
Join Date: Feb 2007
Location: Montreal
Posts: 11,404
Default

Quote:
Originally Posted by ET3D View Post
Nice article. I also registered to the forums just because of it.

I found the frame rate graphs particularly enlightening. They show how meaningless average / min / max FPS can be. The Just Cause 2 one in particular shows how frame rate can vary significantly, with NVIDIA and AMD trading places, and there's a specific stretch where the AMD card just suffers. Though the AMD card gets a higher maximum, it's quite meaningless as it's just a blip.

This means that even this length of time doesn't tell the entire story. Is the dip for AMD representative of most gameplay, or is the (larger, in this sample) section where the cards show reasonable parity more representative? And if a card dips under 30 fps for one second out of 2 minutes, is that really significant for gameplay? Because it's clearly impossible to tell how often these dips happen just from the min/avg figures.

It's be interesting to consider whether different statistics could give a better feeling to how the cards handle the games.
I agree. Like I said, there is no perfect yet semi time-conscious way to benchmark a game unfortunately. Any graphs that are put up only show a small slice of the whole story but I would rather a 2-4 minute slice rather than a 15-30 second sample.
__________________
Reply With Quote
  #13 (permalink)  
Old June 16, 2010, 06:49 AM
gingerbee's Avatar
Hall Of Fame
F@H
 
Join Date: Jan 2009
Location: Orillia, Ontario
Posts: 6,824

My System Specs

Default

this is why we come here thanks HWC
__________________
Fav quote "One should strive to achieve; not sit in bitter regret." Ronin Harris
Reply With Quote
  #14 (permalink)  
Old June 16, 2010, 07:04 AM
jcmaz's Avatar
Hall Of Fame
F@H
 
Join Date: Feb 2009
Location: Vancouver, BC
Posts: 2,453

My System Specs

Default

You should send this article to the "2 dozen" or so site and ask for feedback :)
__________________
“Wise men speak because they have something to say; Fools because they have to say something.”

-Plato
Reply With Quote
  #15 (permalink)  
Old June 16, 2010, 07:59 AM
FreeKnight's Avatar
MVP
F@H
 
Join Date: Jul 2009
Location: Edmonton, AB
Posts: 488

My System Specs

Default

Great article Sky, I'm with gingerbee in that it's the extra mile that you guys go that keeps me coming to HWC first for reviews and information. When I'm looking for a specific product I'll read every review I can find on it but 9/10 it's your, AkG's, etc articles that I'm using as the best guides for the info I want.

Keep up the good work
Reply With Quote
  #16 (permalink)  
Old June 16, 2010, 09:27 AM
rjbarker's Avatar
Hall Of Fame
 
Join Date: Feb 2008
Location: Courtenay, B.C
Posts: 5,806

My System Specs

Default

Nice Sky.....personally when I make a GPU Upgrade I run my new Cards througha small "Series" of Benches to check improved performance over old and can relate to the time-consuming part of it......always appreciated ;)

Quote:
It's a "noble" cause to try to clean up everybody's mess, start from scratch and give us the right information.

I find that, opposed to a few years back, there is now too many sites doing reviews. Now, you can find reviews that tell you "what you want to hear" basically. You cannot really make your mind by reading a lot of reviews, unless you go crazy and buy the most expensive piece of kit.

It reinforces the "brand loyalty", because nobody will risk to buy anything else but the "XFX or EVGA FLAGSHIPS, Corsair HX/RAM" or insert your "safe buy". Only the best gear, reaching max performance, max reliability and max overclockability keeps some sort of universal support, the rest of the crowd is just a grey zone...it's getting to the point where more and more systems look the same.

One site will tell you that one piece of hardware is the next big thing after sliced bread, and the other site talks about major flaws in design, performance or reliability of such hardware.

Same thing for Graphics card's performance...too many sites, to many different (and opposite) conclusions.

A real mess...really.

Thanks for going above and beyond to feed us the right information.
^^^ Thats a tad blown out of proportion........as soon as the nVidia Fermi cards were released I personally found all the available Reviews fairly accurate as to performance of the latest n greatest GPU offerings, obviously HWC was the best ;)...for Performance it still stands

1. 480
2. 5870
3. 470
4. 5850

^^ Pretty straight forward really.......throw price:performance ratio into the mix and its up to the purchaser to make a decision based upon their own expectations and budget obviously.....
Reply With Quote
  #17 (permalink)  
Old June 16, 2010, 10:04 AM
stoanee's Avatar
Hall Of Fame
F@H
 
Join Date: Jun 2007
Location: Lacombe Alberta
Posts: 3,228

My System Specs

Default

Nice discourse on the whole vid card testing routine. This will probably not be enough to appease the rabid fanboys though. It doesn't help that we are perceived as a pro Nvdia site. ;) (sarcasm intended)
__________________
heatware
Reply With Quote
  #18 (permalink)  
Old June 16, 2010, 10:25 AM
chrisk's Avatar
Folding Captain
 
Join Date: Jul 2008
Location: GTA, Ontario
Posts: 7,396

My System Specs

Default

Good work and appreciated. I too wonder how long some sites bench a particular game as it is not always available. I agree with Stoanee though that it will not detract the fan-boys, but for those with a open mind this article should be required reading.
__________________
Fold for team #54196
Reply With Quote
  #19 (permalink)  
Old June 16, 2010, 11:21 AM
SKYMTL's Avatar
HardwareCanuck Review Editor
 
Join Date: Feb 2007
Location: Montreal
Posts: 11,404
Default

The whole NVIDIA versus ATI debate doesn't really factor into this in any way IMO. I made no mention of either card in the conclusion and only a few vague references throughout the article. The whole point was to show what SHOULD be benchmarked and what should be avoided at all costs.

Unfortunately, we all know that this will do nadda when it comes to "cleaning up" the vast majority of GPU reviews out there. We still won't see details about benchmarking processes, etc just because it isn't in the interest of some sites to be transparent and admit they are using the wrong tools.
__________________
Reply With Quote
  #20 (permalink)  
Old June 16, 2010, 01:11 PM
MpG's Avatar
MpG MpG is offline
Hall Of Fame
 
Join Date: Aug 2007
Location: Kitchener, ON
Posts: 3,136
Default

Excellent article, worth the wait, and the effort that went into it is much appreciated! While this isn't the first I've seen of canned benchmarks failing to match up to in-game experience, it's nice to see the numbers laid out to get an idea of exactly how much discrepancy there actually is.

My own two cents is that I REALLY like the full framerate graphs. As good as it is to get average and minimums (and know the exact methodology that made them), I find seeing the FPS-spread just provides a whole extra depth of information to use when making a purchasing decision.
__________________
i7 2600K | ASUS Maximus IV GENE-Z | 580GTX | Corsair DDR3-2133
Reply With Quote
Reply


Thread Tools
Display Modes

Similar Threads
Thread Thread Starter Forum Replies Last Post
Guide to the Hardware Canucks GPU Benchmarking Process SKYMTL Video Cards 18 April 3, 2014 05:58 PM
Christmas '08: GPU Performance Part II Comment Thread SKYMTL Video Cards 26 January 15, 2009 09:43 AM
The Games of Christmas '08 GPU Performance Comment Thread SKYMTL Video Cards 33 December 12, 2008 06:39 PM