Go Back   Hardware Canucks > HARDWARE > Video Cards

    
Reply
 
LinkBack Thread Tools Display Modes
  #1 (permalink)  
Old June 15, 2010, 08:16 PM
SKYMTL's Avatar
HardwareCanuck Review Editor
 
Join Date: Feb 2007
Location: Montreal
Posts: 11,662
Default GPU Benchmarking Methods Investigated: The Comment Thread

GPU performance has been a hotly debated topic recently and Hardware Canucks has decided to jump face-first into the murky world of games, timedemos, in-game rolling benchmarks and more. In this article we take an in-depth look at various benchmarking methods used throughout the industry and put their accuracy (or lack thereof) to the ultimate test.

[ Read full article ]
Reply With Quote
  #2 (permalink)  
Old June 15, 2010, 08:36 PM
Banned
F@H
 
Join Date: Aug 2007
Location: mtl
Posts: 12,694
Default

lets take a look on your industry here :)
Reply With Quote
  #3 (permalink)  
Old June 15, 2010, 08:43 PM
Delavan's Avatar
Hall Of Fame
 
Join Date: May 2007
Location: Mtl
Posts: 1,149
Default

Good work SKYMTL and HWC team!

It's a "noble" cause to try to clean up everybody's mess, start from scratch and give us the right information.

I find that, opposed to a few years back, there is now too many sites doing reviews. Now, you can find reviews that tell you "what you want to hear" basically. You cannot really make your mind by reading a lot of reviews, unless you go crazy and buy the most expensive piece of kit.

It reinforces the "brand loyalty", because nobody will risk to buy anything else but the "XFX or EVGA FLAGSHIPS, Corsair HX/RAM" or insert your "safe buy". Only the best gear, reaching max performance, max reliability and max overclockability keeps some sort of universal support, the rest of the crowd is just a grey zone...it's getting to the point where more and more systems look the same.

One site will tell you that one piece of hardware is the next big thing after sliced bread, and the other site talks about major flaws in design, performance or reliability of such hardware.

Same thing for Graphics card's performance...too many sites, to many different (and opposite) conclusions.

A real mess...really.

Thanks for going above and beyond to feed us the right information.
__________________
i7 4770K @ 4.5GHz / Corsair H100i / Maximus VI HERO / 8GB Corsair Vengeance Red LP 1.5V 1866 / N560GTX Ti Twin FROZR II OC / Corsair Performance Pro 128GB SSD / Intel 520 120GB SSD / WD Black 640 GB storage

ROG G53SX laptop: i7 2630QM / 12GB DDR3 / GTX560m 2GB / 750GB 7200rpm (14 000 3Dmarks06)

Last edited by Delavan; June 15, 2010 at 08:56 PM.
Reply With Quote
  #4 (permalink)  
Old June 15, 2010, 09:40 PM
Trial Member
 
Join Date: Jun 2010
Posts: 1
Default

I thought I'd register just to say that this was an excellent and informative article!

Something that came to mind while reading the article: how do driver revisions factor into the benchmark process? I understand that different websites have different policies regarding which version of display driver they use in their testing, whether it be using the latest stable drivers, or using beta drivers. Obviously results are going to vary between driver releases, so how does (or how should) that play into benchmarking?

The situation I'm thinking of is when comparing a product against its previous generation counterpart, perhaps a GTX470 versus a GTX260, and the reviewer cannot be bothered to retest older cards. It seems like a situation where differing driver versions might mislead the reader into thinking some product performs far better than its predecessor when in fact the difference is nowhere near as great.
Reply With Quote
  #5 (permalink)  
Old June 15, 2010, 09:49 PM
SKYMTL's Avatar
HardwareCanuck Review Editor
 
Join Date: Feb 2007
Location: Montreal
Posts: 11,662
Default

Quote:
Originally Posted by theDarkHorse View Post
I thought I'd register just to say that this was an excellent and informative article!

Something that came to mind while reading the article: how do driver revisions factor into the benchmark process? I understand that different websites have different policies regarding which version of display driver they use in their testing, whether it be using the latest stable drivers, or using beta drivers. Obviously results are going to vary between driver releases, so how does (or how should) that play into benchmarking?

The situation I'm thinking of is when comparing a product against its previous generation counterpart, perhaps a GTX470 versus a GTX260, and the reviewer cannot be bothered to retest older cards. It seems like a situation where differing driver versions might mislead the reader into thinking some product performs far better than its predecessor when in fact the difference is nowhere near as great.
Thanks for the compliments.

I didn't want to wade into the driver revision debate just yet since those deserve an article all on their own.

Here's where we stand currently: reviewers usually bend over backwards to test with the latest driver revisions and it consumes a massive amount of time and effort. In my opinion and from testing dozens of revisions now, when it comes to single card performance in existing games, performance increases are next to nil in most cases. Granted, every now and then we will get a massive update that pushes things to new levels but for the most part I have found that developer updates / patches affect performance more than any driver update can.

This wasn't always the case but now that NVIDIA and ATI both have good developer relations, they are more prepared with optimizations when a game is released rather than improving performance months down the road.
__________________
Reply With Quote
  #6 (permalink)  
Old June 15, 2010, 09:49 PM
burebista's Avatar
Allstar
 
Join Date: Sep 2007
Location: Romania
Posts: 598

My System Specs

Default

Thanks SKY. This is one of the best article I've read in 13 years of reading the net. It should be a reference.
__________________
If it ain't broke... fix it until it is.
Reply With Quote
  #7 (permalink)  
Old June 15, 2010, 10:44 PM
Banned
F@H
 
Join Date: Aug 2007
Location: mtl
Posts: 12,694
Default

the game patches should be an eye opener, to say the least.
Reply With Quote
  #8 (permalink)  
Old June 15, 2010, 11:10 PM
Rookie
 
Join Date: Jun 2010
Posts: 8
Default

Nice article. I also registered to the forums just because of it.

I found the frame rate graphs particularly enlightening. They show how meaningless average / min / max FPS can be. The Just Cause 2 one in particular shows how frame rate can vary significantly, with NVIDIA and AMD trading places, and there's a specific stretch where the AMD card just suffers. Though the AMD card gets a higher maximum, it's quite meaningless as it's just a blip.

This means that even this length of time doesn't tell the entire story. Is the dip for AMD representative of most gameplay, or is the (larger, in this sample) section where the cards show reasonable parity more representative? And if a card dips under 30 fps for one second out of 2 minutes, is that really significant for gameplay? Because it's clearly impossible to tell how often these dips happen just from the min/avg figures.

It's be interesting to consider whether different statistics could give a better feeling to how the cards handle the games.
Reply With Quote
  #9 (permalink)  
Old June 15, 2010, 11:14 PM
Trial Member
 
Join Date: Jun 2010
Posts: 1
Default

nice article, good job
Reply With Quote
  #10 (permalink)  
Old June 16, 2010, 12:42 AM
Rookie
 
Join Date: Jun 2010
Posts: 2
Default

Thank you team HWC
Reply With Quote
Reply


Thread Tools
Display Modes

Similar Threads
Thread Thread Starter Forum Replies Last Post
Guide to the Hardware Canucks GPU Benchmarking Process SKYMTL Video Cards 18 April 3, 2014 05:58 PM
Christmas '08: GPU Performance Part II Comment Thread SKYMTL Video Cards 26 January 15, 2009 09:43 AM
The Games of Christmas '08 GPU Performance Comment Thread SKYMTL Video Cards 33 December 12, 2008 06:39 PM