Does nvidia dictate what games can and cannot be benched?

[AkG]

There is a difference between working with the Devs to ensure a great game which runs great on XYZ brand GPU "out of the gates"....and working with the devs ensure a great game which runs great on XYZ brand GPU AND like crap on the competition. THAT is simply not cricket. THAT hurts us the consumers more than anything else. Can you imagine having to buy TWO gpus (and gaming rigs) to ensure that every game gave a good game experience? Ugh.

[Soultribunal]

Originally Posted by AkG

There is a difference between working with the Devs to ensure a great game which runs great on XYZ brand GPU "out of the gates"....and working with the devs ensure a great game which runs great on XYZ brand GPU AND like crap on the competition. THAT is simply not cricket. THAT hurts us the consumers more than anything else. Can you imagine having to buy TWO gpus (and gaming rigs) to ensure that every game gave a good game experience? Ugh.

Yes, thank you for touching base on this AkG, I forgot to and it is really relevent to the conversation as well.

-ST

[sswilson]

Originally Posted by AkG

There is a difference between working with the Devs to ensure a great game which runs great on XYZ brand GPU "out of the gates"....and working with the devs ensure a great game which runs great on XYZ brand GPU AND like crap on the competition. THAT is simply not cricket. THAT hurts us the consumers more than anything else. Can you imagine having to buy TWO gpus (and gaming rigs) to ensure that every game gave a good game experience? Ugh.

Exactly. It's like exclusive releases for a particular console, I don't play FPS so therefore don't own a console, but if I did would probably avoid even the AAA titles that don't release across all platforms.

[Squeetard]

I'll third those sentiments. That is the type of competition that only hurts consumers. Wouldn't it be nice to have both AMD and nVidia send people to help dev a game and ensure it runs on both platforms well? Most games already detect what card you are running and have different code paths.

[SKYMTL]

I'm really sorry I didn't see this when it was first posted. I'll take a crack at answering this. PLEASE FEEL FREE TO REPOST THIS as I am only saying this once.

First and foremost, I can tell you straight off the bat that neither NVIDIA nor AMD requires that certain benchmarks be used. Both companies issue what's called a Reviewer's Guide that gives reviewers an idea of general performance numbers for the graphics card in question against the competition. Under no circumstance are the games mentioned in the RG a requirement. Rather, they are a rough guideline that reviewers are supposed to use to ensure their review unit is operating within specified norms.

On the other hand, both AMD and NVIDIA love it when reviewers go above and beyond the call of duty. That means calling out and possibly benchmarking features that they talk about in marketing materials and that they feel are beneficial to end users. Discussing PhysX, Eyefinity or 3D Vision may seem like fanboyism to some but ignoring them is a huge mistake IMO. The trick is not to let too many of those "extra" features impact one's overall conclusion as they are mostly value added items rather than an integral part of the gaming experience for most people.

Now, the real question lies in why many writers use games like Sleeping Dogs, Dirt Showdown, Shogun 2, Batman AC and in the past Far Cry 2, Just Cause 2, Metro 2033 and others within reviews. Every one of those titles has a major feature that some reviewers appreciate: a built in or stand-alone benchmarking tool. NVIDIA and AMD love having developers add the tools since it virtually guarantees that certain lazy reviewers will use these sponsored titles in their benchmarking suites. Brilliant, isn't it? This is why we see so many Gaming Evolved or TWIMTBP titles in reviews.

To expand upon my point of reviewer laziness, let's focus upon some situations and how accuracy has been thrown into the wind in favor of production speed. While built-in benchamarks can be accurate, all too often their performance is under no circumstance reflective of in-game situations. Take a look at Sleeping Dogs or Batman: Arkham City for examples of this; a player will NEVER be doing a "flythough" of a game scene. Ever. We also can't forget that both NVIDIA (examples: Just Cause 2 & Batman: AC) and AMD (examples: Sleeping Dogs & Shogun 2) have been known to optimize drivers for specific instances occurring within the aforementioned predetermined sequences.

Unfortunately, most reviewers don't care about accuracy, they just care about getting comparative benchmark numbers, regardless of whether a GPU performs a certain way when actually playing a game or not. Plus, what could be easier than running a script which does the preset benchmarking for them, spits out results and doesn't involve actually sitting down for hours to playing the games? You can pretty much see who does this (and yes, even sites that say they use runthroughs are sometimes lying) by simply running through the built-in benchmark with a similar system and comparing the results.

With that being said, benchmarking a single runthrough has issues as well since using one instead of a repetitive built-in benchmark sequence introduces a certain amount of variance into the equation. Variance is bad when a a mere 5% differential could separate one GPU from another and skew a conclusion. We try to eliminate that by running through the same scene three individual times. However, HWC also uses built-in sequences (right now the only one we're using is Dirt 3 since it can be highly modified from its original form) but ONLY after extensive validation is done that ensures the sequence A) depicts an actual gameplay sequence B) lines up with in-game performance.

Sorry for this post's length but there were quite a few things to cover.

[velias]

Great and detailed response sky, thanks!