A new look at game benchmarking

[r0bc]

SKY have any thoughts on this.
I looked around and didn't see it posted anywhere, remove it if it has.

Inside the second: A new look at game benchmarking - The Tech Report - Page 1

[r0bc]

Could the increased micro shutters in his testing been caused by the 1GB limitation of the lower end cards at that resolution?

[JJThomp]

Looks interesting, I actually asked myself this at christmas because I noticed this type of thing happening on my crappy laptop, where my fps average would stay above 30 but I would have some stuttering. On the resolution he was testing on 1 gb was more than enough. For example I read in a review that metro 2033 maxed out on 7680x1600 was only using 2.2 gb so really startcraft is not a GPU intensive game, and it wasn't anywhere near that resolution.

[Dzzope]

i know the article says the "stuck" frames are less of an issue than the "jitter", But I would argue that if the frame "sticks" often enough (maybe ever 30 sec to minuite or worse if more) this would deter me from buying a card as I have experienced this on a HD 5770 in my old rig.
And although in no way conclusive or tested near thoroughly enough i noticed it was far more prevalent on the AMD cards rather than the nvidia ones through out the graphs.


However, I'm glad to see this brought up here, as I always think minimum FPS is far more important than max / average. Stuttering or "jitter" is equally important and IMO "stuck" frames are a high priority too.
Some sort of testing along the lines of the graphss (not the percentile charts as they don't show the stuck frames well IMO and as mentioned in the article have certain problems) would, if not tooo much extra work, be a welcome addition.
That said it's very conditional to the segment that is chosen to show the data from and a full graph for a full test would be huge (mebby linked to rather than embedded, or a scroll box).

Anyway.. good article, worth a read and some thought.

[Bond007]

Nice read. Not saying its the perfect answer, but it would be nice to see "microstutter/jitter" somewhat accounted for in reviews so that people could get a better real world picture of performance and playablilty.

[SKYMTL]

I read through the whole thing twice the other day. Basically, I agree with some of what he says and don't agree with other parts. I won't go into too much detail but there are some very important point that everyone should take away from the article:

- Microstuttering is only APPARENT to the viewer in very few situations. Scott mentions this a few times and I agree.

- There is no way to conclusively test for microstutter. The polling rate in FRAPS and even PlayClaw isn't frequent enough to accurately test for it.

- Due to the two issues above, there is absolutely no way to really test for user-viewable microstutter. Some people may experience it while others won't.

- Like it or not, a playable game experience is still defined by the 30-60FPS "rule". Below 30FPS for most games and below 60fps for fast paced games and the player will experience begin to suffer. So the standard type of benchmark is still perfectly valid.

- Frametime as he discussed (first example on the first page) can be completely invalidated by items that don't include the GPU's ability to render a scene. He showed frametime increasing exponentially for a single frame and deemed that unplayable if it happened often. I disagree. A good example of this is the Witcher 2. In order to provide a fluid area to area experience, the game preloads upcoming areas which will cause frametime to spike and cause a "stutter" every few minutes. This is a FEATURE within the game (rather than a GPU issue) that is being used more and more (Dues Ex: HR, Dragon Age 2, etc) to provide a seamless gaming experience without load screens.


One thing I didn't see is what software he used. Without that, no one can repeat and validate his results...and THAT is concerning. Methinks it is FRAPS but PlayClaw can achieve the same....and with slightly different results.

In short, the article was interesting but regardless of the hyperbole, we won't be including additional testing until there is a tool available that can poll framerates and sync issues at very quick intervals. Hence his "inside a second" title.

On the flip side of the coin, I have been planning to add frametimes for some time now. It adds an interesting dynamic to the testing. Basically, instead of using AA and non AA tests, I'll likely be only using AA and adding frametimes. But we will see if that actually happens since the tools available don't measure up IMO.

[r0bc]

Originally Posted by SKYMTL

This is a FEATURE within the game (rather than a GPU issue) that is being used more and more (Dues Ex: HR, Dragon Age 2, etc) to provide a seamless gaming experience without load screens.

I actually noticed this, wandered what it was.
Thanks for your reply, was curious what you thought about the article.