To Include 3DMark Vantage testing or not with GPU reviews??

[FiXT]

I say keep both -
3dMark06 has been a backbone of reviews for the past 3 years - Keep the resolution and settings at default. It is a globally known benchmark no matter how poor of one.

Develop a new strategy for Vantage. Becuase the developers have left it so open to play with, most review websites are not going to be using the same settings across the board anyways. Come up with a plan - I like the entry, preformance, extreme scenario for different price points. Heck maybe even run two - Preformance as a default and high/extreme for higher cards or entry/performance for lower end cards?

That being said, don't divide it by price. Price's change, things like the 8800GT is known for there low price/performance ratio and there has to be room for play.

[burebista]

SKYMTL your site is one of the firsts when I read trusty reviews. For benching games I like alot [H]OCP's methodology, but it's time consuming...
As for games, CoD4, Bioshock, Supreme Commander, maybe.

[Eldonko]

Originally Posted by FiXT

Keep the resolution and settings at default. It is a globally known benchmark no matter how poor of one.

I agree, I think it is more important for users to compare the review 3dmark scores to their own than see the scores at high settings and such. The majority of people use default settings for 06, and sites like hwbot require it for a fair comparison.

[SKYMTL]

Originally Posted by burebista

SKYMTL your site is one of the firsts when I read trusty reviews. For benching games I like alot [H]OCP's methodology, but it's time consuming...
As for games, CoD4, Bioshock, Supreme Commander, maybe.

Unfortunately, I will not make any comments about HardOCP's methodology since I don't want to spark a war between websites. Let's just say that their approach will never be done by Hardware Canucks. Ever.

Last I checked CoD4 did not have the option for a timedemo

Ditto with Bioshock

And Supreme Commander results are very much affected by the processor so we did away with that early on in our reviews.

[Spaceman-Spiff]

I say use it anyway, but if the results are too erratic or inconsistent (compared to the rest of the tests), you don't need to post it.

I suppose in a month or two with newer drivers and patches, it will be a better testing tool.

[SKYMTL]

All right, a decision has been taken and I will be doing the following:

I WILL NOT be using 3DMark Vantage until:

1: Futuremark releases a patch to take care of a number of performance issues, massive result inconsistencies, crashes and general instabilities.

2: ATI releases a WHQL driver to support it (their "hotfix" is essentially a beta driver)

3: Nvidia releases a WHQL driver for the 8-series (hell, the last 8-series WHQL was released in DECEMBER) and 9-series which properly supports it

So, once the stars align just right you will see us benching with Vantage


We will ADD the following starting immediately:

- Enemy Territory benchmark (timedemo)
- OpenGL Fur Render Benchmark
- 3DMark runs with the following:

1280x1024 @ defaults
1600x1200 @ defaults
2560x1600 @ defaults

1280x1024 @ 4xAA / 16xAF
1600x1200 @ 4xAA / 16xAF
2560x1600 @ 4xAA / 16xAF (only for high-end)


We will ADD (sometime in the near future) the following:

Company of Heroes: Opposing Fronts 30 minute in-game level frames per second (the in-game benchmark does NOT reflect actual gaming experience)


We will REMOVE the following:

Call of Juarez Benchmark

Hope that sits well with everyone. ;)

[Eldonko]

Looks good to me. Hopefully futuremark listens to users and allows for offline submissions (on the $20 version) and fixes these bugs, awardfabrik has over 400 sigs on their petition already.

[Eldonko]

Hotfix coming next week :) Link