(05-27-2012, 07:39 AM)Starscream Wrote: [ -> ] (05-27-2012, 07:34 AM)dEnigma Wrote: [ -> ]Why should I do that
To show the tangible in-game speed difference?
Yeah, because obviously screenshots can't be faked and are soo representative of overall performance...
(05-27-2012, 07:39 AM)Starscream Wrote: [ -> ] (05-27-2012, 07:34 AM)dEnigma Wrote: [ -> ]the fps fluctuate anyway,
How can you tell if you gained any fps during the game if the fps is fluctuating?
FRAPS outputs a spreadsheet with the rendering time/frame and it's easy to compute min/avg/max FPS from there.
Okay, so no real measurable difference in-game, that's what I thought.
(05-27-2012, 08:05 AM)Starscream Wrote: [ -> ]Okay, so no real measurable difference in-game, that's what I thought.
Yes there is, going from 2819 to 3134 frames per minute in the first benchmark and 2843 to 3234 in the second one. This is a far better method of measuring performance increase than looking at the FPS counter at random points in time.
Okay delroth, your hard work is much appreciated. Thanks for letting us test this.
Quote:Also I might just as well forge the screenshots if I was out to fool you
True but it's still a lot more credible than text.
(05-27-2012, 12:21 PM)NaturalViolence Wrote: [ -> ]Quote:Also I might just as well forge the screenshots if I was out to fool you
True but it's still a lot more credible than text.
uuuh it isnt. even less so.
both can be faked just as easily. but the screenshot doesnt give any indication wether performance has truly increased whereas the spreadsheets most certainly do
Quote:both can be faked just as easily.
........
Um....no. While screenshots can be faked it's harder to fake a screenshot then to just type some text saying whatever you want or fill out a spreadsheet.
Quote:but the screenshot doesnt give any indication wether performance has truly increased
Yes it does. Dolphin has an fps counter and a gamespeed counter.
The FPS and gamespeed counter are only snapshots of a specific instant of the game, you can't use these counters to compute average/stddev, which are the two really useful metrics to show performance changes. The spreadsheet exported by FRAPS is a lot better.
I don't know why you assume people getting strange results are "cheating" by faking measurements. I can understand assuming people make mistakes in their testing protocol (not using the same settings, shader cache not cleared, warm disk/code cache, ...) but seriously, faking? Why?
I don't think he was actually saying someone was faking. I was saying that an actual game-shot was a better method of testing and then someone mentioned the word "faked" (you). It was all hypothetical talk after that. From what I read on IRC at the time and from my own testing, I'm pretty sure everyone agreed that any speedup with this was either imaginary or the fault of bad testing. That's probably why I was having a hard time believing his speedup claims.
(07-09-2012, 09:30 AM)Starscream Wrote: [ -> ]I was saying that an actual game-shot was a better method of testing
It is not, the statistical significance of a single datapoint is near zero. Testing that way actually encourages bias by making people try to choose the datapoint that matches the best with what they think (which differs from faking results: the datapoint is still real but it could be over the 80-90th percentile for example).
I'm surprised by this speedup too (and still have a hard time believing it) but his measurements are most likely correct and definitely more credible than a screenshot for anyone who knows about stats. I'm just not sure if he's measuring the right thing (settings issue, warm cache, everything I said in my previous post basically).