Benchmark Talk (Luigi´s Mansion GPU Benchmark c.s. (Unofficial))

[oliverfrancisco]

Well, well...

The score table is 3 to 0 (at this moment in time ), so that means that everyone here will have to wait until 4.0 is released.

Anyway I´m still waiting for some votes...

[Shonumi]

moosehunter Wrote:1) I don't know if running a dtm on multiple computers with different specs and IO speeds would result in the same emulation.

I don't see how I/O speeds (disk, I assume is what you're talking about specifically) would be an issue. A .dtm is nothing more than a save state that records button presses at specific times. Since the benchmark will likely be stationary, it's nothing more than a save state at that point. Disk I/O shouldn't play any importance, at least none that we should be able to detect. As long as we choose a game that will significantly stress a specific component while reducing the importance of other components (e.g. this benchmark is pretty good, WW is pretty minimal on the GPU side), we should be good even with computers of different specs.

moosehunter Wrote:2) I don't think there's a way to launch a recording from the command line.

Aside from automation, what's the advantage to running it from the command line? Ideally, the .dtm will start at the point where the benchmark is supposed to take place, wait something like 10 seconds, then stop. Again, it's like a save state. You can open Dolphin and play a .dtm movie without actually having to manually run the game yourself (at least I think that's how it works, I haven't tried it in over a year), so even when recording FPS to a file, it wouldn't mess up the results because someone can't, for example, spend the first 30 seconds fiddling through the intro screen. So yeah, a playing the .dtm file is pretty automatic already (which is very convenient for our purposes ).

moosehunter Wrote:3) There isn't a command line option that'll stop the emulation after a certain number of frames, or any GUI option for that matter.

I think a better way than by doing it after a certain number of frames (I think .dtm files play the game normally after they're "finished", so the game resumes after the "movie" is done) is to just measure how many frames were processed per second. Have the .dtm file start off wherever the benchmark takes place, record the FPS to a file, but only the first 10 seconds of data will count. As long as the .dtm was recorded for a safe enough length (maybe 60 seconds, just for those real speed demons ) there'd be no issue as far as I can tell.

moosehunter Wrote:Maybe I'm just making things too complicated. Anyway, it's just a little idea I had.

Probably. In the end, it's not the hard numbers that matter the most, it's the application: finding the relative performance of certain hardware. It doesn't need to be 100% exact to be useful, although we should aim to be as accurate as possible. If some things are beyond our control, we needn't lose any sleep.

[oliverfrancisco]

Oh boy, I just can´t wait for Dolphin 4.0 to be released. It´s so exciting!

[moosehunter]

(08-30-2013, 03:32 AM)Shonumi Wrote: I don't see how I/O speeds (disk, I assume is what you're talking about specifically) would be an issue. A .dtm is nothing more than a save state that records button presses at specific times. Since the benchmark will likely be stationary, it's nothing more than a save state at that point. Disk I/O shouldn't play any importance, at least none that we should be able to detect. As long as we choose a game that will significantly stress a specific component while reducing the importance of other components (e.g. this benchmark is pretty good, WW is pretty minimal on the GPU side), we should be good even with computers of different specs.

I don't think it would make a difference on most computers, but I've seen some games take fewer frames to load depending on how fast the device they're being loaded from is. And if the benchmark was based off of save states, why not just run the save states so there's no need to own the game. You can't load savestates without a game now, but I seem to remember loading save states without the ISO when I was ripping music from some games about 2 years ago. Depending on the game, it would just continue if it read from the disc, or hang. Although I could just be remembering wrong. If ZWW didn't load anything after the state was saved, then it would be useful to make Dolphin able to load savestates without an ISO.
I was trying to make it so that distributing a save state wouldn't be necessary, but if it's better that way, then it doesn't really matter.

(08-30-2013, 03:32 AM)Shonumi Wrote: Aside from automation, what's the advantage to running it from the command line? Ideally, the .dtm will start at the point where the benchmark is supposed to take place, wait something like 10 seconds, then stop. Again, it's like a save state. You can open Dolphin and play a .dtm movie without actually having to manually run the game yourself (at least I think that's how it works, I haven't tried it in over a year), so even when recording FPS to a file, it wouldn't mess up the results because someone can't, for example, spend the first 30 seconds fiddling through the intro screen. So yeah, a playing the .dtm file is pretty automatic already (which is very convenient for our purposes ).

I think a better way than by doing it after a certain number of frames (I think .dtm files play the game normally after they're "finished", so the game resumes after the "movie" is done) is to just measure how many frames were processed per second. Have the .dtm file start off wherever the benchmark takes place, record the FPS to a file, but only the first 10 seconds of data will count. As long as the .dtm was recorded for a safe enough length (maybe 60 seconds, just for those real speed demons ) there'd be no issue as far as I can tell.

Basically just that. Automation. It could make it easier to automate multiple runs, but it's not necessary. And I forgot that when playing a .dtm, Dolphin will stop when it finishes.

(08-30-2013, 03:32 AM)Shonumi Wrote: Probably. In the end, it's not the hard numbers that matter the most, it's the application: finding the relative performance of certain hardware. It doesn't need to be 100% exact to be useful, although we should aim to be as accurate as possible. If some things are beyond our control, we needn't lose any sleep.

Well, right now, it seems like a savestate + .dtm + fps logging would be the best and simplest method. It would be even better if an ISO wasn't required to load a save state so that everybody could run the benchmark.

[oliverfrancisco]

The thread is renamed... again.