Unit tests / Game tests

[cruonit]

Are you using any unit tests before releasing because I see with every release there has to be community feedback if a game is broken or not.

Have you thought about making some sort of game test/ unit test engine to test various games automatically or semiauto before releasing.

I know it would be hard to write an automatically test for various games but a test that uses user feedback would be nice like an application that takes screenshots from games at specified time and a webpage where the community manually compares the picture with an working one (from last release or similar - a way to see if the release has broken a game).

If this makes sanse i would like to contribute coding this little semiauto testing system. Maybe some autoamtization could be done with neural networks - image recognition but the problem is the game speed could be different from release to release so an hardcore image comperation could't work.

[Chaoscode]

wth are you talking about?

[cruonit]

When someone changes the code some games can get broken (whats happening from release to release).

In order to get feedback what is broken/what games that previously worked the developers need feedback from the community (testers).

I am suggesting a more sistematical method how to semiauto(or auto) test if some games got broken after a release.

let's say mario worked on release 4460 with some options enabled so we have a saved picture at time 0:20 after the game was launched(let's say its the menu of the game)

but let's say on release 4525 the game is broken and at 0:20 after the game was lounched automatically by the testing application we only see a black screen or some textures are missing.

It's hard to write unit tests to test this by coding. It's easier to detect it visually by comparing two screenshots in this example one archived from the release 4460 and the other generated by the testing application from the 4525 release.

So the testers only need to compare screenshots to see if a game is broken and such compares can be done quickly.


Maybe this wouldn't be useful for the developers but the community will be able to see how the game worked in different releases (they could also see the FPS from the screenshots)

[Chaoscode]

and who is going to write this program?

[cruonit]

I think i have a idea how to do it but only if it makes sense and if it would be useful to the project.

[Chaoscode]

its open source, go for it

[shuffle2]

(11-11-2009, 09:59 AM)cruonit Wrote: Are you using any unit tests

not really, read on!

(11-11-2009, 09:59 AM)cruonit Wrote: Have you thought about making some sort of game test/ unit test engine to test various games automatically or semiauto before releasing.

There are unit tests to ensure specific things are working correctly, check http://code.google.com/p/dolphin-emu/source/browse/trunk#trunk/Source/UnitTests , but not for "games", per se.

(11-11-2009, 09:59 AM)cruonit Wrote: I know it would be hard to write an automatically test for various games but a test that uses user feedback would be nice like an application that takes screenshots from games at specified time and a webpage where the community manually compares the picture with an working one (from last release or similar - a way to see if the release has broken a game).

Most testing is done by developers who make the changes. Only recently has the community been needed more than normal; mainly because code is being changed in the graphics plugins which makes older cards cranky. (People, update your stuff already! ;p )

(11-11-2009, 09:59 AM)cruonit Wrote: If this makes sanse i would like to contribute coding this little semiauto testing system. Maybe some autoamtization could be done with neural networks - image recognition but the problem is the game speed could be different from release to release so an hardcore image comperation could't work.

Like chaos said, go for whatever

[ector]

(11-11-2009, 12:47 PM)cruonit Wrote: When someone changes the code some games can get broken (whats happening from release to release).

In order to get feedback what is broken/what games that previously worked the developers need feedback from the community (testers).

I am suggesting a more sistematical method how to semiauto(or auto) test if some games got broken after a release.

let's say mario worked on release 4460 with some options enabled so we have a saved picture at time 0:20 after the game was launched(let's say its the menu of the game)

but let's say on release 4525 the game is broken and at 0:20 after the game was lounched automatically by the testing application we only see a black screen or some textures are missing.

It's hard to write unit tests to test this by coding. It's easier to detect it visually by comparing two screenshots in this example one archived from the release 4460 and the other generated by the testing application from the 4525 release.

So the testers only need to compare screenshots to see if a game is broken and such compares can be done quickly.


Maybe this wouldn't be useful for the developers but the community will be able to see how the game worked in different releases (they could also see the FPS from the screenshots)

I agree it would be GREAT to have something like this, someone just has to do it What's needed is to have to little hooks in dolphin to make it auto-screenshot at frame number 200 (for example) and exit or load the next game, etc, and then make a program or script to drive it all.

The only problem is that anyone who'd like to run it needs to have all or at least many of the games being tested.

[dstruct2k]

(11-11-2009, 07:55 PM)ector Wrote: What's needed is to have to little hooks in dolphin to make it auto-screenshot at frame number 200 (for example) and exit or load the next game, etc, and then make a program or script to drive it all.

Wasn't support for the LUA scripting engine recently added to Dolphin? Seems that LUA should be able to pull this off quite easily. Just measure how long each game takes to display 1000 frames, take screenshots of the 500th and 1000th frames for comparison, and load the next game in the list.

The only part where an external app would be needed is auto-image-comparison to detect broken features, although having a library of "perfect" screenshots should allow a pixel-by-pixel comparison with a percentage grade given to the Dolphin screenshots. (My only question is: where do we get the "perfect" screenies?)