• Login
  • Register
  • Dolphin Forums
  • Home
  • FAQ
  • Download
  • Wiki
  • Code


Dolphin, the GameCube and Wii emulator - Forums › Dolphin Emulator Discussion and Support › Development Discussion v
« Previous 1 ... 100 101 102 103 104 ... 116 Next »

Unit tests / Game tests
View New Posts | View Today's Posts

Thread Rating:
  • 3 Vote(s) - 2.33 Average
  • 1
  • 2
  • 3
  • 4
  • 5
Thread Modes
Unit tests / Game tests
11-11-2009, 09:59 AM (This post was last modified: 11-11-2009, 10:02 AM by cruonit.)
#1
cruonit Offline
Junior Member
**
Posts: 5
Threads: 1
Joined: Jul 2009
Are you using any unit tests before releasing because I see with every release there has to be community feedback if a game is broken or not.

Have you thought about making some sort of game test/ unit test engine to test various games automatically or semiauto before releasing.

I know it would be hard to write an automatically test for various games but a test that uses user feedback would be nice like an application that takes screenshots from games at specified time and a webpage where the community manually compares the picture with an working one (from last release or similar - a way to see if the release has broken a game).

If this makes sanse i would like to contribute coding this little semiauto testing system. Maybe some autoamtization could be done with neural networks - image recognition but the problem is the game speed could be different from release to release so an hardcore image comperation could't work.
Find
Reply
11-11-2009, 12:29 PM
#2
Chaoscode Offline
Senior Member
****
Posts: 382
Threads: 11
Joined: Mar 2009
wth are you talking about?
CPU: E8500 Not OC
Memory 8 Gigs DDR2-800Mhz
GPU: NVidia GeForce 8800GT 256
OS: Windows 7 64bit
Monitor: LCD TV 42' 720P
Find
Reply
11-11-2009, 12:47 PM (This post was last modified: 11-11-2009, 12:50 PM by cruonit.)
#3
cruonit Offline
Junior Member
**
Posts: 5
Threads: 1
Joined: Jul 2009
When someone changes the code some games can get broken (whats happening from release to release).

In order to get feedback what is broken/what games that previously worked the developers need feedback from the community (testers).

I am suggesting a more sistematical method how to semiauto(or auto) test if some games got broken after a release.

let's say mario worked on release 4460 with some options enabled so we have a saved picture at time 0:20 after the game was launched(let's say its the menu of the game)

but let's say on release 4525 the game is broken and at 0:20 after the game was lounched automatically by the testing application we only see a black screen or some textures are missing.

It's hard to write unit tests to test this by coding. It's easier to detect it visually by comparing two screenshots in this example one archived from the release 4460 and the other generated by the testing application from the 4525 release.

So the testers only need to compare screenshots to see if a game is broken and such compares can be done quickly.


Maybe this wouldn't be useful for the developers but the community will be able to see how the game worked in different releases (they could also see the FPS from the screenshots)
Find
Reply
11-11-2009, 12:49 PM
#4
Chaoscode Offline
Senior Member
****
Posts: 382
Threads: 11
Joined: Mar 2009
and who is going to write this program?
CPU: E8500 Not OC
Memory 8 Gigs DDR2-800Mhz
GPU: NVidia GeForce 8800GT 256
OS: Windows 7 64bit
Monitor: LCD TV 42' 720P
Find
Reply
11-11-2009, 12:52 PM
#5
cruonit Offline
Junior Member
**
Posts: 5
Threads: 1
Joined: Jul 2009
I think i have a idea how to do it but only if it makes sense and if it would be useful to the project.
Find
Reply
11-11-2009, 12:53 PM
#6
Chaoscode Offline
Senior Member
****
Posts: 382
Threads: 11
Joined: Mar 2009
its open source, go for it Tongue
CPU: E8500 Not OC
Memory 8 Gigs DDR2-800Mhz
GPU: NVidia GeForce 8800GT 256
OS: Windows 7 64bit
Monitor: LCD TV 42' 720P
Find
Reply
11-11-2009, 01:30 PM (This post was last modified: 11-11-2009, 01:30 PM by shuffle2.)
#7
shuffle2 Offline
godisgovernment
*
Project Owner  Developers (Administrators)
Posts: 698
Threads: 17
Joined: Mar 2009
(11-11-2009, 09:59 AM)cruonit Wrote: Are you using any unit tests
not really, read on!
(11-11-2009, 09:59 AM)cruonit Wrote: Have you thought about making some sort of game test/ unit test engine to test various games automatically or semiauto before releasing.
There are unit tests to ensure specific things are working correctly, check http://code.google.com/p/dolphin-emu/source/browse/trunk#trunk/Source/UnitTests , but not for "games", per se.
(11-11-2009, 09:59 AM)cruonit Wrote: I know it would be hard to write an automatically test for various games but a test that uses user feedback would be nice like an application that takes screenshots from games at specified time and a webpage where the community manually compares the picture with an working one (from last release or similar - a way to see if the release has broken a game).
Most testing is done by developers who make the changes. Only recently has the community been needed more than normal; mainly because code is being changed in the graphics plugins which makes older cards cranky. (People, update your stuff already! ;p )

(11-11-2009, 09:59 AM)cruonit Wrote: If this makes sanse i would like to contribute coding this little semiauto testing system. Maybe some autoamtization could be done with neural networks - image recognition but the problem is the game speed could be different from release to release so an hardcore image comperation could't work.
Like chaos said, go for whatever Smile
Find
Reply
11-11-2009, 07:55 PM
#8
ector Offline
PPSSPP author, Dolphin co-founder
*
Project Owner  Developers (Administrators)
Posts: 189
Threads: 2
Joined: Mar 2009
(11-11-2009, 12:47 PM)cruonit Wrote: When someone changes the code some games can get broken (whats happening from release to release).

In order to get feedback what is broken/what games that previously worked the developers need feedback from the community (testers).

I am suggesting a more sistematical method how to semiauto(or auto) test if some games got broken after a release.

let's say mario worked on release 4460 with some options enabled so we have a saved picture at time 0:20 after the game was launched(let's say its the menu of the game)

but let's say on release 4525 the game is broken and at 0:20 after the game was lounched automatically by the testing application we only see a black screen or some textures are missing.

It's hard to write unit tests to test this by coding. It's easier to detect it visually by comparing two screenshots in this example one archived from the release 4460 and the other generated by the testing application from the 4525 release.

So the testers only need to compare screenshots to see if a game is broken and such compares can be done quickly.


Maybe this wouldn't be useful for the developers but the community will be able to see how the game worked in different releases (they could also see the FPS from the screenshots)

I agree it would be GREAT to have something like this, someone just has to do it Smile What's needed is to have to little hooks in dolphin to make it auto-screenshot at frame number 200 (for example) and exit or load the next game, etc, and then make a program or script to drive it all.

The only problem is that anyone who'd like to run it needs to have all or at least many of the games being tested.
Website Find
Reply
11-12-2009, 05:30 AM (This post was last modified: 11-12-2009, 05:32 AM by dstruct2k.)
#9
dstruct2k Offline
Ex-Dev
***
Posts: 109
Threads: 3
Joined: Aug 2009
(11-11-2009, 07:55 PM)ector Wrote: What's needed is to have to little hooks in dolphin to make it auto-screenshot at frame number 200 (for example) and exit or load the next game, etc, and then make a program or script to drive it all.

Wasn't support for the LUA scripting engine recently added to Dolphin? Seems that LUA should be able to pull this off quite easily. Just measure how long each game takes to display 1000 frames, take screenshots of the 500th and 1000th frames for comparison, and load the next game in the list.

The only part where an external app would be needed is auto-image-comparison to detect broken features, although having a library of "perfect" screenshots should allow a pixel-by-pixel comparison with a percentage grade given to the Dolphin screenshots. Smile (My only question is: where do we get the "perfect" screenies?)
System Specs:
[Image: 670320.png]
Find
Reply
« Next Oldest | Next Newest »


  • View a Printable Version
  • Subscribe to this thread
Forum Jump:


Users browsing this thread: 1 Guest(s)



Powered By MyBB | Theme by Fragma

Linear Mode
Threaded Mode