Desync Prevention Idea

[Lobsterzelda]

Hello everybody!

I am writing here on this forum to get some feedback/advice on an idea that I have.

My goal is to create a branch of Dolphin with more accurate/complete savestates, which will hopefully prevent a specific type of desync, which I will describe below:

1. You TAS a game (recording your inputs in a dtm file) from power on up to frame 100, and then make a savestate.
2. You advance to frame 105, then load a save state to go back to frame 100.
3. From there, you keep TASing to frame 200, and then you save the resulting dtm file that this created.
4. When you try to play your TAS back from power on, it desyncs after frame 100! Creating a TAS from power on up to frame 200 without making or loading a single save state, however, results in a TAS that will sync from power on.

After seeing this type of desync, my working theory of why this can happen is that certain information is either not stored when you make a save state, or not written back out when you load a save state (likely both are happening, since it's unlikely that a dev altered save states to save certain info when saving but not to load that info back when loading a save state).

From all of this, there are a few potential culprits which could be not getting fully saved and creating the problem:

1. RAM
2. Registers
3. System Clock
4. Basic Properties, like idle skip on (though this is unlikely, since it's stored in the dtm file specifically)
5. Some other type of memory.
6. Some graphics or audio being different, and this causing something different to be written to memory (though I don't know how/if this could happen?)


I've started looking through the State.h and State.cpp files to figure out how save states work in Dolphin, although I can't quite figure out where a key function is. State.h is located in (from the root of the dolphin project) Source/Core/Core. saving a save state appears to occur in the doState(PointerWrap& P) function (which also handles loading save states). One of the function calls in this class seems to handle this functionality, but I'm not quite sure where. I think it's Movie:: DoState(PointerWrap). However, Since PointerWrap is an external dependency, I'm not sure which class it would be when Dolphin is actually running.

If anyone has any experience with the save state code on Dolphin or has any ideas for how I could proceed from here, I would greatly appreciate it!

Many thanks!

[JosJuice]

Here's PointerWrap: https://github.com/dolphin-emu/dolphin/blob/5513d5f4f732fb1e436765ab87e7d60ba02b1ad6/Source/Core/Common/ChunkFile.h

Just so you know, trying to find the cause of a desync like this is a very hard task. But if you do manage to find it I think a lot of people will be happy. Feel free to ask more questions if there are any specifics of the codebase that you're wondering about, but I'm afraid I don't have any good overarching strategy for trying to find desyncs.

[Lobsterzelda]

(04-22-2021, 05:31 PM)JosJuice Wrote: Here's PointerWrap: https://github.com/dolphin-emu/dolphin/blob/5513d5f4f732fb1e436765ab87e7d60ba02b1ad6/Source/Core/Common/ChunkFile.h

Just so you know, trying to find the cause of a desync like this is a very hard task. But if you do manage to find it I think a lot of people will be happy. Feel free to ask more questions if there are any specifics of the codebase that you're wondering about, but I'm afraid I don't have any good overarching strategy for trying to find desyncs.

I have an idea for how I could potentially find the cause, but it's contingent on Dolphin having certain functions to dump various memory and settings values to a text file.
My plan is:

1. I would create a TAS of Star Fox Adventures from power on, and play up to a certain point (making sure that no desyncs would occur). 
2. I would play the TAS back up to a certain frame, create a save state, and close out Dolphin. I would choose this state so that when I play the TAS back starting from this save state, a desync would occur (in this game, if you make a save state outside of a cutscene, close out dolphin, and reload the state, it usually causes a desync).
3.  I would then play back my original TAS from power-on up to the frame that I created the save state on.  From here, I would run a script which would log all memory addresses and settings for every frame to a text file.
4. I would then play back my TAS from the save state, and run the script to log all memory addresses and settings for every frame to a different text file.
5. I would use a program to find where the first differences occur between the two files. More specifically, I want to see when a value in RAM first becomes different between the two games on the same frame, and what this memory address is.
6. From here, I would advance to that frame in each respective condition (from power-on and from the save state that causes the desync), and run a script which would step through each PowerPC instruction step-by-step, looking for the moment when that memory address is first set to a different value between the two situations.

From all of this, I would hopefully be able to figure out where the non-determinism is being introduced, which would allow me to know what I have to change about the info stored in Dolphin save states in order to prevent desyncs.

[JosJuice]

If you turn on the debugging interface at Config > Interface, then the Memory tab will let you dump the contents of RAM. There's no convenient way to dump the contents of the various hardware registers and such, though.

[Lobsterzelda]

(04-23-2021, 12:38 AM)JosJuice Wrote: If you turn on the debugging interface at Config > Interface, then the Memory tab will let you dump the contents of RAM. There's no convenient way to dump the contents of the various hardware registers and such, though.

Is there any way to access them?

If not, then I guess I could make a debug dolphin branch that will allow me to have easy access to them.

[JosJuice]

(04-23-2021, 01:07 AM)Lobsterzelda Wrote: Is there any way to access them?

You'd have to write custom code for it. Please keep in mind that these are spread out throughout the codebase, so if you want to systematically dump them all you have a lot of work ahead of you. (The easiest way to find them is to look through the savestate code, but since you're suspecting that there's something the savestates aren't saving, this isn't something you can rely on if you're trying to find that one thing.)

[Lobsterzelda]

You know, on second thought, I just thought of a better and easier way to find out what's causing desyncs.

I'll create the initial TASes described above. Then, i'll just only dump the RAM memory of the game, and ill dump from each frame that the savestate was made on until the desync occurs.

Ostensibly, since savestates store all RAM values, on the exact frame that the savestate was made on, the RAM should be an exact match between the two TASes.

If the RAM never got out of sync between the 2 TASes, then it wouldn't matter if system clock or whatever is causing the issue wasn't saved (since the TASes would be indistinguishable if the game state is identical in both of them at every point in time). Of course, the external factor has to eventually impact RAM, otherwise the desync wouldn't happen.

Once I find the exact desync frame (which will likely be the 1st frame after the save state was made), then I plan to do something which assumes that Dolphin has a way of logging what code/instructions were executed. Assuming this exists (which I figure it must, otherwise debugging Dolphin would be basically impossible), then I will use a sort of binary search strategy here.

To explain this, suppose that 1 visual frame involves a sequence of 300,000 instructions being executed in a row. I know that at the 0th instruction, everything is synced, and at the 300,000th instruction, I am at or past the desync point. Now, I play the TAS back to this frame, and then play through 150,000 instructions in each TAS. I now check to see if a RAM desync has happened. If it has, then I am at or past the desync point, and 150,000 is the new upper bound of my search. If not, then I am before the desync point, so I set 150,000 as the new minimum bound. This process then repeats, dividing the number of instructions left to search in half each time. Eventually, I will hit the exact instruction that causes the first change in RAM between the two TASes. At this point, I will output the instruction that came right before this, the desync instruction itself, and the instruction right after this.

I am hoping that after I do this, I will find some variable that needs to be added to Dolphin save states to prevent desyncs.

After this, I will try the entire TASing process described above again. If I find another desync, then I will repeat the above process to figure out what other variables need to be saved, and so on. I will then repeat this until I am unable to make a desync occur, no matter when I savestate.

---------

Whew! That was a mouthful! So, what do you guys think of my idea? (and also, how, assuming it's possible, do you dump instructions executed in a GameCube game in Dolphin to a text file?)

[JosJuice]

I suppose that should work, but I think you're on your own here with writing code to log the instructions executed.