Important Desync Debugging Progress

[Lobsterzelda]

Hey everyone.

I'm making another thread to announce that I have found a way to track down where changes get introduced into memory at the instruction level. Based on this, I discovered that one of the 24 values that was desyncing at the end of the first frame that a desync occurred on first became out of sync at the 785th instruction after the start of the frame (where the first instruction is numbered 1). More specifically, the value at Address 0X3764804 becomes different on this instruction (this is the value stored in r5 when the write to the memory address stored in r5 occurs on the previous instruction).

I then generated log files for the first 1000 instructions for each TAS (both the syncing one and the desyncing one on this frame). As it turns out, both TASes execute the same instructions in the same order for these first 1000 instructions, so the change isn't introduced due to one TAS branching to a different location in the code.

Since the RAM and all register states are set to an exact match of the syncing TAS at the start of this frame (right before instruction 0), the only way the 2 TASes could get out of sync is if a value from outside of the game's main RAM was loaded into a register in a certain instruction. This change could then propagate downwards, up until the point where the contents of r0 are written to 0X3764804.

I am not very familiar with what to look for with respect to which instructions could access variables/info from outside of RAM, so I was hoping that a more experienced Dolphin developer could help me out.

Below, I have included the debug file for the first 1000 lines of code being executed. Each line says the instruction number, the PC value, and the actual instruction which was executed. Since the desyncing TAS and the syncing TAS produced identical log files for these 1000 instructions, I have only included one file here.

If anyone could help me out or give me any advice, I would greatly appreciate it!

If all goes well, then within a day or two, I should have some idea how to eliminate this specific kind of desync from Dolphin!

First 1000 Lines: https://pastebin.com/yvwQRb0Z

[JosJuice]

If you know that the RAM and the registers are the same, then the issue is probably in some MMIO. (But do note that there's a bunch of system registers other than the normal GPRs and FPRs - those should be checked as well if you haven't checked them. But I believe they're all savestated.)

To check what MMIOs the game is accessing, we can log all the memory accesses the game makes across these 1000 instructions. Run the game with the interpreter, and add a line of code in this function (outside the ifdef) that prints the address that's being accessed: https://github.com/dolphin-emu/dolphin/blob/5.0/Source/Core/Core/PowerPC/MMU.cpp#L450

[Lobsterzelda]

(05-04-2021, 05:06 PM)JosJuice Wrote: If you know that the RAM and the registers are the same, then the issue is probably in some MMIO. (But do note that there's a bunch of system registers other than the normal GPRs and FPRs - those should be checked as well if you haven't checked them. But I believe they're all savestated.)

To check what MMIOs the game is accessing, we can log all the memory accesses the game makes across these 1000 instructions. Run the game with the interpreter, and add a line of code in this function (outside the ifdef) that prints the address that's being accessed: https://github.com/dolphin-emu/dolphin/blob/5.0/Source/Core/Core/PowerPC/MMU.cpp#L450

I took your advice, and decided to do this check. Also, I did a more thorough check, and saw that while the first RAM desync occurs on instruction 785, on instruction 784, register 0 takes on a different value than it does in the other TAS (which is the first register difference).

I then enabled memcheck, and saw that between instructions 783 and 784 only one memory access is performed: Address #C84E82F4. This is a read of the address, but I can't help but wonder if this is what causes the register to get modified (which in turn is what modifies RAM, since the next instruction is writing the value in register 0 out to RAM)

Could this be the special address that holds the key?

[Lobsterzelda]

So, I did some digging around, and found out that this address is somewhere inside the EFB for Dolphin (also known as the entity frame buffer, which can be found in FrameBufferManager.h).

This struct never has its state saved when a save state is created!

As such, this could very well be what's causing this desync!

[JosJuice]

In newer versions of Dolphin, I believe this is fixed if you have the setting "Save Texture Cache to State" enabled. Could you confirm if it is?

[Lobsterzelda]

(05-05-2021, 05:21 PM)JosJuice Wrote: In newer versions of Dolphin, I believe this is fixed if you have the setting "Save Texture Cache to State" enabled. Could you confirm if it is?

Yeah, the newest version of Dolphin does save the entity frame buffer. This change was added in 2018 or so.

For me, my next step now is to add in the code to save state the EFB in Dolphin 5.0 Stable. After that, I can test out if my change was able to prevent desyncs from occurring in this game (and if not, I will search for the next issue).

[Lobsterzelda]

Btw, do you guys think that not savestating the various buffer pointers inside of VertexMangerBase could also be causing desync issues? Or are those variables which can't affect a TAS syncing?