Netplay - Questions for Devs or Experts

[BlackLionPT]

Hi everyone.
I've been using Dolphin with a friend for quite some time now with Netplay, and it works like a charm. However, when 3 Players gets involved i's another story. We Are using GIT600 and also tested everything with GIT619.

I've created this topic not only to report the performance, but to see if somethings can be done or not in future development. I will try to be as much explicit and precise as i can with what i would like to know if such improvement can be or not be done, putting on BOLD the questions i would like to get answered if you don't mind.

# Compressing Data in order to allow more players with less bandwith

Let me start with a 2 Players game, Sonic Adventure 2 Battle. It works like a charm in dual core mode, but ONLY in racing stages. Hunting emerald stages requires to use Single Core in order to get synced. We can play perfectly fine with a PAD BUFFER of 3 for hours BTW. We can run it @ Full speed without an itch, even on Single Core.

Now, things get ugly when we add more players, specially on slower internets.

We decided to dust out or Game Cube copies of Sonic Riders, but to prevent possible errors, i grabbed mine and passed to my other friends (we DO have the original game, im not lying seriously!) in order to prevent desyncs and somethings. We did all according to book, save games and all.

Dolphin pinged us and said a minimum of 0.95 ~ 1.1 PAD for 60 FPS was required, so i inserted 3 (like always). The problem starts now.

Here are our internet connections Speeds and pings with each other (we are from Portugal, and unfortunately we don't live very close so online gaming is our best option):

Me:
Download: 2 Mbps
Upload: 864 Kbps
Ping to them: 24 ms

Friend 1:
Download: 10 Mbps
Upload: 1024 Kbps
Ping to me: 24 ms

Friend 2:
Download: 2 Mbps
Upload: 512 Kbps
Ping to me: 24 ms

When we tried to play the game, we noticed that the game simply would not stop "hanging", not "fatal hang" hang, but it stooped like it's waiting for the data bytes to arrive and then proceed. I changed the pad buffer to 20, 100 and 200 no help. My friend with the faster upload upload did the server this time, and still no Go. After closer looking, i found that dolphin needs +/- 20 KBps per controller.

Question 1: Now, there is one thing i can't understand on dolphin: While playing with 3 or 4 Players, how is the connection done?

Player 2 and 3 send the Data to me (server), and I send to them? Meaning, 60 KBps to them and 40 KBps income (20 from wach of them)? Or do Player 2 and 3 send their data to each other, and i send mine to them and receive theirs too? Meaning, 40 KBps upload and Download? in short, how does Dolphin really sync all 3 / 4 Players?

Question 2: Now, where i'm trying to get is: I think the data connection files are "too big" for the "nowadays" Internet Speed Connections. Is it possible to make them smaller using on the fly compression and decompression? Like, in order to archive, for example, per player upload 10 or less KBps?

This way it would be possible to play without crazy lag due to bandwidth problem, which if i'm not mistaken was being caused by the lower upload of my #2 Friend.

# Problems with Syncing COM Players In Game (possible Random Clock / RAM sync issue?).

Now, in Sonic Riders, while playing with 2 or 3 Players, for those who played the game know there are some COM Players in the game. I tested Playing Two Players with my Friend #1, using single core, Disabling Idle Skipping, and all the remaining settings the same, and we got a problem with Desynced COM Players. We have the same exact memory card btw.

Question 3: The problem is, when we get in game, the Com Players get desyned. Meaning, in my game they are for example, Shadow, Sonic, Rouge, and in His game they are Eggman, Jet and Sorm for example, this using Idle Skipping ON and / or Dual Core ON. If we use Single Core without Idle Skipping, we get the same characters In Game, but they behave differently. For example, during the start count down, my Jet goes forward but on his computer he stays on his place or does something differently. What could be causing this? Probably a random "key" in the game that is not being correctly synced?

We Also tried with Single Core - Idle Skipping Disabled, and activated the Per Game options "Enable MMU", "Enable BAT" and "Accurate VBeam Emulation", but the Com players would still react differently in each others computer.

What could be happening in Dolphins cycles in order for this to happen?

-----

Thank you very much for the time taken to read this post. Any answerers will be appreciated, and i hope my testing was usefull.
If you need any more data, just ask.

[BlackLionPT]

Should i post this in Dolphins Google Code instead? Or in the main future feature request thread here in the forum (since no one answered :/)?

[neobrain]

Just visit our IRC chan and poke Billiard about it, I'm sure he knows wtf we're currently doing and if what you're suggesting is possible.

[Billiard26]

1. Each frame, each client sends the states of all locally controlled pads to the server. The server relays those states to the other clients and sends any states it controls to all clients. Yes, the server could end up dealing with a lot of traffic.
This could be turned into a more P2P setup.

2. 8 bytes of input data + 2 bytes of message overhead + 4 bytes of SFML packet overhead * ~60 times a frame, for each pad. With 3 pads in use, 2 clients and 1 server, each with one pad, my calculations are 26.25 kbps upload + TCP overhead for the server.

The packet sizes could absolutely be reduced. SFML's packets are not needed, diffs could be sent instead of entire pad states, etc.
Compression is going to cause more input delay.
The protocol is not optimized at all.

3. Dolphin is not perfectly deterministic, any bit of inconsistency can completely throw off the synchronization in netplay. Errors in the netplay code are also possible, but because it works properly with certain games I do not think that is the case.

Because Dolphin is not yet perfectly deterministic (especially not in Dual Core mode) I do not think improving netplay at this time is worth it.

[BlackLionPT]

Hi Billiard26.
Thank you for the reply, i just wanted to know that Well, it works rather good for now anyway, thank you for the time to reply

[BlackLionPT]

Hi again. I'm sorry for the double post but, i have some developments about the desync issues (and also a question about a feature not implemented yet).

Well, lets start from the beginning. I Downloaded the latest revision of dolphin, just to make it easier and to prevent that some code change might affect it's behavior. Since my computer is capable of it, i fired up 4 dolphins (single core), each one on their folder, and went to play in 4 player mode (and also 2 player mode, to test the COM players) (Sonic Riders) in order to try and find the cause of the desync. This latest revision does not desync the com players do not desync, it might have been a bad one i cough the other time.

Now, the interesting thing: The desync is not cause by a random dolphin error, but, as far as i can tell, by an error on Netplay plugin on sending the PAD keys (or, but less probable, a problem with dolphin receiving the pad comands).

You see, let me see if i can clear you up a bit the situation: In Dolphin 1 (Player 1 controls window), i do every single thing normally,. and the other dolphins receive the keys. But, after some mashing, and some more "hard" gameplay, one of them, or 2 of them, or even 3 of them at once desync. I'm almost 100% sure its the Netplugin's fault because, well, I pressed boost on my screen, and the other 3 didnt' change. I had the game manually limited on all 3 dolphins at 45 FPS, with pad buffer 0 in this clear test (although the game can run at 60 FPS, but just to be sure i caped the framerate, i tested online caped at 30 FPS with a friend in 2 player mode, tested locally (on my computer too at 30 FPS, 45 and 60 FPS), all the times i got the same result:

One or more dolphins didn't received the key commands correctly.

Here, have a screen of the problem showing exactly what i just said (in this test i had locked the game at 45 FPS): http://imageshack.us/photo/my-images/819/20120529224455.jpg/

Further testing, showed me that there are even a worse problem with the Netplay Plugin: It seems it can not keep all 3 emulators synced properly with high pad buffers.

Now, to test this and affirm this, i tested playing on my own computer with pad buffer 20, the default value.

What happens is, instead of all 4 dolphins change position at once, with a pad 20 lag, they will get all crazy, and change each one at a different timing. This results in a very quickly desync across all 4 dolphins. This might be due to a problem in lag check confirmation, or lag input preservation i think thats what it's called.

Meaning, if i press left on my screen, i turn left, and then second screen turns left, and then 4th and then 2nd, for example (the order can be sorted) meaning, they won't change at the same time as it does in pad 0 - 6.

Here, have a pause screenshot to show +/- what im saying. After i unpaused the game kept running in sync for a while, but only for like, 20 ~ 30 seconds, then the delay in sending the keys just was too much and it desynced one of my screens (the 4th one in this picture, although it would seem like it would be the third one, but it was the forth that eventually desynced. Remember, PAD 20 was used on this test).

Link (the game was locked on AUTO for this test): http://imageshack.us/photo/my-images/51/20120529225005.jpg/

And just this extra one: As you can see, all 3 dolphins didn't receive one keymap, meaning, all of them desynced in the same place, but kept going in sync: http://imageshack.us/photo/my-images/528/20120529225838.jpg/

Thats why i believe there is something wrong with the PAD packets delivery.

Now, just to finish up! I would like to ask you about the unimplemented feature "pad switching on-the-fly". Did this feature had not been implemented because of some incompatibility with dolphins code? I tested SA2B online with 2 friends, being me the spectator and they playing, and the game went smooth without any major desyncs (well, occasionally it would desync, probably because of what i said above. We used Pad3 Btw). Could this be implemented, if you have a bit of spare time?

I Hope this testing helps with anything regarding Netplay. At least, i think in single core, dolphin is pretty much accurate, being the only problem here the Key input delivery

PS: Yes, i did change the controls in netplay server part, meaning, Dolphin 2 used PAD2 mappings, Dolphin 3 PAD3 and Dolphin 4 PAD4, they did not use all the player one controller (Pad1 by default).

Again, thank you for your time, and i hope this can help with anything

EDIT: minor language corrections.

[skid]

That's interesting, Palito. Games do not necessarily have to read the pad once every frame. Some games read the pad multiple times per frame. This kinda lines up with what you are seeing. Do you think it could also be something to do with the framelimiter/throttle making the game desync?

[Billiard26]

It is not impossible that Dolphin's netplay is broken.
Perhaps it is only broken when trying to synchronize 3 Dolphins. Is that what your testing has shown?

"Further testing, showed me that there are even a worse problem with the Netplay Plugin: It seems it can not keep all 3 emulators synced properly with high pad buffers."

Just be aware that netplay means to synchronize inputs in game-time, not real-time. One simulation may appear to have a button pressed before others even when synchronized. The higher pad buffer would allow more drift between game-time and real-time.

[BlackLionPT]

Well, i will try to do my best since i don't really know how dolphin really manages the netplay.

Here, i did a quick video (and a bigger one that shows it better), i will upload a bigger one showing the differences between the different pads. In this video, the pad is 20. As you can see, one of the 3 dolphins clearly desyinced. They were all running single core of course, at full speed (dual core, even with pad 0 desyncs. Dolphin can not sync the random events correctly with dual core, making every single aspect of the gameplay that needs a random value generated cause an immediate desync). Now, this doesn't always happen on the same dolphin! It can happen in all 3 of them, obtaining 3 different results. It's very hard to get ingame synced with pad buffer 20, usually desyncs right on Track load. With pad 0 ~ 6, it usually get always inside perfectly. The more the pad, the more the chances of not syncing.

I mean, even single core needs to have a perfect timing per frame on the PAD. Although dolphin is very precise with the random events, a small delay can be enough to trow the game out of sync. If all dolphins connected don't do the same exact operation at the same time, it has a very big chance of desync.

The problem is that the pads are being delivered too late to maintain sync for some reason. Here, watch the video below:

Small Video Link (sorry for the bad quality, cellphone cam used): http://www.youtube.com/watch?v=Nwv_TbNZwZY&feature=youtu.be

Bigger Video With differences in the results with different Pad Buffers: http://youtu.be/ZKYrF9ukYkI
[NOTES: In this full video, you can clearly see the random issue with big pad buffers. First, the emulator 1 and 2 where synced +/-, but the third one isnt (at 3:23), then, at 4:10, you can see that the first one got out opf sync with the other 2. This happens too during game start (no pad bufffer change during game play), as the quick video shows, so, i'm sure that is not the issue.]

Now, notice this! The desync occurs independently of the number of dolphins in game! It can be 2, 3 or 4, the results are always the same!

The big problem here is that dolphin can't provide a true 20 pad buffer on the fly. The emulators should all react at the same exact time, even if the pad was 60 (1 second of reaction time, assuming the game is 60 FPS). But they aren't. This can be due to a bug in the packet travel time calculator (ping as usually is called), that is not working properly.

Now, i think this can be prevented making dolphin sync every frame in every machine. The throttling netplay uses is already working pretty good, it can maintain sync for a very long time, but something isn't right with the way it does it. There is one way, but i think that it would require a lot of bandwidth, and it would probably don't work due to the lag hit it would take on the gameplay.

Imagine, if the dolphin in Machine 2 can not process all the frames in time, the other machine (1) should slowdown it's frames per second too so it could only process the same frames as machine 2 did (i don't know how this can be done more accurately but, maybe sending the time per frame, and compare which machine was faster and then slow down the fastest machine? Then, execute the needed pad operations). Now, this already happens as you know it, but, they can not sync properly with ping fluctuations. See, a 6 ms fluctuation in ping (from 30 to 36 ms for example), can be enough to desync the games in the emulator. I think there should be a way to maintain the real ping in real time always entering the equation, but it would probably cause too much overhead. I'm certain this would make all emulators be in perfect sync, but i just don't know if this is A) Possible to do / B) If possible, Wouldn't cause a very hit on the bandwidth.

The problem in netplay is clearly the way it keeps in sync the machines. Dolphin, although it's still incomplete and can not emulate 100% accurately, has a very, very high accuracy in processing the same game in different computers, meaning, the results will be the same as long as everything happens at the same exact time.

About Dual Core Desync Problem: I don't know if this is already done or tried but, if there was a way to force the random values that need to be generated be always the same, i think dual core would benefit from it, making dual core synced across all machines during netplay. Games like Sonic Adventure 2 Battle can keep sync in dual core if no random events are activated, for example, the Race Mode, it will remain in sync until a pad isn't sent / received / executed correctly in one machine, or a random event occurs. Only Race is playable like this (disabling auto handicap keeps it sync), the rest of the modes simply desync on level start.

So, in summary yes, i think the problem is in the way the frame limiter/throttle of netplay is working that haves the problem, and not only that, the way the data is sent to each computer that lacks time accuracy, which is the big bomb that causes desync.

Sorry if i was a bit confusing, sometimes i just don't know how to explain things properly

----
EDIT:
Oh i'm sorry, i only noticed now that Billiard26 answered too. Yes, i'm aware that the sync is In Game time not in Real time (it's very noticeable when you lock the game to 30 FPS, the upload / download usage is cut in half). Also, no, like i said above, this type of desyncs occurs with 2 dolhpins, 3 dolphins or 4 dolphins. it's very unpredictable, but one thing i know it's affecting the sync: The higher the pad buffer, the faster the game desyncs. Even with pad Buffer 3, after like 30 minutes of gameplay, the game desyncs due to some control not read in time (or received) by one of the dolphins (it doesn't have to be the server, it can be the server or the client).

[Billiard26]

Have you tried DSP LLE?