Netplay Lag spikes on game start-up

[Riceykins]

I've seen that you guys don't support the 4.0-648/4.0-652 DC netplay branch but I figured I'd ask anyway in case it isn't related to that specifically.

Whenever I connect to someone through netplay I have a normal ping until the game itself is started, then it will either run perfectly fine with no lag spikes at all during the entire session or it will spike massively at the start and never stop. It also doesn't seem to be related to where a person is and when I restart dolphin sometimes it works after spiking with the same person.
I have no clue what would cause this and through my testing all I have left to guess is something weird going on with my ISP. Any suggestions?

[JMC47]

It's likely a problem with your ISP. While we don't support 4.0-648/652; I usually will field questions about it because I know it's so popular, and I had a hand in that.

Maybe it's the fact that it uses random ports, as well? It's a strange issue I haven't run into, that's why I'm not willing to blame the emulator right off.