Okay, my findings:
1) Saving and loading the state does NOT fix the problem
2) Doing a save state, shutting down the emulator, and loading that state doesn't fix it either
3) This happens with the LLE interpreter
4) This would probably happen with any game with any DSP plugin
5) The audio gets messed up after the DSP plugin pushes the audio to the mixer
I had Dolphin dump the audio as it was coming out of the DSP plugin, and at it's normal spot. There was absolutely nothing wrong with the audio as it came from the DSP, but there was the issue in the final output.
Image of the issue on a sine wave:
Confirmed, happens in LLE interpreter. Ninja'd dammit. Ran Brawl in LLE Interpreter with DSP on Thread disabled for the 90 minutes (and fullspeed, my computer rocks). Started popping around the 92 minute mark, and then was REALLY popping by 95 minutes. By 100 minutes it was total garbage.
Btw, you should let it keep going after the 90 minute mark. It does some funky stuff. Like an echo almost... Maybe I've just been listening to static too long. lol
This fixes the audio in the audio dump, but not what the emulator plays.
Here is a single patch that should apply to current master. Hopefully that should narrow it down a lot.
I know it's already confirmed but I can also confirm it does it in the LLE interpreter. I had to wait longer, it went halfspeed for me

Wow, thanks moosehunter, that's incredibly helpful! What this means is the issue is:
1. definitely in the AudioCommon code (most likely the Mixer).
2. not audio corruption, just some samples being read twice in a row - could be indexR and indexW desyncing, though I don't see why this would happen.
Could anyone try reproducing the issue with this build:
http://dl.dolphin-emu.org/wips/delroth-dolphin-master-3.5-1575+-x64.7z
It is patched to bound indexR/indexW in the Mixer instead of always incrementing until overflow.
I tested 1575+. No change. Also, you can mark off "Is the timestamp at which the issue starts happening always the same for one particular game". 92 minutes. I used Brawl on the Lylat level, and even though it didn't exactly cut out on any specific second, each time was the same. It starts at 92 minutes, then gets worse and worse till it just is godawful about 5 minutes later.
I think this issue appears because of our audio output buffer (Mixer.cpp):
We do accumulate the samples in this buffer in
here. We also store our read and write pointers in the next two lines. As this is redunand, they could differ because of math accurateness, specially in our
interpolation. There you'll find our ratio variable, which usually should be 32k/48k * 2**16 = 43690.66, which isn't storeable as int.
I've tried to fix this buffer handling in a
wip build. Interpolation still isn't fixed (so still converting to 48000.7Hz), but the 1h30 but should be fixed.
Known issues:
- 1.8E-6 semitone deeper sound (also on master)
- maybe small crack every 3 minute after 210min gameplay (_only_ if you run the game for 210min at 100% _all_ the time) as the buffer gets fuller over time
Thanks degasus! I will be trying this out over the next few days. So we finally find a penalty to having a PC that is too fast eh?...Kidding :p