You neglected to say what version of dolphin you're using, but, thankfully you filled out your profile.
Use the latest dev builds, there was a fix to audio latency down the line that will quell your issues.
Use the latest dev builds, there was a fix to audio latency down the line that will quell your issues.