You might want to try render to main and borderless fullscreen. If OBS has problems with closing emulation and starting a new game, it might be because of the different windows. WIth render to main Dolphin should display everything in one window. Exclusive fullscreen(borderless fullscreen off) might output the image in a way that is hard to record for OBS, which is why i'd try borderless fullscreen.
Last but not least, try the different video backends and see if one of them works better with OBS than the others.
Last but not least, try the different video backends and see if one of them works better with OBS than the others.