Turning off Skip EFB Access from CPU causes massive lag spikes

[Aplayer12345]

As the title says, I've been experiencing massive stutters when disabling Skip EFB Access from CPU in games like Pikmin on Wii and Super Mario Sunshine. Both are basically unplayable because of this. I've attached a video demonstrating what's going on in Pikmin, both with most enhancements turned off and with most of the enhancements I normally use when playing (minus Per-Pixel Lighting). I didn't show it in the video, but disabling Ubershaders doesn't do anything. I've turned on Rivatuner statistics to show that my PC is not being stressed in any way.
Video demonstration

Worth noting, the issue seems to go away when turning off Dual Core in the settings, but I consider that a band-aid solution as it leaves a bunch of performance on the table.

PC Specs:
OS: Windows 10 x64 22H2
Motherboard: MSI B450 GAMING PLUS MAX
RAM: Corsair Vengeance LPX 2x8 16 GB 3200 MHz
CPU: AMD Ryzen 7 3700X
GPU: GIGABYTE NVIDIA GeForce GTX 1060 6 GB G1 Gaming OC

Not sure if this could be a Ryzen-related problem, but if it is, I also want to mention that I installed the latest AMD Chipset drivers (5.05.16.529) and I turned on the AMD Balanced power plan.

I'm running Dolphin 5.0-19708.