Viewtiful Joe Real XFB slows down gameplay even when in full speed

[DolphinPC]

I can stand the graphics, but unfortunately, like Metroid Prime for example, enabling XFB (BOTH real and virtual) slows down the game. I mean I'm still in full speed, but the animation seem heavier and not as smooth and the response time is slower. The point is that for Metroid Prime you don't need XFB, but for Viewtiful Joe you need only for the videos, and for the gameplay it's better to disable (for graphics, and most important smooth gameplay).

Is there any way to keep the real XFB enabled only for the videos and disabled for gameplay (cause XFB is only being used during pre-rendered videos).

Edit: Maybe it's possible to create a patch or something?