DorsalVR: VR interface with motion controls

[MichaelJW]

Hi folks, just uploaded a project I've been working on:

https://github.com/MichaelJW/DorsalVR

It's a Windows app that mirrors your desktop to your VR headset, and outputs your VR controllers' motion data using the DSU/Cemuhook protocol used by Dolphin. This means you can use your VR controllers as a Wii remote and Nunchuk for Dolphin, and see Dolphin's output on a massive virtual (optionally 3D) TV screen.

There are a few issues and restrictions, partly because this uses OpenXR which is still new and kind of buggy in places, partly because I've only been able to test this on my own setup so far. But once it's running, it works well, in my experience.

I'd love to get your feedback if you do try it out!

Big thanks to iwubcode, jordan-woyak, and everyone involved in this excellent article for sharing all their work around motion control which I have learned so much from

[oldgoose]

Hi, I've tried testing DorsalVR but I can't get past the load screen. It's giving an error of


Sorry, this display is unsupported...

Desktop duplication API does not support being run against the discrete GPU on a microsoft hybrid system. To work around this issue, run the application on the integrated gpu instead of the discrete gpu on a microsoft hybrid system

I'm running a GTX1070 with a reverb G2 on steamVR
Any ideas? thanks

[kurt91]

I don't have my headset available at the moment, so I can't test this myself. But, with the free-look camera mapped to the mouse, and using something like VorpX so head movement is mapped to mouse movement (meaning, I look to the right, and the computer sees it as "mouse moves to the right"), and using this to get the controllers working properly, would it match everything DolphinVR can do?

Considering we have a pretty good free-look camera system now, a handful of various ways to do 3D rendering, and there are open-source methods of programming for VR controllers, I'm kind of surprised that the official Dolphin version doesn't just throw something together for basic VR compatibility. I mean, for one thing, it would eliminate having to constantly remind people that DolphinVR isn't supported because there would no longer be any reason to use DolphinVR since the compatibility isn't as good as the normal current version of Dolphin.

Personally, I'd love to be able to only have one version of Dolphin so I can play Metroid Prime, Mario Kart & F-Zero, and Sonic Unleashed/Colors in VR. Surprisingly, Paper Mario TTYD works amazingly well in VR. It looks like a beautiful living diorama in front of you, things are textured that you normally would never be able to see, so you rarely see broken graphics even if you walk into the "diorama" and try to see things at angles you shouldn't, and the theater room used for battles is surprisingly fully-modeled with even a back wall behind you, so it truly feels like you're in the audience watching the fight. Unfortunately, DolphinVR is outdated as far as compatibility, and you have to wrestle with it to get it to play PM-TTYD properly, and even then it's slightly slowed-down.

[iwubcode]

(04-07-2021, 10:52 AM)kurt91 Wrote: Considering we have a pretty good free-look camera system now, a handful of various ways to do 3D rendering, and there are open-source methods of programming for VR controllers, I'm kind of surprised that the official Dolphin version doesn't just throw something together for basic VR compatibility. I mean, for one thing, it would eliminate having to constantly remind people that DolphinVR isn't supported because there would no longer be any reason to use DolphinVR since the compatibility isn't as good as the normal current version of Dolphin.

From a Free Look perspective, my end-goal isn't VR but just a very versatile camera system with lots of fun bells and whistles.

However, from what I understand from talking with Michael, VR is still a ways away from working well.  From a Dolphin perspective the current implementation works for 3DOF but is a bit nausea inducing due to drift.

No guarantee they will be merged but I have open merge requests to support 6DOF and also fix the drift (with some hacks on the DSU side).

Just curious but what did you have in mind if all the pieces are available?  I suppose we could eventually have a "VR" profile.

It's also worth noting there is still interest by one developer to support VR natively, so that may eventually come to pass.

[MichaelJW]

(04-06-2021, 04:22 PM)oldgoose Wrote: Hi, I've tried testing DorsalVR but I can't get past the load screen. It's giving an error of

Sorry, this display is unsupported...

Desktop duplication API does not support being run against the discrete GPU on a microsoft hybrid system. To work around this issue, run the application on the integrated gpu instead of the discrete gpu on a microsoft hybrid system

I'm running a GTX1070 with a reverb G2 on steamVR
Any ideas? thanks

I haven't come across this before, but from what I've read, this is an issue with the Desktop Duplication API (which I use to mirror the screen to the headset) - if you have both an integrated GPU and a discrete GPU, the VR app tries to run on the discrete GPU (your 1070) but mirror the primary screen which is probably running on the integrated GPU, and the API doesn't support that. (I think I have that the right way around. More here: https://www.freemancw.com/2020/11/desktop-duplication-on-hybrid-graphics-systems/)

I'm not sure how to fix this, but I have some ideas. First, do you know if your computer actually have two GPUs? If so, you can try changing your graphics settings to change which one is used for Dolphin and for DorsalVR to see if that helps: https://www.tenforums.com/tutorials/103965-set-preferred-gpu-apps-windows-10-a.html. (This guide also shows how to see which GPUs you have available.) DorsalVR is currently hard-coded to mirror the primary monitor; soon I will add an option to change that, but in the mean time, you could try plugging your primary monitor directly into the motherboard rather than into your 1070, to force it to use the integrated GPU.

Hope that helps. Please let me know how you get on, and thanks for letting me know about this. If there's something I can do within DorsalVR to fix this I will try to implement it.

(04-07-2021, 10:52 AM)kurt91 Wrote: I don't have my headset available at the moment, so I can't test this myself. But, with the free-look camera mapped to the mouse, and using something like VorpX so head movement is mapped to mouse movement (meaning, I look to the right, and the computer sees it as "mouse moves to the right"), and using this to get the controllers working properly, would it match everything DolphinVR can do?

I have tried out iwubcode's work-in-progress no-drift headtracking with a widescreen hack, stereoscopic 3D, and motion controls mapped to VR controllers, and it all works nicely and feels cool - but it's not quite the "geometric 3D" that DolphinVR and VorpX offer. What I mean is, the scene is still rendered onto a rectangle rather than being all around you, and so it doesn't feel like you're inside the scene; it's more like gyro-aiming in a 3DS game (albeit with a much bigger screen, and much more precise head tracking).

(DorsalVR does map the headset movement to the controller in slot 0, and you can use the

Code:

-selfie

command-line option to make the VR screen move with your head, so you can map the headset "controller" to Free Look and test this out, but as iwubcode says it does induce a little nausea at the moment because the tracking is not exact. If/when his drift fix is merged I will update DorsalVR to properly support that.)

[iwubcode]

(04-08-2021, 06:13 AM)MichaelJW Wrote: I have tried out iwubcode's work-in-progress no-drift headtracking with a widescreen hack, stereoscopic 3D, and motion controls mapped to VR controllers, and it all works nicely and feels cool - but it's not quite the "geometric 3D" that DolphinVR and VorpX offer. What I mean is, the scene is still rendered onto a rectangle rather than being all around you, and so it doesn't feel like you're inside the scene; it's more like gyro-aiming in a 3DS game (albeit with a much bigger screen, and much more precise head tracking).

Ah that's a shame. I knew you had a tv mode but thought there was a way to have the 3d scene mode as well. Not sure why I thought that, if I think about it technically it doesn't seem plausible.

Still very cool but I guess we'll need to have a proper VR solution after all. So much for my Free Look dreams of a simple solution!

I will say I'm a bit surprised. I would think that if you had a per eye image + 3d stereo data + 6dof headtracking, it would be hard to distinguish between that and proper VR. Maybe it is the per eye logic that adds to the realism? Then again, I've never experienced real vr.

[MichaelJW]

(04-08-2021, 08:00 AM)iwubcode Wrote: Ah that's a shame.  I knew you had a tv mode but thought there was a way to have the 3d scene mode as well.  Not sure why I thought that, if I think about it technically it doesn't seem plausible.

Still very cool but I guess we'll need to have a proper VR solution after all.  So much for my Free Look dreams of a simple solution!

I will say I'm a bit surprised.  I would think that if you had a per eye image + 3d stereo data + 6dof headtracking, it would be hard to distinguish between that and proper VR.  Maybe it is the per eye logic that adds to the realism?  Then again, I've never experienced real vr.

Your comment made me take another look, and it turns out I was completely misremembering the problem!

So - you're right - with stereo 3D and 6dof and a wide/tall FOV and a massive virtual screen, it does look very like proper VR. Maybe not quite indistinguishable (I haven't used it enough to tell yet), and it might need some fiddling with the depth and convergence settings, but it's certainly close. 

The actual problem is with the UI. Here's Mario Kart Double Dash with regular settings:



And with the FOV adjusted:



By "mounting" the virtual screen to the headset (so it's always directly in front of the user's eyes), and using your Free Look changes to move the in-game camera as the headset moves, and making the virtual screen so huge that it fills the FOV of the headset, it does feel like you're "inside" the view you had before. 

The view you get right in front of you isn't exactly the same as without the adjustments, as you can see in this image where I've layered one over the other:



... but I don't think that really matters. Or, at least, I think it can be worked around. It's hard to tell within the game, anyway.

The problem is actually with the UI, and any other "flat" elements; since these tend to be placed around the edge of the screen, you can only just see them out of the corner of your eye, and of course moving your head to look directly at them doesn't work. This is really inconvenient in a lot of games.

Do you know if there's a way to change the region of the screen in which the flat elements get rendered? When I was looking into this before I wondered if changing how the orthographic projection worked might be the key - like an option to say "when using orthographic projection, only render to the middle X% of the screen". I'm guessing that this wouldn't be a one-size-fits-all solution, though, and I gather that some games have more complicated layering logic.

[AnyOldName3]

Your attachments haven't linked properly. For some reason, I can only see them in the reply mode's thread review, and not when looking at the post normally.

[MichaelJW]

(04-09-2021, 06:01 AM)AnyOldName3 Wrote: Your attachments haven't linked properly. For some reason, I can only see them in the reply mode's thread review, and not when looking at the post normally.

Thanks - fixed.

[iwubcode]

(04-09-2021, 05:07 AM)MichaelJW Wrote: So - you're right - with stereo 3D and 6dof and a wide/tall FOV and a massive virtual screen, it does look very like proper VR. Maybe not quite indistinguishable (I haven't used it enough to tell yet), and it might need some fiddling with the depth and convergence settings, but it's certainly close. 

Thanks, glad to hear the idea was at least plausible.

(04-09-2021, 05:07 AM)MichaelJW Wrote: The problem is actually with the UI, and any other "flat" elements; since these tend to be placed around the edge of the screen, you can only just see them out of the corner of your eye, and of course moving your head to look directly at them doesn't work. This is really inconvenient in a lot of games.

Do you know if there's a way to change the region of the screen in which the flat elements get rendered? When I was looking into this before I wondered if changing how the orthographic projection worked might be the key - like an option to say "when using orthographic projection, only render to the middle X% of the screen". I'm guessing that this wouldn't be a one-size-fits-all solution, though, and I gather that some games have more complicated layering logic.

So, I am planning on looking into adding an orthographic projection matrix customization system. Assuming it is useful at all. I thought it might be fun for 2d games. It could be used to fix this specific game issue most likely.

However, 2d are problematic in general for Dolphin. Games do all sorts of crazy things.

For instance, the Last Story will draw a 3d scene, then it needs to do something so it draws over that scene in a small corner in the upper left. WHOOPS. So it then draws on top of that portion of the screen with just the portion that was missing. It does this by drawing a 2d segment.

Games like to do large screen spaced effects and they often do that in 2d. In VR that's odd. Should those stretch or should they be just in front of you? I wouldn't know what to do.

It doesn't mean this problem is impossible to solve but it is very difficult to solve generically (and one I've been looking into for other reasons). I have ideas but nothing I can say will work for sure at this time.

This is a bit of a downer but at the same time, I think as opposed to not having VR at all, it's ok (I say this as an outsider!). While most games have a HUD, there's probably an even split of the ones that need it and those that don't. Though menus are very important. Maybe you could have a toggle that takes you into "TV" mode that users would use for menus? Inconvenient but it's something.

Still, looking at my collection, what I wouldn't give to experience them in this way. Even if they weren't "playable".

--

Keep up the good work MichaelJW. I have more surprises for you soon