If it doesn't work in any OS newer than XP, then it's likely using an ancient version of D3D. VMWare's support is designed to primarily give the guest OS access to the versions of D3D/OGL that the host OS actually supports. If your software can't map its D3D calls to D3D calls for a newer version by itself, then expecting VMWare to do that and get everything right is a little bit of a stretch. It'll work better when using D3D9 software on a system that supports D3D9 as there's a 1:1 function mapping. If it's trying to run D3D7 software, it has to pretend that functions not in newer versions of D3D can be emulated with ones that are, which isn't necessarily the case (for example, using a function that takes 16-bit floats as an argument and returns 16-bit floats won't always get the same results as converting everything to 32-bit floats, calculating it with a newer function and then converting back again).
OS: Windows 10 64 bit Professional
CPU: AMD Ryzen 5900X
RAM: 16GB
GPU: Radeon Vega 56
CPU: AMD Ryzen 5900X
RAM: 16GB
GPU: Radeon Vega 56
