This is how I created a macOS 10.5.6 VM on my 2020 6 Mac mini running macOS 10.5.6 and Fusion Pro 11.5.5. I was also successful with using this process on my 2016 MacBook Pro running macOS 10.5.6 and Fusion Pro 11.5.5.
Get “Install macOS Catalina” app from the App Store
Create a new VM and select “Install from disc or image”
Drag the “Install macOS Catalina” app onto “Install from disc or image”
Continue with VM creation
I used “Customize Settings” to increase CPU and RAM before first boot
Run the macOS installation as normal
After successfully booting and logging in install VMware Tools