For years I’ve had a dream of building a rack mounted PC capable of splitting its resources to host multiple GPU intensive VMs:
- a few gaming VMs
- a VM for work that can run Davinci Resolve and Blender renders
- an LLM server
- a Stable Diffusion server
- media server
Just to name a few possibilities…
Everytime I’ve looked into it, it seemed like the technology just wasn’t there yet. I remember a few years ago Linus TT took a shot at it, but in the end suggested the technology (for non-commercial entities) just wasn’t in a comfortable spot yet.
So how far off are we? Obviously AI focused companies seem to make it work, but what possibilities exist for us self-hosters who might also want to run multiple displays in addition to the web gui LLM servers? And without forking out crazy money for GPU virtualization software licenses?
The technology has “been there” for a while, it’s trivial do setup what you’re asking for, the issue is that games have anti cheat engines that will get triggered by the virtualization and ban you.
Which games do that? Running pasthrough gpu on windows for destiny and halo at least gave me 0 issues for years
Anything using vanguard such as valorant and league of legends, battleye such as pubg, destiny 2, and rainbow 6 siege, and easy anti cheat such as fortnight blocks virtual machines. Vanguard is especially bad because it will not allow to run the game with Intel-VT/AMD-V enabled even if you are running bare metal as of its last update.
I’ve been doing exactly that at home for a couple years now. First with Parsec, now Sunshine/Moonlight.
Host is Proxmox on Ryzen 5800x, 64gm RAM GPU is 2070 Super, with VGPU patched drivers from https://gitlab.com/polloloco/vgpu-proxmox
When I’m gaming I’ll dedicate the full 8Gb to my windows Vm, otherwise I split it in 2 or 4Gb chunks to Jellyfin or my home camera monitoring. 8gb can’t split very many ways, and most things require at least 2 to run.
Locally at home I can run 1440p 60fps rock solid over wifi on any device, from my phone/old laptop/apple tv/raspberry pi. Remote I can do 1080p60, but a bit more hit or miss depending on my network connection.
Experimenting with LLMs I’ve done through the same windows VM, or to a ubuntu dev VM. Works the same way. I’m thinking of transitioning my gaming VM to Linux too.
The amount of VRAM is the hard limitation to get past, the virtualization tech itself has been there for a while.
But to be perfectly honest……it really was just a “let’s see if I could do this” type task, direct GPU pass though is more straightforward and it’s not really worth splitting 8Gb these days. Unless you get a card with significantly more VRAM passthrough is much less work.
This is really amazing! In theory, can you can use 2gb with 4 different VMs?
Sure, but you’ll get diminishing returns most likely as consumer hardware doesn’t really have the resources to scale that way very well if all the VMs are running demanding apps simultaneously.
Even for something like 4 VMs that just do NVenc, there are limits for how many streams the GPU can do. I think there’s another patch that lets you raise that, but at some point you’ll run out of resources quick. Even powerful consumer gear isn’t really designed to be used by more than one user/app and it starts to show the more you virtualize and split those resources.
Hmm. I’m running a 3090 and 4090. Looks like vgpu is not possible yet for those cards.
How does the vGPU compare to running it on the bare metal? Last I tried things were painful but technically usable.
I don’t see any performance differences with the vgpu actually. I have more performance bottlenecks with the CPU, and my RAM isn’t the fastest, so I think I’m more CPU limited. Benchmarks I have run that are GPU focused seem to show little to no difference from what the physical card would do.
Have you tried or do you have any knowledge about utilizing the display ports on the gpu while virtualizing either in lieu or in tandem with streaming displays?
I bought a cheap used Dell R710 on Facebook marketplace for like $100 or so, as well as an ups, rack, 10g switch, etc, from various other sellers. All told, I’ve got about $500 in my server setup.
Installed proxmox on it. It’s “free” if you don’t buy a license. You just have to put up with a little nag screen when you open the control panel but it still works 100%, much like winrar.
Works great.
Edit: just realized this is in c/selfhosted AND I misunderstood the post. I’m gonna leave it here just on the off chance it’s useful to somebody, but I acknowledge it’s not what you’re looking for.
I run a few servers myself with proxmox. FYI there is a script that removes that nag screen as well as configures some other useful things for proxmox self-hosters.
I’ve also wanted to do this for a while, but there were always a few too many barriers to actually spin up the project. Here’s just a brain dump of things I’ve seen recently.
vGPUs continue to be behind a license. But there is now vgpu_unlock.
L1T just showed off PCIe “fabric” from Liqid that can switch physical devices between machines.
Turning VMs on and off isn’t as slick as either of the above, but that is doable today. You’ll just have to build all the switching automation yourself. That could just be a shell script running QEMU/libvirt commands, at a minimum.
Why are vGPUs behind a license? They work fine on Linux as they are part of KVM and Virtio.
Why? Product segmentation I suppose. Last I looked, the Virtio project’s efforts were still work-in-progress. The Arch wiki article corroborates that today. Inconsistent behavior across brands and product lines.
Here is an alternative Piped link(s):
L1T just showed off PCIe “fabric” from Liqid that can switch physical devices between machines.
Piped is a privacy-respecting open-source alternative frontend to YouTube.
I’m open-source; check me out at GitHub.
I know you asked about VMs, but fwiw there are GPU-capable containers now: https://docs.nvidia.com/datacenter/cloud-native/container-toolkit/latest/install-guide.html
Used one of these and the setup is as easy as it sounds. It can run Houdini, Stable Diffusion.
