Multi-OS gaming w/o dual-booting: Excelent graphics performance in a VM with VGA passthrough

Note: This articles is a technology/technique outline, not a detailed guide and not a how-to. It explains what is VGA passthrough, why you might be interested in it, and where to start.

Even with the current abundance of Linux native games (both indies and AAAs), with WINE reliably running almost any not-so-new software, many gamers who use Linux on a daily basis tend to switch to Windows for playing games. Regardless of one’s attitude towards non-free software, it has to be admitted that if you wish to try out some of the newest titles, you have no other choice than running them on a Windows installation. This is why so many gamers dual-boot: having installed two operating systems on the same machine and using Windows for playing games and Linux for virtually anything else, they limit their usage of Microsoft’s OS for gaming only. This popular technique seems handy – you get the luxury of using a Linux, and the gaming performance of Windows.

But dual-booting is annoying because of the need of reboot to switch your context. Need to IM your friend while playing? Save your game, shut down Windows, reboot to Linux, launch IM, reboot to Windows, load your game. Switching takes a long time, is inconvenient, and therefore the player may feel discouraged to do so.

What if you could run both operating systems at once? That’s nothing new, run a virtual machine in your Linux, install Windows within it, and voilà! But a virtual machine is no good for gaming, the performance will be utter cr terrible. Playing chess might work, but any 3D graphics won’t do because of the lack of hardware acceleration. The VM emulates a simple graphics adapter to display it’s output in a window of the host OS.

And that is where VGA passthrough comes in, and solves this issue.

1. The idea

The key to getting neat graphics in a VM is to grant the virtual machine a full access to your graphics card. This means that your host OS will not touch this piece of hardware at all, and the guest OS will be able to use it as any other (emulated) hardware. The guest OS (presumably Windows) will load it’s own drivers for the graphics adapter, and will communicate with it natively! Therefore it will have full access to hardware acceleration and any other goodies that gear might provide (eg. HDMI audio). The idea of passing a VGA adapter to a virtual machine is usually named VGA passthrough.

Sounds crazy? Let me tease you: my setup is capable of smoothly running Watch_Dogs, Tomb Raider (2013) on Ultra settings at 60+ FPS within that virtual machine, using NVIDIA’s GTX 770. And I get the luxury of running both OS at once – so I can switch between them in just a glimpse, without shutting down either one! This is astonishingly convenient.

Because the dedicated graphics hardware will be reserved for the guest system, the host will need another graphics adapter to display anything. So there comes the first hardware requirement: you need at least two graphic adapters. However, it is not uncommon – many new Intel processors have a build-in GMA – and if you are a gamer, chances are you have invested in a dedicated graphics card – so that makes two graphics adapters already. Let the host system use integrated graphics, and the guest will get the powerful dedicated graphics for games. Because both graphic adapters will work independently and there is no way to compose their video output¹, you will need two separate displays, one for each system. This means either a set of two monitors, or a monitor with two video inputs (so that you can switch between them). You might also experiment with a KVM switch.

Also keep in mind that it is not an easy thing to set up. While some claim they have succeeded on their first try, many others have struggled a lot. Personally, I spend about two weeks tuning things up to get my VGA passthrough running – and if we count hardware searching and preparations, then it took me two months. But it was completely worth it! My current setup contains of:

• Intel i7-4790K (4 x 2 x 4.0GHz)
• ASRock Z97 Extreme6
• NVIDIA GTX 770 4GB
• and some 16 gigs of RAM
• also, a monitor with multiple video inputs (I switch video source using buttons on the monitor)
• Ubuntu 14.04

As I have mentioned, this set is capable of running very demanding games at maxed settings with amazing results. How does it work in practice? It feels as if I was running both systems at once. For example, while playing a game under Windows, my Linux has an IM client running. Because I mix the sound from both systems, I can hear the notification when I get a message. So I pause the game, switch monitor video source with a hotkey shortcut, respond to the message, and switch the video back. If only I had two monitors, I would play on one of them, with the host system using the other one – so I wouldn’t even need to touch the monitor to switch the OS, I would just need to rotate my head a little bit ;-)

Getting here was a lot of work, but a lot of fun too! The first step is to meet the…

2. Hardware requirements

Yeah. Not every machine will be able to do this trick. As already mentioned, you need two graphics adapters. However, it is not possible to passthrough the graphics integrated in your CPU! This is because passthrough works by separating a PCI device from the host system, and attaching it to the guest OS. Therefore you can only pass a dedicated graphics hardware. Not much of a problem, probably, but it’s probably an important note.

You also need to ensure that your CPU and mainboard support IOMMU – extensions for I/O visualisation, which are necessary for passing through a PCI device. Intel calls their IOMMU technology “VT-d“, while AMD refers to it as “AMD-V“. This is an absolute must, so if you are buying new hardware, make sure both your processor and the chipset will support IOMMU²!

Also, if you plan to use a CPU integrated graphics adapter for the host system, make sure that the mainboard supports it, and that it has a video output!

You will get best results if you use a multi-core CPU. Demanding games will require not only powerful graphics hardware, but a decent CPU as well! It is possible to reserve some of CPU’s cores for the VM – this way you can ensure that the guest OS will be granted enough computational power. For example, in my setup, the host OS uses 2 cores, while other 6 are at Windows’ disposal.

Also, as explained, you need a monitor with several inputs, or a set of two. I am not aware of any way to get this working on a laptop, as most of laptops I know have just one monitor, and you cannot manually switch between video sources¹.

So the full list of requirements is:

• IOMMU compatible CPU and mainboard
• A dedicated PCI graphics adapter (for passing through)
• Graphics hardware for the host OS (can be integrated in CPU)
• Monitor with multiple video inputs (recommended two monitors)
• (Recommended: multi-core CPU).

Warning: Note that you DO NOT NEED a multi-OS graphics card! Contrary to popular belief, non-Quatro NVIDIA cards will work well, with no hardware modifications of any kind!

3. Methods

There are two popular passthrough techniques – one involves Xen virtualization, and one using Qemu and VFIO. Having played around with both, I am personally a fan of the Qemu way – it seems it is much easier to set up, I get more control over my VM, customizations are easier, and, most importantly, it works with virtually any PCI graphics adapter!

There is a lot of confusion on the Internet concerning what results each method may yield. Some say that Qemu method can never grant any decent performance, they claim that only Xen can perform primary VGA passthrough, while Qemu’s secondary VGA passthrough will be very inefficient. However, numerous people (including me) confirm that they have awesome performance with Qemu. On the other hand, it is clear that passthrough with Xen will only work with multi-OS graphic cards. This is not a problem for Radeon users, as probably all new Radeons will do just fine with Xen. However, if your NVIDIA card is not an NVIDIA Quadro, you have no chances with Xen! – unless you burn several resistors on the board, which can mod your card so that it thinks it is a Quadro… I do not recommend such hardware modifications to anyone, even if you trust the Internet too much, the risk of rendering your precious hardware useless is far too high to make it work the effort. Qemu, on the other hand, should work well with absolutely any PCI card.

Given these reasons, as well as customization options, I have decided to stick with Qemu. For the rest of this article, I will be describing this particular method.

There is one particular comprehensive guide on how to setup everything using the Qemu method here – at the time of writing this forums thread has more than 2500 replies, so learning details from here may be hard, but on the other hand every possible scenario is covered somewhere in there :) I can highly recommend that guide, but if you want to learn about the general idea first, stay with me before you jump there!

4. The software

Obviously things won’t work out of the box. There are also necessary preparations on the software side.

First, you will need to patch your kernel a bit, and compile it with several options enabled. At the time of writing, ASC override patches and VGA arbiter fixes need to be applied manually, as they are not (yet?) included in the kernel. You can find details in that guide I linked.

You will need to configure your kernel a bit. The key is not only to ensure it activates appropriate IOMMU modules, but also to forbid it from loading any drivers to the card you will want to pass through.

Most likely it will be also necessary to use the git development version of Qemu – some necessary features are not yet available in stable releases. Also, when playing with qemu, it is worth to try KVM – chances are that hardware virtualization might significantly improve virtual machine’s CPU performance.

You may want to write a bit of scripts that set up few other details (binding the PCI card to vfio module) before you start qemu to run the virtual machine.

Also, it may be tricky to get the right order of installing drivers in the guest OS. It took me a while to realize that I need to disable qemu’s emulated VGA – otherwise NVIDIA drivers won’t detect the dedicated hardware :-)

The greatest issue I have met is that Windows is very sensitive to hardware changes. Even slightest changes in my virtual machine (different qemu options) would immediately cause my Windows to never boot anymore, and any web guides on dealing with these particular BSoDs on boot never helped… So eventually I had to re-install the whole guest OS, after ~10 times I am completely fed up with it. However, if I do not experiment with qemu settings, there are no such problems at all.

There is one more thing that I believe is worth setting up. It’s cool to play Windows games, but there are also many great Linux-native titles. Obviously, if your system boots up with disabled dedicated graphics hardware, any demanding game won’t run. For this reason I have configured my GRUB so that I can choose whether I wish to boot my system to use the graphics card, or whether I want to disable it. This probably can’t be done any simpler, there is no way to get Xorg to switch from one VGA adapter to another while it is running… But it’s not that much of a hassle anyway.

5. Peryphetials

How about keyboard/mouse, should you pass them through too? You might, but this is not necessary; I use Synergy for sharing my mouse/keyboard between systems just as if they were two displays of one system. Very convenient. The script that starts qemu for me also launches synergy server on my Linux, the client running in Windows starts automatically on boot.

If you want, you can also setup networking for the guest system – qemu has very good support for interface bridging, so it is not difficult to grant internet access for the guest OS.

One could also pass-through audio devices, but I believe this is not necessary – especially if you do not care about hardware audio acceleration; in such case you can get qemu to emulate a sound device and play it as any other app in the host OS would do. In result you can hear both systems on same speakers/headphones!

Personally, I have even went so far that I prepared a simple app that talks to my monitor via I²C and tells it when to switch video input – this way I can use a hotkey shortcut instead of navigating it’s OSD menus. The same hotkey will switch my keyboard/mouse between systems, thanks to synergy’s customizability.

 6. Conclusions

I have used this configuration for a few weeks now, and I am yet to find a game that would not perform outstandingly in this environment. Graphics performance is just as if I dual-booted, CPU performance is only a tiny bit worse (but still awesome). The ability to keep all my apps running under Linux while I play games, be it a web browser, IM client, teamspeak or whatever else might be useful – is incredibly convenient!

Switching between systems in less then a second is really a game-changer for me (pun intended…)!

If you are excited about this technique, go ahead and read the guide. Be ready for a challenge, and do not give up it things won’t work – you won’t regret it! Good luck!

Want to know more? I will be happy to answer your general questions, but if you need help or want to learn about technical details, the best place to find answers is here.

---

¹) Unless your motherboard has a video multiplexer, like NVIDIA Optimus… but using it would be difficult, as you would need to manually control the mux. I believe this might be achievable, but most certainly would require specialized drivers, that do not exist right now.

²) It’s not as simple as “all new hardware supports it”, both in case of CPUs and mobos. You may find some lists of IOMMU-compatible hardware on the Internet, but it is probably best to ask the manufacturer itself – if they do not list it on their website, try dropping an email – from my experience all manufacturers are very keen to respond to enquiries concerning such sophisticated features! ;-)