X86 virtualization


x86 virtualization is the use of hardware-assisted virtualization capabilities on an x86/x86-64 CPU.
In the late 1990s x86 virtualization was achieved by complex software techniques, necessary to compensate for the processor's lack of hardware-assisted virtualization capabilities while attaining reasonable performance. In 2005 and 2006, both Intel and AMD introduced limited hardware virtualization support that allowed simpler virtualization software but offered very few speed benefits. Greater hardware support, which allowed substantial speed improvements, came with later processor models.

Software-based virtualization

The following discussion focuses only on virtualization of the x86 architecture protected mode.
In protected mode the operating system kernel runs in kernel space at the most privileged level that allows it to configure the MMU, manage physical memory, and directly control I/O peripherals, while applications run in user space at a lower privilege level, where they are confined to their own virtual address spaces and must invoke system calls to request I/O operations or other privileged services from the kernel.
In software-based virtualization, a host OS has direct access to hardware while the guest operating systems have limited access to hardware, similarly to other user space application of the host OS. One approach used in x86 software-based virtualization to implement this mechanism is called ring deprivileging, which involves running the guest OS at a ring higher than 0, so that attempts to execute privileged instructions can be intercepted and handled by the hypervisor.
Three techniques made virtualization of protected mode possible:
  • Binary translation is used to rewrite certain ring 0 instructions in terms of ring 3 instructions, such as POPF, that would otherwise fail silently or behave differently when executed [|above] ring 0, making the classic trap-and-emulate virtualization impossible. To improve performance, the translated basic blocks need to be cached in a coherent way that detects code patching, the reuse of pages by the guest OS, or even self-modifying code.
  • A number of key data structures used by a processor need to be shadowed. Because most operating systems use paged virtual memory, and granting the guest OS direct access to the MMU would mean loss of control by the virtualization manager, some of the work of the x86 MMU needs to be duplicated in software for the guest OS using a technique known as shadow page tables. This involves denying the guest OS any access to the actual page table entries by trapping access attempts and emulating them instead in software. The x86 architecture uses hidden state to store segment descriptors in the processor, so once the segment descriptors have been loaded into the processor, the memory from which they have been loaded may be overwritten and there is no way to get the descriptors back from the processor. Shadow descriptor tables must therefore be used to track changes made to the descriptor tables by the guest OS.
  • I/O device emulation: Unsupported devices on the guest OS must be emulated by a device emulator that runs in the host OS.
These techniques incur some performance overhead due to lack of MMU virtualization support, as compared to a VM running on a natively virtualizable architecture such as the IBM System/370.
On traditional mainframes, the classic type 1 hypervisor was self-standing and did not depend on any operating system or run any user applications itself. In contrast, the first x86 virtualization products were aimed at workstation computers, and ran a guest OS inside a host OS by embedding the hypervisor in a kernel module that ran under the host OS.
There has been some controversy whether the x86 architecture with no hardware assistance is virtualizable as described by Popek and Goldberg. VMware researchers pointed out in a 2006 ASPLOS paper that the above techniques made the x86 platform virtualizable in the sense of meeting the three criteria of Popek and Goldberg, albeit not by the classic trap-and-emulate technique.
A different route was taken by other systems like Denali, L4, and Xen, known as paravirtualization, which involves porting operating systems to run on the resulting virtual machine, which does not implement the parts of the actual x86 instruction set that are hard to virtualize. The paravirtualized I/O has significant performance benefits as demonstrated in the original SOSP'03 Xen paper.
The initial version of x86-64 did not allow for a software-only full virtualization due to the lack of segmentation support in long mode, which made the protection of the hypervisor's memory impossible, in particular, the protection of the trap handler that runs in the guest kernel address space. Revision D and later 64-bit AMD processors added basic support for segmentation in long mode, making it possible to run 64-bit guests in 64-bit hosts via binary translation. Intel did not add segmentation support to its x86-64 implementation, making 64-bit software-only virtualization impossible on Intel CPUs, but Intel VT-x support makes 64-bit hardware assisted virtualization possible on the Intel platform.
On some platforms, it is possible to run a 64-bit guest on a 32-bit host OS if the underlying processor is 64-bit and supports the necessary virtualization extensions.

Hardware-assisted virtualization

The first generation of x86 hardware virtualization addressed the issue of privileged instructions. The issue of low performance of virtualized system memory was addressed with MMU virtualization that was added to the chipset later. In 2005 and 2006, Intel and AMD created new processor extensions to the x86 architecture, resulting in two separate, binary incompatible x86 virtualization extension variants - Intel's VT-x and AMD-V.

Central processing unit

Virtual 8086 mode

Because the Intel 80286 could not run concurrent DOS applications well by itself in protected mode, Intel introduced the virtual 8086 mode in their 80386 chip, which offered virtualized 8086 processors on the 386 and later chips. Hardware support for virtualizing the protected mode itself, however, became available 20 years later.

AMD virtualization (AMD-V)

AMD developed its first generation virtualization extensions under the code name "Pacifica", and initially published them as AMD Secure Virtual Machine, but later marketed them under the trademark AMD Virtualization, abbreviated AMD-V.
On May 23, 2006, AMD released the Athlon 64, the Athlon 64 X2 and the Athlon 64 FX as the first AMD processors to support this technology.
AMD-V capability also features on the Athlon 64 and Athlon 64 X2 family of processors with revisions "F" or "G" on socket AM2, Turion 64 X2, and Opteron 2nd generation and third-generation, Phenom and Phenom II processors. The APU Fusion processors support AMD-V. AMD-V is not supported by any Socket 939 processors. The only Sempron processors which support it are APUs and Huron, Regor, Sargas desktop CPUs.
AMD Opteron CPUs beginning with the Family 0x10 Barcelona line, and Phenom II CPUs, support a second generation hardware virtualization technology called Rapid Virtualization Indexing, later adopted by Intel as Extended Page Tables.
As of 2019, all Zen-based AMD processors support AMD-V.
The CPU flag for AMD-V is "svm". This may be checked in BSD derivatives via dmesg or sysctl and in Linux via /proc/cpuinfo. Instructions in AMD-V include VMRUN, VMLOAD, VMSAVE, CLGI, VMMCALL, INVLPGA, SKINIT, and STGI.
With some motherboards, users must enable AMD SVM feature in the BIOS setup before applications can make use of it.

Intel virtualization (VT-x)

Previously codenamed "Vanderpool", VT-x represents Intel's technology for virtualization on the x86 platform. On November 14, 2005, Intel released two models of Pentium 4 as the first Intel processors to support VT-x. The CPU flag for VT-x capability is "vmx"; in Linux, this can be checked via /proc/cpuinfo, or in macOS via sysctl machdep.cpu.features.
"VMX" stands for Virtual Machine Extensions, which adds 13 new instructions: VMPTRLD, VMPTRST, VMCLEAR, VMREAD, VMWRITE, VMCALL, VMLAUNCH, VMRESUME, VMXOFF, VMXON, INVEPT, INVVPID, and VMFUNC. These instructions permit entering and exiting a virtual execution mode where the guest OS perceives itself as running with full privilege, but the host OS remains protected.
, almost all newer server, desktop and mobile Intel processors support VT-x, with some of the Intel Atom processors as the primary exception. With some motherboards, users must enable Intel's VT-x feature in the BIOS setup before applications can make use of it.
Intel started to include Extended Page Tables, a technology for page-table virtualization, since the Nehalem architecture, released in 2008. In 2010, Westmere added support for launching the logical processor directly in real mode a feature called "unrestricted guest", which requires EPT to work.
Since the Haswell microarchitecture, Intel started to include VMCS shadowing as a technology that accelerates nested virtualization of VMMs.
The virtual machine control structure is a data structure in memory that exists exactly once per VM, while it is managed by the VMM. With every change of the execution context between different VMs, the VMCS is restored for the current VM, defining the state of the VM's virtual processor. As soon as more than one VMM or nested VMMs are used, a problem appears in a way similar to what required shadow page table management to be invented, as described above. In such cases, VMCS needs to be shadowed multiple times and partially implemented in software in case there is no hardware support by the processor. To make shadow VMCS handling more efficient, Intel implemented hardware support for VMCS shadowing.