XCP-ng is a type-1 hypervisor that uses Xen as its virtual machine monitor. Because Xen is a type-1 hypervisor, it also functions as the kernel. Virtualization was (and to some extent still is) investigated where the guest being virtualized is unaware of the virtualization whatsoever. But this incurs performance overheads, sometimes significant ones. What if the guest was aware and cooperated with the host virtualizing the guest? This design is called paravirtualization.

Xen was specifically designed for paravirtualizing virtual machines to greatly enhance the performance of virtual machines by allowing the virtualized operating system to know it is being run as a virtual machine. Paravirtualization allows several steps to be skipped, greatly improving the performance of certain virtual machine behaviors.

This post looks like it is long (and it is), but that is because of the error logs included in the post to make future searches easier. Hopefully, this post gets marked as a useful search result for people trying to set up XCP-ng. Setting up NFS on XCP-ng through a Xen-Orchestra manager can be particularly tricky.

I discuss a small amount of system virtualization history in this section. If you want migrating virtual machines to just work™, then skip this section, and go to the next section. This historical information was gathered when I took Dr. Kyle Hale's CS 562 - Virtual Machines course.

Historically, virtualization research has focused on virtualizing guest operating systems such that they are unaware they are being virtualized at all. In theory (and often in practice), this is a good idea; the virtualized guest should behave exactly the same in the virtual machine as if they were running on real hardware. As first formalized in Formal Requirements for Virtualizable Third Generation Architectures, the hypervisor must emulate the execution of a small number privileged instructions. This means the performance penalty on a virtual machine's execution comes from a small set of privileged instructions which are typically very expensive to emulate.

An example that happens all the time is allocating memory to run a new program. If we had no virtualization:

1. The user program would call into the kernel with a syscall
2. The kernel would find and allocate the memory
3. The kernel returns the newly allocated memory to the user program.

1. The user program calls into the virtualized kernel with a syscall
2. The kernel finds and allocates memory
3. The act of allocating memory is trapped by the hypervisor and behavior of the allocation is emulated by the hypervisor
4. The hypervisor returns some memory to the virtualized guest which then actually allocates the memory
5. The virtualized kernel returns the newly allocated memory to the user program.

The majority of the cost of virtualization comes from the hypervisor's trap-and-emulate behavior, which according to Popek, is a formal requirement for correct virtualization. If we care about minimizing the performance penalty of virtualizing a machine, we could remove quite a number of indirections (steps 3 and 4 from above) by letting the virtualized machine know it is running as a virtual machine. Then the hypervisor would let the virtualized guest make hypercallsHypercalls are the same concept as kernel system calls/syscalls, but hypercalls are made to the hypervisor. to the hypervisor to fulfill certain actions. This concept is paravirtualization.

We could (and do) also add new hardware to processors to allow them to execute some portions of the trap-and-emulate steps in hardware, effectively turning what would be an emulated operation into a native one. One example of this is shadow paging, which automatically maps virtualized-guest virtual pages to host virtual pages. Both Intel and AMD support this on their x86_64 CPUs with Extended Paging (Intel) and Nested Paging (AMD). These hardware virtualization extensions are required to make running virtual machines efficient.

Xen is one of the few hypervisors that offers full support for paravirtualization, has neat datacenter features built into it, and remains a free-to-use product for individuals. I am most interested in using the hot migration feature, where a running virtual machine moves between running hosts without stopping"Without stopping" is a little vague and too strong, but it is what the user of the VM would see, so it is hand-wavy in the right ways..

In many cases, a virtual machine is only intended to run on a single machine. However, what happens when you (as the administrator) need to perform maintenance on the hypervisor host without causing any downtime? One solution is to have the work being performed by the virtual machine (such as hosting a website) be done by several virtual machines on physically-separate hosts and have a load-balancer redirect traffic as virtual machines and/or physical hosts disappear. But what about jobs that do not or cannot work that way? You can migrate a virtual machine from one host to another!

Migrating a virtual machine is intuitive; it closely matches mental models for migrating a physical machine. When you migrate your computer between physical machines, you unplug all of the cables on the outside, unplug the network connection, unplug the disk, and move them to another machine. When you migrate a virtual machine, you are taking the same steps as when you migrate physically, but in pure software instead. This post will go into greater details about how to set up XCP-ng to allow virtual machines to be migrated.

In many ways, virtual machines are identical to regular programs; they:

• Execute on the CPU (albeit there may be emulation to "change" the architecture).
• They consume memory on the executing host.
• They consume disk space (regular programs do this with their read-only program file, rather than a modifiable virtual disk).
• They can use the network to communicate with other computers.

When running virtual machines completely locally, as many software developers do, you do not need to think about these factors in too much depth. However, when you are trying to have a virtual machine run across several different hosts with potentially different CPU types, CPU topologies, memory/disk/network configurations, this becomes a problem, and you end up needing some standardization. For XCP-ng, this standardization comes from the pool the hosts are a member of. Each pool shares some information with every XCP-ng host in the pool so virtual machines can be migrated.

• Handling the CPU architecture is a matter of running XCP-ng on the correct CPU architecture. The Xen hypervisor that XCP-ng uses only runs on x86_64XCP-ng will gain support for ARM CPUs in the near future., so that is already "handled" for you.
• For a migratable VM, the receiving host must have access to the original/booted-from virtual disk that the VM uses. Intuitively, this makes sense; migrating a virtual machine is analogous to moving a system between physical machines. If you did not also migrate/move the disk, the new/receiving system would have no idea what to do. Setting Up the Storage Repository discusses this in more detail.
• For a migratable VM, the receiving host must have a network card that matches what the guest VM configuration expects. Again, intuitively, this means that both the sender and receiver must have a network interface that the guest understands. For example, if the VM configuration is expecting a network card to be connected to the pool with name eth0, it must match on both the sending side and receiving side. Setting up the Network discusses this in more detail.

Virtual machine disks can either exist locally (on the XCP-ng host) or remotely (on another computer). You can migrate a virtual machine regardless of the location of the disk. However, if the disk is local, then when the virtual machine is migrated, the disk must also migrate with it. This makes migrations quite slow. If the disk is remote, then only the running portions of the virtual machine needs to be moved, and the disk remains on the remote host. Migrations with remote disks are significantly faster because the memory (RAM) being used by the virtual machine is (usually) significantly smaller than the disk size.

I recommend you use a remote system to hold the virtual machine disks to make migrations faster and separate the concerns of each physical machine.

I run a local TrueNAS file-serving system locally, which means that I can place all the virtual machine disks on a physically separate machine. TrueNAS then serves virtual machine disks over NFS to my hypervisors. To make the XCP-ng computer aware of this, I export a share from the file server using NFS. You (as the administrator) must add the NFS export as a storage repository (SR) before you are able to use it.

To add a new storage repository, you go to Home>Pools in Xen-orchestra, click the name of the pool you want to modify, then there should a disk-like icon in the upper right that when hovered over says "Add SR". You can also handle this with the New option in the left-hand side navigation button. Click that, and fill in the information required to connect to your file server host.

NOTE: When you use NFS, the Server field under Settings is the IP address or the fully-qualified hostname of the file server, excluding the exported path! Once you enter the path to the storage server, press the magnifying class icon to the right of the Server field. This will probe for the server, and if it is reachable, an additional Path field will show up asking which NFS export to mount.

This section can be skipped if you have no errors. However, some of the error messages Xen and Xen-Orchestra return to you are not always easily understood.

As with any networked application that involves multiple operating systems, multiple applications, multiple users, and NFS, you will likely receive some cryptic errors that you need to tease out. I am including the errors I had when mounting my NFS share on XCP-ng that I needed to resolve to create a VM that can be migrated. Hopefully you find them useful.

You can see a trace of these logs in Settings>Logs.

Error 140 is a simple DNS failure.

SR_BACKEND_FAILURE_140(, Incorrect DNS name, unable to resolve., )

This says that when Xen attempts to perform an NFS lookup to determine the NFS server's exported paths, the hostname you provided could not be resolved.

The expanded body of this error message was:

sr.probeNfs
{
  "host": "49058e64-761a-4453-93f5-b9b4cce5a77d",
  "server": "nfs-nas.localdomain"
}
{
  "code": "SR_BACKEND_FAILURE_140",
  "params": [
    "",
    "Incorrect DNS name, unable to resolve.",
    ""
  ],
  "call": {
    "method": "SR.probe",
    "params": [
      "OpaqueRef:07f854df-0c34-4ffe-84bb-c748d09066f1",
      {
        "server": "nfs-nas.localdomain"
      },
      "nfs",
      {}
    ]
  },
  "message": "SR_BACKEND_FAILURE_140(, Incorrect DNS name, unable to resolve., )",
  "name": "XapiError",
  "stack": "XapiError: SR_BACKEND_FAILURE_140(, Incorrect DNS name, unable to resolve., )
    at Function.wrap (file:///opt/xen-orchestra/packages/xen-api/_XapiError.mjs:16:12)
    at file:///opt/xen-orchestra/packages/xen-api/transports/json-rpc.mjs:35:21
    at runNextTicks (node:internal/process/task_queues:60:5)
    at processImmediate (node:internal/timers:447:9)
    at process.callbackTrampoline (node:internal/async_hooks:128:17)"
}

To fix error 140, you must correct the fully-qualified hostname for the NFS server or use an IP address instead of the hostname.

SR_BACKEND_FAILURE_88(, NFS SR creation error [opterr=remote directory creation error is 13], )

The expanded body of this error message was:

sr.createNfs
{
  "host": "49058e64-761a-4453-93f5-b9b4cce5a77d",
  "nameLabel": "test",
  "nameDescription": "test",
  "server": "nfs-nas.localdomain",
  "serverPath": "/mnt/store/xcpng-vm-disks/"
}
{
  "code": "SR_BACKEND_FAILURE_88",
  "params": [
    "",
    "NFS SR creation error [opterr=remote directory creation error is 13]",
    ""
  ],
  "call": {
    "method": "SR.create",
    "params": [
      "OpaqueRef:07f854df-0c34-4ffe-84bb-c748d09066f1",
      {
        "server": "nfs-nas.localdomain",
        "serverpath": "/mnt/store/xcpng-vm-disks/"
      },
      0,
      "test",
      "test",
      "nfs",
      "user",
      true,
      {}
    ]
  },
  "message": "SR_BACKEND_FAILURE_88(, NFS SR creation error [opterr=remote directory creation error is 13], )",
  "name": "XapiError",
  "stack": "XapiError: SR_BACKEND_FAILURE_88(, NFS SR creation error [opterr=remote directory creation error is 13], )
    at Function.wrap (file:///opt/xen-orchestra/packages/xen-api/_XapiError.mjs:16:12)
    at file:///opt/xen-orchestra/packages/xen-api/transports/json-rpc.mjs:35:21
    at runNextTicks (node:internal/process/task_queues:60:5)
    at processImmediate (node:internal/timers:447:9)
    at process.callbackTrampoline (node:internal/async_hooks:128:17)"
}

Error 13 could be from a variety of reasons including (but not limited to):

• The mapped user (or group) on the remote host not having the correct permissions.
• This could be because of the No Such Directory as well.

Error 73 has a variety of different possible meanings.

This error occurs because the Xen-local NFS client cannot detect (and therefore cannot connect) to the remote NFS server.

SR_BACKEND_FAILURE_73(, NFS mount error [opterr=Failed to detect NFS service on server 172.16.60.200], )

This could happen because you have a variety of networking issues (Ethernet MTU, TCP TTL, etc.), NFS issues (you are running NFS or rcpbind on non-default ports). I am not sure the best way to diagnose and fix this, as I have never encountered this issue myself, only finding this error through this issue.

This is a strange error, and I have not figured it out quite yet. This appears to be related to the storage repository SR-UUID created inside the path specified during configuration, namely the "No Such Directory" section below.

SR_BACKEND_FAILURE_73(, NFS mount error [opterr=mount failed with return code 32], )

The full error.

sr.destroy
{
  "id": "49c1d405-218d-6673-c6b7-e2ce5de2b853"
}
{
  "code": "SR_BACKEND_FAILURE_73",
  "params": [
    "",
    "NFS mount error [opterr=mount failed with return code 32]",
    ""
  ],
  "call": {
    "method": "SR.destroy",
    "params": [
      "OpaqueRef:38069976-cca5-44a2-8e7c-c930fc5f6773"
    ]
  },
  "message": "SR_BACKEND_FAILURE_73(, NFS mount error [opterr=mount failed with return code 32], )",
  "name": "XapiError",
  "stack": "XapiError: SR_BACKEND_FAILURE_73(, NFS mount error [opterr=mount failed with return code 32], )
    at Function.wrap (file:///opt/xen-orchestra/packages/xen-api/_XapiError.mjs:16:12)
    at file:///opt/xen-orchestra/packages/xen-api/transports/json-rpc.mjs:35:21
    at runNextTicks (node:internal/process/task_queues:60:5)
    at processImmediate (node:internal/timers:447:9)
    at process.callbackTrampoline (node:internal/async_hooks:128:17)"
}

This is because Xen Orchestra relies on being able to mount subdirectories within the NFS export!

Failed to scan SR 14f804e6-b45d-17c0-8529-6a5fa3032f94 after attaching, error The SR is not available [opterr=no such directory /var/run/sr-mount/14f804e6-b45d-17c0-8529-6a5fa3032f94]

To fix this, either

1. Check the All dirs on TrueNAS's Sharing>Unix Shares (NFS)>(your-share).
2. Pass the -alldirs flag to the particular NFS export in /etc/export.

NOTE: This is documented in XCP-ng's storage documentation! I missed this for a long time during my setup.

The Storage Repository (on Xen's side) will show up in /var/run/sr-mount/<SR-UUID>/. Because /var/run/ is symbolically linked to /run/, storage repositories also show up in /run/sr-mount/sr-mount/<SR-UUID>/. You do not need to worry about this.

Setting up a virtual network can be tricky, depending on what you want machines connected to that network to be able to do. But, the common case for a virtual machine is that it can connect to a wider network, like the Internet. This is done by connecting the virtual machine's Virtual InterFace (VIF) to a network that has a Physical Interface (PIF) connected to the Internet. This does not mean that all of your virtual machines need to be aware of one another.

XCP-ng's installer will attempt to figure out the network for you. However, it can make errors accidentally. In particular, look at how XCP-ng names network interfaces.

The myriad of ways you have configured your network both physically and with your router means that this blog post can not answer every question you may have. However, I have a very simple network (no subnets, no VLANs, etc.), which means the issues I face should be the most fundamental ones.

VM_REQUIRES_NETWORK(OpaqueRef:622a2cd9-537c-4a49-933c-e47ce1f8555e, OpaqueRef:b24a7465-1170-44e7-b308-5d67f2bdf146)

vm.migrate
{
  "vm": "6c79edf3-702b-493b-2648-a6e2446c5a39",
  "migrationNetwork": "45fd7ebd-75cb-5ebd-8d5d-5c45416b9cc6",
  "sr": "05e64867-cd3e-be30-d598-68167f01bcc3",
  "targetHost": "b20a2b96-ef76-4eb9-87b0-dd70159e9be7"
}
{
  "code": "VM_REQUIRES_NETWORK",
  "params": [
    "OpaqueRef:622a2cd9-537c-4a49-933c-e47ce1f8555e",
    "OpaqueRef:b24a7465-1170-44e7-b308-5d67f2bdf146"
  ],
  "task": {
    "uuid": "63f8c0cb-d9ff-4e78-cbbd-a00d4027e415",
    "name_label": "Async.VM.assert_can_migrate",
    "name_description": "",
    "allowed_operations": [],
    "current_operations": {},
    "created": "20240204T07:16:19Z",
    "finished": "20240204T07:16:20Z",
    "status": "failure",
    "resident_on": "OpaqueRef:07f854df-0c34-4ffe-84bb-c748d09066f1",
    "progress": 1,
    "type": "<none/>",
    "result": "",
    "error_info": [
      "VM_REQUIRES_NETWORK",
      "OpaqueRef:622a2cd9-537c-4a49-933c-e47ce1f8555e",
      "OpaqueRef:b24a7465-1170-44e7-b308-5d67f2bdf146"
    ],
    "other_config": {},
    "subtask_of": "OpaqueRef:NULL",
    "subtasks": [],
    "backtrace": "(((process xapi)(filename ocaml/xapi/rbac.ml)(line 205))((process xapi)(filename ocaml/xapi/server_helpers.ml)(line 95)))"
  },
  "message": "VM_REQUIRES_NETWORK(OpaqueRef:622a2cd9-537c-4a49-933c-e47ce1f8555e, OpaqueRef:b24a7465-1170-44e7-b308-5d67f2bdf146)",
  "name": "XapiError",
  "stack": "XapiError: VM_REQUIRES_NETWORK(OpaqueRef:622a2cd9-537c-4a49-933c-e47ce1f8555e, OpaqueRef:b24a7465-1170-44e7-b308-5d67f2bdf146)
    at Function.wrap (file:///opt/xen-orchestra/packages/xen-api/_XapiError.mjs:16:12)
    at default (file:///opt/xen-orchestra/packages/xen-api/_getTaskResult.mjs:11:29)
    at Xapi._addRecordToCache (file:///opt/xen-orchestra/packages/xen-api/index.mjs:998:24)
    at file:///opt/xen-orchestra/packages/xen-api/index.mjs:1032:14
    at Array.forEach (<anonymous>)
    at Xapi._processEvents (file:///opt/xen-orchestra/packages/xen-api/index.mjs:1022:12)
    at Xapi._watchEvents (file:///opt/xen-orchestra/packages/xen-api/index.mjs:1195:14)"
}

This error does not always have a single reason, but the underlying cause seems to always be the same; the VM is crossing a network boundary. We pool multiple Xen hosts together to produce a distributed set of hypervisors that we can migrate VMs between. These virtual machines must be connected to some greater network to communicate with the wider world.

To give an analogy: When you migrate a VM, you are effectively "moving the computer to a different physical location, but keeping the Ethernet cable plugged in". The VM_REQUIRES_NETWORK error is returned when you attempt to move the computer to a different physical location and unplug the Ethernet cable simultaneously.

Xen chooses which NICs belong in which pools together in a simple way; by simply matching against the ethX assigned to each PIF. This caused a problem in my case, because the two machines had the management NIC assigned a different ethX name, which meant the two were connected to two different pool networks. One Xen machine had eth4 as the management interface and the other had eth0, which meant that the two Xen machines could communicate with one another (because Xen uses IP addresses for that communication), but guest machines could not move between the physical hosts.

Xen documentation calls a physical NIC a PIF (Physical InterFace). There are also VIFs (Virtual InterFaces), which are virtual NICs for guest virtual machines. A network is a collection of PIFs and VIFs which can talk to one another. Xen creates pool-wide networks by matching against the ethX PIFs, which are assigned at boot-time. Xen's internal database can force specific names by matching against MAC addresses, ensuring PIFs are pooled correctly.

To fix my issue, I needed to manually rename the PIFs (NICs) on the host that named the PIFs incorrectly. I consulted both XCP-ng's documentation and Citrix XenServer's documentationXCP-ng and Citrix XenServer both use the Xen hypervisor/kernel, so the documentation is often interchangeable. for this.

• XCP-ng Documentation for NIC Replace/Rename
• Citrix Xen Support for NIC Replace/Rename

After renaming the incorrect eth4 to eth0 on the offending hypervisor, both machines placed their management NICs in the same pool-wide network, and a virtual machine could hot and cold migrate between the two hypervisors without modification!

After all of this setup, you should be able to migrate your virtual machine between any two hosts that reside in the same pool. This also works when you send one of the hosts into XCP-ng's "Maintenance Mode"Setting Maintenance Mode on a host means it should not be considered as part of the pool, and should have all running virtual machines evicted from it. This allows operators to work on virtualizing hosts without worrying about downtime., which allows you to perform maintenance on that host without any virtual machines running.

There are a few limitations of Maintenance Mode. The biggest one is merely a byproduct of virtualization itself. You cannot migrate a virtual machine which has specialty hardware dedicated to it.

Overall, migrating a virtual machine in XCP-ng is not difficult, but you must deliberately design your system to behave that way. In many cases, it is intuitive the reason why you must do this, but error messages and diagnostic information from Xen Orchestra might not give you the most helpful information. I hope that this post's collection of error messages, possible issue reasons, and actual solutions is helpful in setting up an XCP-ng pool for yourself.

• Formal Requirements for Virtualizable Third Generation Architectures
• XCP-ng Documentation