Over the past weekend, I was upgrading my infrastructure and the last piece I needed to do was update my checkout of Xen Orchestra. But, this turned into a multi-hour endeavor to figure out why my updates were not applying (first) and not building (second).

The first problem was easily answered: I had multiple copies of the Xen Orchestra repository in multiple spots, and I was pulling to the wrong one. Whoops.

The second error was less obvious. I tend to have many small virtual machines running at one time, ideally one per broad task. This means I have separate virtual machines for my Calibre server, my Emby server, and any other tasks; all of which are separate from the Xen Orchestra virtual machine. To make this work, the xen-orchestra virtual machine is a super-minimal Debian 12 (Bullseye) installation with just one vCPU and one GiB of RAM statically allocated. This Debian install only has Node 18 and Yarn installed, since this machine exists only to run the management website.

While following the documentation for installing dependencies, the yarn build step was always erroring out. One package in particular refused to build, the other 24 packages built perfectly fine.

Like the start of any good debugging misadventure, I did not read the logs before I began. Once I read the logs two to three hours later, I figured out what happened. The 1GiB of memory was actually running out while building! I was OOM-killing the virtual machine building one of Xen Orchestra's modules, and Linux decided (appropriately) to kill the build.

Normally, the fix for this would be trivial; increase the amount of memory assigned to the virtual machine through the Xen Orchestra web interface.The interface handles the necessary work of broadcasting changes you make on one host or virtual machine to any other pertinent machines (along with updating the Xen database). This is the traditional way to change a virtual machine's parameters on XCP-ng servers. But I had taken Xen Orchestra down to upgrade it! To further add on the complications, I did a git clean -dfx while debugging this problem (thinking the problem was outdated build artifacts), which removed all compilation artifacts. This prevented me from using yarn start, as starting the Xen Orchestra application requires any unbuilt modules to be built.

I was stuck in between a rock and a hard place. Fortunately, XCP-ng's control domain (often called Dom0) exposes a command-line interface to manipulate all the properties of the system. In this case, I was lucky to quickly find exactly what I wanted. CitrixCitrix owns Citrix XenServer, which is another enterprise hypervisor product built on top of Xen. Support had a post about how to increase the RAM assigned to a virtual machine through the command line.

xe vm-memory-limits-set uuid=<uuid of the valid VM> \
                          static-min=<nn>GiB/MiB \
                          dynamic-min=<nn>GiB/MiB \
                          dynamic-max=<nn>GiB/MiB \
                          static-max=<nn>GiB/MiB

Usually, the dom0 virtual machine's shell is able to tab-complete a UUID, given enough uniqueness in its input. But you can also use the human-assigned name of the virtual machine if that is easier.

xe vm-memory-limits-set vm=<name of the valid VM> \
                          static-min=<nn>GiB/MiB \
                          dynamic-min=<nn>GiB/MiB \
                          dynamic-max=<nn>GiB/MiB \
                          static-max=<nn>GiB/MiB

Make sure to set the static-max field of your virtual machines high enough! This is well-covered in both XCP-ng's and Xen Orchestra's documentation, but was something I thought nothing of because I figured "How much RAM could it take to transpile TypeScript?", or something like that.

Give your Xen Orchestra virtual machine at least 2GiB of RAM if you are running from sources!