Running Android VMs on ARM: Rebuilding the Minisforum MS-R1 Kernel for Cuttlefish

Part 1 of 2. This part covers getting a kernel that can actually host virtual machines. Why bother I wanted a box that could run a dozen Android instances at once — real ones, not emulated-on-x86 ones — to benchmark peer-to-peer sync behaviour at scale. Native arm64 Android on native arm64 silicon, no translation layer, enough cores and RAM to make the peer count interesting. The Minisforum MS-R1 looked ideal. It's built on the CIX P1 ("Sky1"), a 12-core ARMv9 SoC, and it's one of the first genuinely affordable ARM desktops with server-class amounts of memory. Google's Cuttlefish — AOSP's official virtual device — runs arm64 Android guests on arm64 hosts with KVM acceleration, with a --num_instances=N flag that does exactly what I wanted. Everything lined up. Then I hit this: $ sudo modprobe vhost_vsock modprobe: FATAL: Module vhost_vsock not found in directory /lib/modules/6.6.10-cix-build-generic This post is what it took to fix that. If you have this hardware and want to run VMs on it, you'll hit the same wall, and there are four separate traps between you and the other side. I hit all of them so you don't have to. Rough time: an afternoon. Most of it is a compile you can walk away from. The problem: no vhost, no Cuttlefish Cuttlefish uses vsock — a virtual socket transport — for all communication between the host and its guest VMs. ADB, logs, control messages, everything. Without /dev/vhost-vsock, Cuttlefish doesn't start. It's not a soft dependency. The kernel Minisforum ships is 6.6.10-cix-build-generic. Check what it thinks about virtualization: grep -E 'VHOST' /boot/config-$(uname -r) On mine, the output was more interesting for what was missing than what was there: # CONFIG_VHOST_NET is not set CONFIG_VHOST_VSOCK doesn't appear at all — not even as "is not set". That happens when the parent CONFIG_VHOST symbol is disabled, so Kconfig never emits the dependent symbols. The vendor didn't disable vsock specifically; they disabled the entire vhost subsystem. Confirmed at the device level: $ ls -l /dev/vhost-vsock /dev/vhost-net /dev/kvm ls: cannot access '/dev/vhost-vsock': No such file or directory ls: cannot access '/dev/vhost-net': No such file or directory crw-rw----+ 1 root kvm 10, 232 /dev/kvm KVM is there. Vhost isn't. This costs you more than Cuttlefish CONFIG_VHOST_NET being off is worth dwelling on. Without it, every VM on the machine gets its virtio networking serviced in QEMU userspace instead of in the kernel. If you're running Proxmox on this box — and the MS-R1 docs walk you through exactly that — all your guests are quietly getting the slow path. So this isn't a niche Cuttlefish fix. It's restoring standard virtualization support to a machine marketed partly on being able to virtualize. Why is it off? Not a hardware limitation. CIX's own mainline config (cixtech/cix-linux-main) sets: CONFIG_VHOST_NET=m CONFIG_VHOST_VSOCK=m CONFIG_VSOCKETS=m CONFIG_KVM=y There's a clue in the vendor tree about why the shipped build differs. Look at what CIX ships as config fragments: arch/arm64/configs/ cix.config 596 lines main platform fragment cix_redroid.config 222 lines redroid (containerized Android) cix_docker.config 40 lines Docker support virt.config 61 lines (see the warning below) There's a 222-line fragment for redroid — containerized Android — and none of the fragments enable vhost anywhere. The vendor's supported Android story is containers, which share the host kernel and need no vsock at all. Their kernel has binder compiled in and vhost stripped out. Kernel and documentation agree with each other; it's just not the story I wanted. A trap worth naming. virt.config sounds like the virtualization options. It isn't. Its header reads "Virtualization guest" — it configures a kernel to run inside a VM, disabling every platform ARCH_* plus THERMAL, SPI, MTD, and REGULATOR. Applying it gets you a kernel that won't drive your hardware. I lost time on this so you don't have to. Choosing a source tree Three options, and the obvious one is wrong. Mainline. CIX Sky1 support landed upstream around v6.17 (LWN coverage). But it's minimal enablement — mailbox, SCMI clocks, basic device tree — and it targets the Radxa Orion O6, a different board with different firmware and different NICs. You'd likely lose networking or fail to boot. cixtech/cix-linux-main. Patch sets for mainline 6.18/7.0/7.1. Its defconfig has everything Cuttlefish needs, which is how I confirmed vhost isn't hardware-limited. But the README says tested on Orion O6 with specific edk2 firmware, and it wants clk_ignore_unused. Jumping 6.6 → 6.18 on unvalidated firmware is the high-risk path. cixtech/cix_opensource__linux, branch cix_p1_k6.6_master. ← this one. Full BSP source tree, CIX P1 specific, same 6.6 lineage as the shipped kernel. This is the GPL source drop your running kernel came from. One wrinkle: branch HEAD is at 6.6.89 while the shipped kernel is 6.6.10. That's 79 stable releases of drift, and as you'll see, it's the direct cause of two of the four traps. It's still the right choice — same BSP, same hardware support — but plan on keeping the original kernel bootable. Before anything else: understand how this thing boots This is where the MS-R1 gets genuinely unusual, and where I'd have bricked my access if I hadn't looked first. $ ls /boot/grub/grub.cfg ls: cannot access '/boot/grub/grub.cfg': No such file or directory $ dpkg -l | grep -E 'grub|systemd-boot|u-boot' (nothing) $ mount | grep -i efi efivarfs on /sys/firmware/efi/efivars type efivarfs (...) No bootloader package. No EFI partition mounted. But: $ cat /proc/cmdline BOOT_IMAGE=/Image console=tty0 console=ttyAMA0,115200 ... acpi=force ... /Image is a raw arm64 kernel. Something is loading it. Mount the ESP and the picture resolves: sudo mkdir -p /mnt/esp sudo mount /dev/nvme0n1p1 /mnt/esp ls /mnt/esp EFI/BOOT/BOOTAA64.EFI ← GRUB, at the UEFI fallback path GRUB/GRUB.CFG ← its config Image ← the actual kernel initrd.img-6.6.10-cix-build-generic SKY1-*.dtb ← device trees for various Sky1 boards So the real chain is: UEFI firmware └─ /dev/nvme0n1p1 (ESP, FAT16, 500 MB) └─ /EFI/BOOT/BOOTAA64.EFI (GRUB) └─ /GRUB/GRUB.CFG ├─ /Image └─ /initrd.img-6.6.10-cix-build-generic Three consequences that break normal kernel-install habits: /boot is decorative. The files there are leftovers nothing reads. Installing a kernel .deb writes to /boot, reports success, and changes nothing about booting. update-grub does not exist. GRUB was placed by Minisforum's installer, not apt. You edit GRUB.CFG by hand. The ESP is small. 500 MB, with ~200 MB free, and the existing initramfs eats 239 MB of it. Space is a real constraint. The shipped GRUB.CFG has two entries: set default=1 set timeout=2 menuentry '0 Cix Sky1 on Meigao (Device Tree)' { ... acpi=off ... } menuentry '1 Cix Sky1 on Meigao (ACPI)' { ... acpi=force ... } Default is entry 1 (ACPI), matching /proc/cmdline. The DTB files aren't used by it. And timeout=2 is two seconds — not enough to select a fallback, which we'll fix. Safety, non-negotiable A new kernel can rename network interfaces. If yours come back different, your bridge config breaks and the box drops off the network — and if you're on SSH, you're locked out. Attach a monitor and USB keyboard. Verify you can log in at the console. Photograph your ip -br a and /etc/network/interfaces. Back up the whole ESP: sudo mkdir -p /root/esp-backup sudo cp -a /mnt/esp/. /root/esp-backup/ 299 MB onto a 1.8 TB disk. It makes the worst case fully recoverable. Building the kernel Tools sudo apt update sudo apt install -y build-essential bc bison flex libssl-dev libelf-dev \ libncurses-dev rsync kmod cpio dwarves debhelper \ fakeroot git zstd screen Source cd ~ git clone --depth 1 -b cix_p1_k6.6_master \ https://github.com/cixtech/cix_opensource__linux.git cix-kernel cd cix-kernel git branch --show-current # cix_p1_k6.6_master head -5 Makefile # VERSION 6, PATCHLEVEL 6, SUBLEVEL 89 --depth 1 matters — the full history is over a million commits and roughly 20 GB. The snapshot is about 2 GB and is all you need to build. Config Start from the running kernel's config so the result is as close as possible to what already works: cp /boot/config-6.6.10-cix-build-generic .config make olddefconfig Now the actual point of the exercise: ./scripts/config --enable VHOST_MENU ./scripts/config --module VHOST_NET ./scripts/config --module VHOST_VSOCK ./scripts/config --module VSOCKETS Plus two more you'd otherwise discover the hard way — details in the traps section: ./scripts/config --enable CIX_THERMAL ./scripts/config --enable CIX_AP2SE_IPC Give the new kernel a distinct name. This is the step that protects your fallback — without it, the new kernel's modules overwrite the old kernel's, and you lose the ability to boot back: ./scripts/config --set-str LOCALVERSION "-cix-vhost" ./scripts/config --disable LOCALVERSION_AUTO Two housekeeping items. The first avoids a classic Debian build failure where the config points at certificate files that don't exist; the second roughly halves build time and saves ~15 GB: ./scripts/config --disable MODULE_SIG_FORCE ./scripts/config --set-str SYSTEM_TRUSTED_KEYS "" ./scripts/config --set-str SYSTEM_REVOCATION_KEYS "" ./scripts/config --disable DEBUG_INFO ./scripts/config --enable DEBUG_INFO_NONE Apply and verify — this is the checkpoint that matters most, because a build missing VHOST_VSOCK completes successfully and is useless: make olddefconfig grep -E '^CONFIG_(VHOST|VSOCKETS|CIX_THERMAL|CIX_AP2SE_IPC)' .config grep '^CONFIG_LOCALVERSION=' .config CONFIG_VHOST_MENU=y CONFIG_VHOST=m CONFIG_VHOST_NET=m CONFIG_VHOST_VSOCK=m CONFIG_VSOCKETS=m CONFIG_CIX_THERMAL=y CONFIG_CIX_AP2SE_IPC=y CONFIG_LOCALVERSION="-cix-vhost" Compile screen -S kernelbuild make -j$(nproc) bindeb-pkg 2>&1 | tee ~/kbuild.log Two habits worth adopting. screen means a dropped SSH doesn't kill a 60-minute build. tee matters more than it looks: when a kernel build fails, your terminal shows a cascade of make[N]: *** Error 2 lines and the actual error is thousands of lines above, often scrolled away. Without a log you'll rebuild just to read the message. 30–90 minutes on this SoC. Thousands of warnings are normal; only error: and a deep make[N]: ... Error 1 matter. The four traps 1. CONFIG_CIX_THERMAL — a compile error drivers/acpi/processor_thermal.c:432:18: error: implicit declaration of function 'processor_get_static_power_cpus' [-Werror=implicit-function-declaration] "Implicit declaration" normally means a missing header. Here the functions are static and defined in the same file: 65: #ifdef CONFIG_CIX_THERMAL 66: cix_get_static_power() ← reads ACPI method \_SB.DPRG 122: processor_get_static_power_cpus() ← the "missing" one 134: processor_get_dynamic_power_cpus() 145: #endif ... 432: processor_get_requested_power() ← calls them, NOT guarded Definitions inside the #ifdef, call sites outside it. With the symbol off, the calls survive and the definitions vanish. It's a vendor bug that never bites CIX because they always build with it on. CIX_THERMAL is CIX's CPU power measurement for this SoC — enabling it is correct configuration, not a workaround. 2. CONFIG_CIX_AP2SE_IPC — a link error, 40 minutes later drivers/soc/cix/cix_dst/blackbox/platform_ap/rdr_ap_adapter.c:149: undefined reference to 'cix_ap2se_ipc_send' make[6]: *** [scripts/Makefile.vmlinux:37: vmlinux] Error 1 The provider is drivers/soc/cix/cix_ap2se_ipc.c, guarded by CONFIG_CIX_AP2SE_IPC. The config had: # CONFIG_CIX_AP2SE_IPC is not set ← provider OFF CONFIG_CIX_DST=y ← consumer ON The cause is the version drift. The 6.6.10 config calls this CONFIG_CIX_SE2AP_MBOX; 6.6.89 renamed it to CONFIG_CIX_AP2SE_IPC. olddefconfig can't map an old name to a new one, so the new symbol took its default (off) while every consumer stayed on. It must be =y, not =m — CIX_DST=y is built into vmlinux and can't link against a module. Catch both before compiling Both traps are the same root cause: a 6.6.10 config grafted onto a 6.6.89 tree. Diff them and you'll see everything queued up: grep '^CONFIG_CIX\|^# CONFIG_CIX' /boot/config-6.6.10-cix-build-generic | sort > /tmp/old.txt grep '^CONFIG_CIX\|^# CONFIG_CIX' .config | sort > /tmp/new.txt diff /tmp/old.txt /tmp/new.txt Reading it: =y in old, absent in new → renamed or removed upstream. Find the new name. New symbol defaulting to off → usually a genuinely new optional feature; fine. Enabled consumer, disabled provider → a link error waiting to happen. On my build the drift was ~9 lines. These were new-and-off and safe to leave: CIX_DSP, CIX_GPIO_READER, CIX_HF_TEST_CASE, CIX_KERNEL_SENSORHUB, CIX_SE_CONFIG, CIX_SF_SCP_SUPPORT. Installing, on a machine where /boot doesn't matter Build output: linux-image-6.6.89-cix-vhost+_6.6.89-g2ec99ef99058-3_arm64.deb linux-headers-6.6.89-cix-vhost+_...deb Note the trailing +. scripts/setlocalversion appends it when building from a git tree whose HEAD isn't a tagged release. Your kernel version is 6.6.89-cix-vhost+, and dropping the + in later commands means quietly targeting a kernel that doesn't exist. (Avoid it next time with touch .scmversion before building.) Never type it — derive it: KVER=$(ls -1 /lib/modules/ | grep vhost) echo "$KVER" # 6.6.89-cix-vhost+ Shrink the initramfs first Order matters: installing the kernel package triggers an initramfs build via its post-install hook. Set the lean option before installing, or you generate a 250 MB file you then have to redo. The default MODULES=most packs drivers for every machine ever built — that's the 239 MB initrd on the ESP. MODULES=dep includes only what this machine needs: sudo cp /etc/initramfs-tools/initramfs.conf /etc/initramfs-tools/initramfs.conf.backup sudo sed -i 's/^MODULES=most/MODULES=dep/' /etc/initramfs-tools/initramfs.conf sudo dpkg -i linux-image-6.6.89-cix-vhost+_*.deb KVER=$(ls -1 /lib/modules/ | grep vhost) [ -f /boot/initrd.img-"$KVER" ] || sudo update-initramfs -c -k "$KVER" ls -lh /boot/initrd.img-"$KVER" Expect 30–60 MB. Then restore the original setting, so your existing 239 MB initrd can never be regenerated with different settings — the fallback must stay exactly as it is: sudo cp /etc/initramfs-tools/initramfs.conf.backup /etc/initramfs-tools/initramfs.conf Copy to the ESP Note which kernel file. /boot/vmlinuz-* is compressed; the bootloader here wants the raw arch/arm64/boot/Image. On my system that's 13 MB versus 38 MB — copy the wrong one and it won't boot. Everything here adds files. Nothing existing is touched: sudo mount /dev/nvme0n1p1 /mnt/esp sudo cp ~/cix-kernel/arch/arm64/boot/Image /mnt/esp/Image-vhost sudo cp /boot/initrd.img-"$KVER" /mnt/esp/initrd-vhost.img sync df -h /mnt/esp Add a menu entry Back up first, then derive the root partition from the running system rather than retyping a UUID: sudo cp /mnt/esp/GRUB/GRUB.CFG /mnt/esp/GRUB/GRUB.BAK ROOTSPEC=$(grep -o 'root=PARTUUID=[^ ]*' /proc/cmdline) echo "$ROOTSPEC" # must not be empty sudo tee -a /mnt/esp/GRUB/GRUB.CFG >/dev/null