↓Skip to main content
more zeros than ones… more zeros than ones...
  1. Posts/

Firecracker FreeBSD

·2728 words·13 mins
Table of Contents

FreeBSD now available as a firecracker VM #

This last week Colin Percival posted about their work on getting FreeBSD running as a guest in Firecracker.

At the end of their post, some simple steps are shown to help you get going, however some are missing for us FreeBSD novices, my post here tries to fill those gaps.

Disclaimer: I am a novice with FreeBSD so this is probably not the most efficient way of getting a running Firecracker FreeBSD VM. The rootfs image I use is pretty large but I’ll post again once I’ve successfully build my own minimal rootfs.

The steps required are:

  • build a kernel (needs to be done on a working FreeBSD system)
  • build a rootfs - somehow…
  • build the custom branch of firecracker
  • put them all together and run it

I’ll try to explain the gaps, things like getting a FreeBSD machine up and running so we can build the custom kernel, get a root UFS filesystem to enable the firecracker VM to run, and putting it all together with a config file so at the end you have a speedy FreeBSD Firecracker VM.

Kernel build environment #

For this post, I use a throw away KVM virtual machine to do the kernel build. You can acquire the qcow2 image from the FreeBSD releases page.

You’ll need to use the xz tool, by default it will delete the archive once it’s expanded, unless you pass the keep flag like so:

 xz -k -d /tmp/FreeBSD-13.1-RELEASE-amd64.qcow2.xz

TIP: make a copy of the qcow2 file, if you follow along you will need to grow the image to be able to build a custom kernel (as there is not enough free space).

Make space for the kernel build #

The VM disk image doesn’t come with enoguh space to build a custom kernel, so this is how I made more disk space.

On the Linux host:

qemu-img resize FreeBSD-13.1-RELEASE-amd64.qcow2 +10G

# now check the space has increased...

qemu-img info FreeBSD-13.1-RELEASE-amd64.qcow2

The qcow2 file now has the space to allow the contained UFS filesystem to grow. To do this we’ll need to boot the VM. I used a quick VM command to get a FreeBSD machine I could use to build a custom kernel for the Firecracker VMs.

qemu-system-x86_64 --enable-kvm FreeBSD-13.1-RELEASE-amd64.qcow2 -net user,hostfwd=tcp::8022-:22 -net nic -monitor stdio

This command is the old style, the recommendation is to move over to passing the hostfwd param to netdev but when I tried it I couldn’t SSH into the machine so was stuck withthe QEMU window. It’s added to my todo list of things I need to learn how to fix.

Basically, that command will create a single CPU VM and bind the host’s port 8022 to the guest’s port 22 (which is SSH). I wanted this so I could use scp to copy the custom kernel onto my Linux host.

Anyway, onto growing the UFS rootfs, log into the throwaway VM and follow these steps to add all (nearly) of the space we just added to the qcow2 onto /:


root@freebsd:/usr # gpart show ada0
=>       3  10552571  ada0  GPT  (15G) [CORRUPT]
         3       123     1  freebsd-boot  (62K)
       126     66584     2  efi  (33M)
     66710   2097152     3  freebsd-swap  (1.0G)
   2163862   8388712     4  freebsd-ufs  (4.0G)

The drive shows as being corrupt, to fix that we just need to use the recover option.



root@freebsd:/usr # gpart recover ada0
ada0 recovered


root@freebsd:/usr # gpart show ada0
=>       3  31524085  ada0  GPT  (15G)
         3       123     1  freebsd-boot  (62K)
       126     66584     2  efi  (33M)
     66710   2097152     3  freebsd-swap  (1.0G)
   2163862   8388712     4  freebsd-ufs  (4.0G)
  10552574  20971514        - free -  (10G)

Now we want to add the free space to the UFS root partition (4)


root@freebsd:/usr # gpart resize -i 4 -s 14G -a 4k ada0
ada0p4 resized


root@freebsd:/usr # gpart show ada0
=>       3  31524085  ada0  GPT  (15G)
         3       123     1  freebsd-boot  (62K)
       126     66584     2  efi  (33M)
     66710   2097152     3  freebsd-swap  (1.0G)
   2163862  29360122     4  freebsd-ufs  (14G)
  31523984       104        - free -  (52K)

You can skip this next step, I ran it to discover there was more to do


root@freebsd:/dev # df -h
Filesystem         Size    Used   Avail Capacity  Mounted on
/dev/gpt/rootfs    3.9G    3.8G   -255M   107%    /
devfs              1.0K    1.0K      0B   100%    /dev
/dev/gpt/efiesp     32M    875K     31M     3%    /boot/efi

Now grow the filesystem into the space we attached to the UFS partition.


root@freebsd:/dev # growfs /dev/gpt/rootfs
Device is mounted read-write; resizing will result in temporary write suspension for /.
It's strongly recommended to make a backup before growing the file system.
OK to grow filesystem on /dev/gpt/rootfs, mounted on /, from 4.0GB to 14GB? [yes/no] yes
super-block backups (for fsck_ffs -b #) at:
 8979008, 10261696, 11544384, 12827072, 14109760, 15392448, 16675136, 17957824, 19240512, 20523200, 21805888, 23088576, 24371264, 25653952, 26936640, 28219328


root@freebsd:/dev # df -h
Filesystem         Size    Used   Avail Capacity  Mounted on
/dev/gpt/rootfs     14G     3.8G  10.2G    73%    /
devfs              1.0K    1.0K      0B   100%    /dev
/dev/gpt/efiesp     32M    875K     31M     3%    /boot/efi

You may have slightly different figures as I ran this after I ran out of space as you can probably see from the root partition being complete full before I resized it. You will have abou a gigabyte free of disk space before the reszie.

Build the kernel #

Now there should be space, we need to enable SSH (so we can copy of the built kernel), pull the source code and build it as per Colin’s instructions.

Just to recap, to run the VM I used:

qemu-system-x86_64 --enable-kvm FreeBSD-13.1-RELEASE-amd64.qcow2 -net user,hostfwd=tcp::8022-:22 -net nic -monitor stdio

Let’s enable SSH so we can later use it to copy files off the VM.

SSH #

To enable SSH, I ran the following:

service sshd status

vi /etc/rc.conf

add the line sshd_enable=YES to the file and save nd quit :wq.

service sshd start

Shutdown shutdown -p now and start again with port mapping to allow host to ssh to guest, like so:

qemu-system-x86_64 --enable-kvm FreeBSD-13.1-RELEASE-amd64.qcow2 -net user,hostfwd=tcp::8022-:22 -net nic

On your host machine, run ssh root@localhost -p8022 to prove it’s working (when the throwaway VM is running).

The instructions so far assume using root, you can however create your own user and precisely assign permissions. As I said, this is a throw away VM for me so I’m just here to build a custom kernel, copy it out to my Linux host and then delete this VM.

Pull kernel source code and build it #

We are now getting to the point where we can follow Colin’s instructions, we just need to install git first.

pkg install git

# now Colin's instructions...

git clone https://git.freebsd.org/src.git /usr/src

cd /usr/src && make buildkernel TARGET=amd64 KERNCONF=FIRECRACKER

You might want to run the VM with more cores than I did as it took quite a while to build. To do that you’ll need to modify the qemu-system-x86_64 command and I think you might need to pass another flag to the make buildkernel command but I didn’t do it so can’t help you there.

Copy the kernel back to your host #

I created a new user as root wasn’t working and I just wanted to copy the kernel file and dump this VM. I used adduser and added the created user to wheel (the second question about groups when following the adduser prompts).

The VM needs to stay running but now we will be running commands on our Linux host.

First create a custom ssh key ssh-keygen on your host and copy it over to the throwaway VM using the new user’s creds (you can scp and use the password method if you want, I just didn’t as I was experimenting with how to do all of this).

 ssh-copy-id -i ~/.ssh/user2id.pub -p 8022 user2@localhost

You can now copy the kernel off the throwaway VM and onto your Linux Host that will be running firecracker.

scp -P 8022 -i ~/.ssh/user2id user2@localhost:/usr/obj/usr/src/amd64.amd64/sys/FIRECRACKER/kernel ./FreeBSD_kernel

With that, the throwaway VM can be shutdown shutdown -p now and thrown away. We will have a file called FreeBSD_kernel which can be used with firecracker. Keep it safe!

“Building” a rootfs #

This is the quickest way of getting a filesystem that I could think of without examining how to build one from scratch (which will most likely need a running FreeBSD system to create).

I decided to create a raw filesystem from the FreeBSD release qcow2 image (the original one, not the 1GB one we resized). I tried to use firecracker with the qcow2 file but without success… So converting it to a raw file seemed like a good alternative.

qemu-img convert FreeBSD-13.1-RELEASE-amd64.qcow2 -O raw disk.ufs

With this, you will get a 5Gb (so not exactly great, but this is just discovery) file caled disk.ufs that contains four partitions, with the rootfs one being the 4th, which turns out to be called vtdb0p4 when mounted to a firecracker VM.

Building Firecracker with FreeBSD support #

You don’t need to use the /usr/src directory I have here.

cd /usr/src/

git clone -b pvh-v3 https://github.com/cperciva/firecracker.git freebsd_firecracker

cd freebsd_firecracker

./tools/devtool build

You should now have a custom firecracker executable that is capable of running your FreeBSD kernel and rootfs, the binary will be located at /usr/src/freebsd_firecracker/build/cargo_target/x86_64-unknown-linux-musl/debug/firecracker if you followed along with my path from the above comamnds.

Running your first FreeBSD firecracker VM #

Firecracker can be run with multiple commands via it’s API server or by passing a configuration file. Here we will be using a config file but the API method should work just as well.

I based my config file off one of the examples from the firecracker github repo.

I’ve modified the values for:

  • kernel_image_path - I just referenced the path (current directory) of the kernel I built.
  • boot_args - this is from Colin’s blog but the ufs path is altered to match the contents of the rootfs image we obtained earlier
  • path_on_host - again, a path but this time to the rootfs file on disk.

{
  "boot-source": {
    "kernel_image_path": "FreeBSD_kernel",
    "boot_args": "console=ttyS0 reboot=k panic=1 pci=off vfs.root.mountfrom=ufs:/dev/vtbd0p4",
    "initrd_path": null
  },
  "drives": [
    {
      "drive_id": "rootfs",
      "path_on_host": "disk.ufs",
      "is_root_device": true,
      "partuuid": null,
      "is_read_only": false,
      "cache_type": "Unsafe",
      "io_engine": "Sync",
      "rate_limiter": null
    }
  ],
  "machine-config": {
    "vcpu_count": 2,
    "mem_size_mib": 1024,
    "smt": false,
    "track_dirty_pages": false
  },
  "balloon": null,
  "network-interfaces": [],
  "vsock": null,
  "logger": null,
  "metrics": null,
  "mmds-config": null
}

We can now run firecracker with the following command to plug in our config:

firecracker --api-sock /tmp/firecracker.socket --config-file vm-config.json

This assumes:

  • the custom firecracker executale is on your path
  • you are in the directory containing the vm-config.json file, custom kernel and rootfs file.

You can use links or you can use absolute paths in your config file as well, I’ve just done it all in the same directory as I’m epxerimenting and want everything together right now.

If successful you should see something like this:

GDB: no debug ports present
KDB: debugger backends: ddb
KDB: current backend: ddb
---<<BOOT>>---
Copyright (c) 1992-2022 The FreeBSD Project.
Copyright (c) 1979, 1980, 1983, 1986, 1988, 1989, 1991, 1992, 1993, 1994
	The Regents of the University of California. All rights reserved.
FreeBSD is a registered trademark of The FreeBSD Foundation.
FreeBSD 14.0-CURRENT #0 main-n258661-713efe05429: Thu Oct 20 03:05:12 UTC 2022
    root@freebsd:/usr/obj/usr/src/amd64.amd64/sys/FIRECRACKER amd64
FreeBSD clang version 13.0.0 (git@github.com:llvm/llvm-project.git llvmorg-13.0.0-0-gd7b669b3a303)
WARNING: WITNESS option enabled, expect reduced performance.
CPU: Intel(R) Xeon(R) Processor @ 2.60GHz (2600.00-MHz K8-class CPU)
  Origin="GenuineIntel"  Id=0x506e3  Family=0x6  Model=0x5e  Stepping=3
  Features=0x1f83fbff<FPU,VME,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CMOV,PAT,PSE36,MMX,FXSR,SSE,SSE2,SS,HTT>
  Features2=0xfffab223<SSE3,PCLMULQDQ,VMX,SSSE3,FMA,CX16,PDCM,PCID,SSE4.1,SSE4.2,x2APIC,MOVBE,POPCNT,TSCDLT,AESNI,XSAVE,OSXSAVE,AVX,F16C,RDRAND,HV>
  AMD Features=0x2c100800<SYSCALL,NX,Page1GB,RDTSCP,LM>
  AMD Features2=0x121<LAHF,ABM,Prefetch>
  Structured Extended Features=0x9c67ab<FSGSBASE,TSCADJ,BMI1,AVX2,SMEP,BMI2,ERMS,INVPCID,NFPUSG,MPX,RDSEED,ADX,SMAP,CLFLUSHOPT>
  Structured Extended Features2=0x4<UMIP>
  Structured Extended Features3=0xac000400<MD_CLEAR,IBPB,STIBP,ARCH_CAP,SSBD>
  XSAVE Features=0xf<XSAVEOPT,XSAVEC,XINUSE,XSAVES>
  IA32_ARCH_CAPS=0x4c<RSBA,SKIP_L1DFL_VME>
  AMD Extended Feature Extensions ID EBX=0x100d000<IBPB,IBRS,STIBP,SSBD>
  VT-x: (disabled in BIOS) PAT,HLT,MTF,PAUSE,EPT,UG,VPID
  TSC: P-state invariant
Hypervisor: Origin = "KVMKVMKVM"
real memory  = 1073741824 (1024 MB)
avail memory = 1015017472 (967 MB)
MPTable: <FC       000000000000>
Event timer "LAPIC" quality 600
FreeBSD/SMP: Multiprocessor System Detected: 2 CPUs
FreeBSD/SMP: 1 package(s) x 2 core(s)
random: registering fast source Intel Secure Key RNG
random: fast provider: "Intel Secure Key RNG"
arc4random: WARNING: initial seeding bypassed the cryptographic random device because it was not yet seeded and the knob 'bypass_before_seeding' was enabled.
ioapic0: MADT APIC ID 3 != hw id 0
ioapic0: Assuming intbase of 0
ioapic0 <Version 1.1> irqs 0-23
Launching APs: 1
random: entropy device external interface
virtio_mmio0: <VirtIO MMIO adapter> at iomem 0xd0000000-0xd0000fff irq 5
vtblk0: <VirtIO Block Adapter> on virtio_mmio0
vtblk0: 5152MB (10552576 512 byte sectors)
kvmclock0: <KVM paravirtual clock>
Timecounter "kvmclock" frequency 1000000000 Hz quality 975
kvmclock0: registered as a time-of-day clock, resolution 0.000001s
aesni0: <AES-CBC,AES-CCM,AES-GCM,AES-ICM,AES-XTS>
cpu0
cpu1
isa0: <ISA bus>
ns8250: UART FCR is broken
ns8250: UART FCR is broken
uart0: <Non-standard ns8250 class UART with FIFOs> at port 0x3f8 irq 4 flags 0x10 on isa0
ns8250: UART FCR is broken
uart0: console (9600,n,8,1)
Timecounter "TSC-low" frequency 1295998999 Hz quality 1000
Timecounters tick every 10.000 msec
random: unblocking device.
Trying to mount root from ufs:/dev/vtbd0p4 []...
WARNING: WITNESS option enabled, expect reduced performance.
Setting hostuuid: 3a006aa7-524a-11ed-b1e3-9da069ca1f6d.
Setting hostid: 0x1406eba8.
Starting file system checks:
/dev/vtbd0p4: FILE SYSTEM CLEAN; SKIPPING CHECKS
/dev/vtbd0p4: clean, 128277 free (21 frags, 16032 blocks, 0.0% fragmentation)
/dev/gpt/efiesp: FILESYSTEM CLEAN; SKIPPING CHECKS
Mounting local filesystems:.
Setting up harvesting: PURE_RDRAND,[CALLOUT],[UMA],[FS_ATIME],SWI,INTERRUPT,NET_NG,[NET_ETHER],NET_TUN,MOUSE,KEYBOARD,ATTACH,CACHED
Feeding entropy: .
ELF ldconfig path: /lib /usr/lib /usr/lib/compat
32-bit compatibility ldconfig path: /usr/lib32
Setting hostname: freebsd.
lo0: link state changed to UP
Starting Network: lo0.
lo0: flags=8049<UP,LOOPBACK,RUNNING,MULTICAST> metric 0 mtu 16384
	options=680003<RXCSUM,TXCSUM,LINKSTATE,RXCSUM_IPV6,TXCSUM_IPV6>
	inet6 ::1 prefixlen 128
	inet6 fe80::1%lo0 prefixlen 64 scopeid 0x1
	inet 127.0.0.1 netmask 0xff000000
	groups: lo
	nd6 options=23<PERFORMNUD,ACCEPT_RTADV,AUTO_LINKLOCAL>
Starting devd.
add host 127.0.0.1: gateway lo0 fib 0: route already in table
add host ::1: gateway lo0 fib 0: route already in table
add net fe80::: gateway ::1
add net ff02::: gateway ::1
add net ::ffff:0.0.0.0: gateway ::1
add net ::0.0.0.0: gateway ::1
Updating /var/run/os-release done.
Creating and/or trimming log files.
Clearing /tmp (X related).
Updating motd:.
Starting syslogd.
Mounting late filesystems:.
Starting sendmail_submit.
Starting sendmail_msp_queue.
Starting cron.
Starting background file system checks in 60 seconds.

Sat Oct 22 21:07:37 UTC 2022

FreeBSD/amd64 (freebsd) (ttyu0)

login: root
Oct 22 21:07:45 freebsd login[529]: ROOT LOGIN (root) ON ttyu0
Last login: Sat Oct 22 21:04:02 on ttyu0
FreeBSD 14.0-CURRENT #0 main-n258661-713efe05429: Thu Oct 20 03:05:12 UTC 2022     root@freebsd:/usr/obj/usr/src/amd64.amd64/sys/FIRECRACKER

Welcome to FreeBSD!

Release Notes, Errata: https://www.FreeBSD.org/releases/
Security Advisories:   https://www.FreeBSD.org/security/
FreeBSD Handbook:      https://www.FreeBSD.org/handbook/
FreeBSD FAQ:           https://www.FreeBSD.org/faq/
Questions List: https://lists.FreeBSD.org/mailman/listinfo/freebsd-questions/
FreeBSD Forums:        https://forums.FreeBSD.org/

Documents installed with the system are in the /usr/local/share/doc/freebsd/
directory, or can be installed later with:  pkg install en-freebsd-doc
For other languages, replace "en" with a language code like de or fr.

Show the version of FreeBSD installed:  freebsd-version ; uname -a
Please include that output and any error messages when posting questions.
Introduction to manual pages:  man man
FreeBSD directory layout:      man hier

To change this login announcement, see motd(5).
root@freebsd:~ #

All the FreeBSDs belong to you, have fun.

Potential Issues #

As I foolishly tried to get the qcow2 image to work as a rootfs without any alteration (it was worth a shot, I thought at least), I’ve seen a couple of issues with the rootfs.

Trying to mount root from ufs:/dev/ada0p4 []...
WARNING: WITNESS option enabled, expect reduced performance.
mountroot: waiting for device /dev/ada0p4...
Mounting from ufs:/dev/ada0p4 failed with error 19.

Loader variables:
  vfs.root.mountfrom=ufs:/dev/ada0p4

Manual root filesystem specification:
  <fstype>:<device> [options]
      Mount <device> using filesystem <fstype>
      and with the specified (optional) option list.

    eg. ufs:/dev/da0s1a
        zfs:zroot/ROOT/default
        cd9660:/dev/cd0 ro
          (which is equivalent to: mount -t cd9660 -o ro /dev/cd0 /)

  ?               List valid disk boot devices
  .               Yield 1 second (for background tasks)
  <empty line>    Abort manual input

mountroot>

The above was caused by assuming I should use ada0p4 rather than the actual value which should be vtbd0p4. I found it by trial and error, slowly and painfully…

If you have issues with mounting the filesystem and get to a similar looking mountroot prompt, you can just type ? then to see what mount points have been detected.

mountroot>?        

List of GEOM managed disk devices:
  diskid/DISK-6631009484011p4 diskid/DISK-6631009484011p3 diskid/DISK-6631009484011p2 diskid/DISK-6631009484011p1 ufs/rootfs ufsid/627cd95609b1c293 gptid/9658b8ee-d1d9-11ec-b6df-0cc47ad8b808 gpt/rootfs gptid/9658b8eb-d1d9-11ec-b6df-0cc47ad8b808 gpt/swapfs msdosfs/EFISYS gptid/9658b8e8-d1d9-11ec-b6df-0cc47ad8b808 gpt/efiesp gptid/9658b8e2-d1d9-11ec-b6df-0cc47ad8b808 gpt/bootfs diskid/DISK-6631009484011 vtbd0p4 vtbd0p3 vtbd0p2 vtbd0p1 vtbd0

Admitedly, it’s not the easiest chunk of text to read, but it will show you each mount point that is available. If you can’t see vtbd0p4 then there is something wrong with the rootfs you’ve configured (maybe a config issue, maybe the filesystem hasn’t been created correctly).

I knew it was the 4th partition from the gpart work we did ealier as the first 3 partitions on the GPT partitioned disk were boot, efi & swap with the 4th being ufs which is the filesystem format that FreeBSD uses in the release VM image.