Development:Setting up RootFS

Setting up a RootFS

For AArch64 hosts you are required to have an x86-64 rootfs. With or without 32bit compatibility libraries.

This isn't strictly required for testing on x86-64 host environments but is useful for ensuring correct behaviour there.

Prereqs

An x86-64 PC OR qemu for chrooting

Getting Started

The easiest way to get started is to just download a prebuilt x86-64 rootfs from your favourite distribution and extract it to where you want.

• Ubuntu - http://cdimage.ubuntu.com/ubuntu-base/releases/
• Debian Debootstrap guide - https://wiki.debian.org/Debootstrap
  • Debootstrap is more involved and not explained here.

Using chroot you can then enter the rootfs and install applications that you desire for testing purposes. It's easiest to set this up on an x86-64 host and then copy it to an AArch64 device.

Cross architecture chroot

You need to have binfmt_misc setup for x86-64 support on your host system.

• ls /proc/sys/fs/binfmt_misc
  • If there are files in this folder then your host supports binfmt_misc

You then need to copy the qemu-x86_64-static over to the rootfs. This way your kernel can find the interpreter after chrooting.

• sudo apt-get install qemu-user-static
• cp /usr/bin/qemu-x86_64-static ${ROOTFS}/usr/bin/
• sudo chroot ${ROOTFS} qemu-x86_64-static /bin/bash

Careful. If you've installed FEX's binfmt_misc then chrooting can become broken. This is because FEX's binfmt_misc conflicts with qemu.

Additional TmpFS folders

Depending on what you're doing in the chroot, you may need or want some tmpfs folders that applications are expecting

• sudo mount -t proc /proc ${ROOTFS}/proc/
• sudo mount --rbind --make-rslave /sys ${ROOTFS}/sys/
• sudo mount --rbind --make-rslave /dev ${ROOTFS}/dev/
• sudo mount --rbind --make-rslave /dev/pts ${ROOTFS}/dev/pts/
• sudo mount --rbind --make-rslave /home ${ROOTFS}/home/

Then of course, make sure to unmount these partitions when you're done

• sudo umount ${ROOTFS}/proc/
• sudo umount -R ${ROOTFS}/sys/
• sudo umount -R ${ROOTFS}/dev/pts/
• sudo umount -R ${ROOTFS}/dev/
• sudo umount -R ${ROOTFS}/home/

Using the rootfs with FEX

If you're using FEXLoader then you can pass the full path of the rootfs with the -R option. Additionally you can put the rootfs folder inside of $HOME/.fex-emu/RootFS/<RootFS name>/ and FEX supports a "named" rootfs option

• eq: -R Ubuntu_2104 will use rootfs $HOME/.fex-emu/RootfS/Ubuntu_2104
• Read the Fex man page with man fex for more information

Setting RootFS with FEXConfig

On the first run of FEXConfig, select the "Load Default Options" configuration.

• Subsequent runs should select "Open Default"

FEXConfig will autopopulate the rootfs list with rootfs folders that exist in $HOME/.fex-emu/RootFS/

You can select one from the list or provide a full path in the RootFS: line to set it manually.

Make sure to select the File->Save option after the fact to save

Additional Steps

Running under WSL1

/etc/hosts and /etc/resolv.conf change on the host side. You'll need to copy those over to the rootfs if you want network functionality to work.

Ubuntu packages to install

Sample packages to install in the rootfs

• apt-get install pulseaudio libgles1 libgles2 libglx-mesa0 libgl1-mesa-dri libgl1-mesa-glx mesa-utils mesa-utils-extra

Pulseaudio specifically is required so pulseaudio will work when talking to the host pulseaudio

File deletion

Some files need to be deleted from the rootfs otherwise some applications will fail to work.

Note that deleting these files WILL break the rootfs and you won't be able to chroot in to it anymore.

A likely better route would be a script that moves these files and folders then brings them back when you want to chroot

Files to remove

• ${ROOTFS}/etc/hosts
• ${ROOTFS}/etc/resolv.conf
• ${ROOTFS}/etc/timezone
• ${ROOTFS}/etc/localtime
• ${ROOTFS}/etc/passwd

Folders to remove

• ${ROOTFS}/boot
• ${ROOTFS}/dev
• ${ROOTFS}/home
• ${ROOTFS}/media
• ${ROOTFS}/mnt
• ${ROOTFS}/proc
• ${ROOTFS}/root
• ${ROOTFS}/srv
• ${ROOTFS}/tmp
• ${ROOTFS}/var/cache/apt
• ${ROOTFS}/var/lib/apt
• ${ROOTFS}/sys
• ${ROOTFS}/opt

Building a Squashfs rootfs

SquashFS is a great way to do binary distribution of the rootfs.

SquashFS is currently only recommended for binary distribution. User should unpack it to not get dangling processes and mounts.

• Offers tooling to mount in userspace
• Supports ZSTD compression
• Is a readonly partition that matches FEX's use case

Tools that FEX needs to run squashfs

• squashfuse
• fusermount

Making the squashfs image

Making the squashfs image is very straightforward

• mksquashfs ${ROOTFS}/ Squashfs.sqsh -comp zstd

This creates a file called Squashfs.sqsh that contains the original rootfs that we generated above.

Using the squashfs image in FEX

• Copy the resulting sqsh file in to ${XDG_DATA_HOME}/.fex-emu/RootFS/
  • If ${XDG_DATA_HOME} doesn't exist then that will be replaced with ${HOME}
• Use FEXConfig to select the sqsh file as a named rootfs
• or pass the name in to the FEX_ROOTFS argument in the various ways FEX supports
• or the absolute path if you don't want it in the FEX RootFS folder

Extracting the squashfs

If you don't want to use squashfs for whatever reason then you can extract it in to the FEX RootFS folder and use the directory directly

• unsquashfs -f -d NewRootFS/ Squashfs.sqsh

Known problems with squashfs

FEX automatically mounts your squashfs file and can leave dangling squashfs mounts around in some instances.

• This is fairly rare but it can occur. Whenever an instance of FEX is run, it registers with the FEXMountDaemon
  • FEXMountDaemon then watches these instances of FEX to ensure they have exited before unmounting
  • There is roughly a ten second grace period of no instances of FEX running before unmounting
  • If an instance of FEX starts running then it will reuse the mount partition and reset the timeout
• To get rid of these dangling mounts if they do manage to manifest, use the following commands
  • fusermount -u /tmp/.FEXMount* ; rmdir /tmp/.FEXMount*
  • These don't require root to unmount
• In some cases the unmounting can end up returning busy status and not unmount
  • Check the output of `mount` for any /tmp/.FEXMount folders
  • This is due to either FEXMountDaemon or squashfuse still running in the background
  • Can force the application to exit
  • step 1: kill `pidof squashfuse`
  • This will attempt to send SIGTERM to all squashfuse applications, which will unmount the partition on terminate
  • Failing that, you can try to be more abusive
  • kill -9 `pidof FEXMountDaemon`
  • This will forcibly terminate FEXMountDaemon which should also take out the child mounted process
• If any instances of the mount still exists then you may need to kill more FEX processes.
  • Any FEXInterpreter process will end up keeping the folder mounted and keep it busy

strace breaks squashfs

• If you run FEX with `strace -f` then the process that mounts your rootfs loses its setuid bit.
  • This means you then can't mount the rootFS
• The only workaround is to not use a squashfs in this instance.