Development:Setting up FEX

Build dependencies

Debian/Ubuntu dependencies

• git
• cmake
• ninja-build
• pkgconf
• ccache (Recommended if you're rebuilding constantly)
• clang
• llvm
• lld (For ThinLTO)
• binfmt-support (For AArch64 binfmt_misc support)
• libssl-dev
• python3-setuptools (For build scripts)
• g++-x86-64-linux-gnu (For building thunks)
• libgcc-XX-dev-i386-cross (For building thunks, version should match your g++-x86-64-linux-gnu package)
• libgcc-XX-dev-amd64-cross (For building thunks, version should match your g++-x86-64-linux-gnu package)
• nasm (only if building tests)
• python3-clang (for struct verifier)
• libstdc++-12-dev-i386-cross (for struct verifier)
• libstdc++-12-dev-amd64-cross (for struct verifier)
• libstdc++-12-dev-arm64-cross (for struct verifier)
• squashfs-tools (For squashfs rootfs support)
• squashfuse (For mounting the squashfs)
• libc-bin or glibc-tools (For unittests, needs catchsegv from this)
• libc6-dev-i386-amd64-cross (For FEXLinuxTests)
• lib32stdc++-XX-dev-amd64-cross (For FEXLinuxTests, version should match your g++-x86-64-linux-gnu package)
• qtdeclarative6-dev (For configuration GUI)

Gentoo dependencies

• dev-vcs/git
• dev-build/cmake
• dev-build/ninja
• dev-util/ccache (for speeding up repeated builds)
• llvm-core/clang
• llvm-core/lld
• llvm-core/llvm
• dev-lang/nasm (for tests)
• dev-libs/xxhash
• dev-libs/libfmt >= 11
• dev-qt/qtbase:6
• dev-qt/qtdeclarative:6
• sys-fs/squashfs-tools

Use gubbins

• >=media-libs/libglvnd-1.3.4 X
• dev-qt/qtbase:6 gui widgets
• sys-fs/squashfs-tools zstd

Fedora dependencies

• git
• cmake
• ninja-build
• pkg-config
• ccache
• clang
• lld
• llvm
• llvm-devel
• openssl-devel
• nasm
• python3-clang
• python3-setuptools
• squashfs-tools
• squashfuse
• erofs-fuse
• erofs-utils
• qt6-qtdeclarative-devel

Build Configuration

• Ensure release mode is enabled
• Disable test building
• Enable LTO in a release build
• Enable LLD to make LTO builds faster

 git clone --recurse-submodules https://github.com/FEX-Emu/FEX.git
 cd FEX
 mkdir Build
 cd Build
 CC=clang CXX=clang++ cmake -DCMAKE_INSTALL_PREFIX=/usr -DCMAKE_BUILD_TYPE=Release -DUSE_LINKER=lld -DENABLE_LTO=True -DBUILD_TESTING=False -DENABLE_ASSERTIONS=False -G Ninja ..
 ninja

Installation

 sudo ninja install

You can install a binfmt_misc handler for both 32bit and 64bit x86 execution directly from the environment. If you already have box86's 32bit binfmt_misc handler installed then make sure to uninstall their's first. Make sure to have run install prior to this, otherwise binfmt_misc will install an old handler even if the executable has been updated.

 sudo ninja binfmt_misc

Quirks

binfmt_misc problems

• Double check that binfmt_misc has worked
  • `ls /usr/share/binfmts/FEX*` should return files
  • `ls /var/lib/binfmts/FEX*` should return files
  • `/proc/sys/fs/binfmt_misc/status` should exist
    • If it doesn't exist then make sure your kernel has `CONFIG_BINFMT_MISC=y`
    • If it still doesn't exist then make sure systemd isn't disabling it `systemctl enable binfmt-support`

DESTDIR quirks

• When using DESTDIR, make sure the path is absolute instead of relative
  • FEX's thunks use subprojects because it needs to cross-compile
  • If DESTDIR is relative then the install paths of the subprojects go to the wrong location
  • Using $(pwd)/install works well

Runtime Configuration

FEXConfig is the application to use for runtime configuration setup.

CTRL+S saves the configuration.

Things to ensure to set

On AArch64 host, a rootfs path is mandatory, while it is optional on x86-64 host

• Core: JIT
• RootFS
• Silent Logging (Useful to get FEX information spam out of the way)
• Block Size (500 is a good default. Optimization passes may break things though)

More Details at: Follow the instructions here

RootFS generation

AArch64 hosts require a rootfs for running applications. FEX provides various x86 images that can be installed through FEXRootFSFetcher

For more detailed instructions see Development:Setting_up_RootFS

Setting up library forwarding

Follow the instructions here.

Binaries

• FEX
  • This is the emulator executable used to run x86 programs
• FEXInterpreter
  • This is a (deprecated) alias for FEX, provided for backwards compatibility
• FEXConfig
  • Used for configuring emulation from a Qt-based GUI
• FEXBash
  • Can be useful for starting a bash instance that is running under emulation
  • It's not a chroot! Don't use sudo inside of this environment!
• FEXMountDaemon
  • This is a mount daemon that will run in the background when you are using a squashfs based rootfs
  • Usually one one instance open at any given moment
  • It usually closes when FEX exits, but not always. See the problems section in setting up a rootfs
    • Has a ten second timeout window in case a another instance of FEX opens and uses it.

Debug binaries

• TestHarnessRunner
  • Used for running FEX's ASM unit tests
  • Runs some binary in a host x86_64 host environment or FEX and checks results
  • Useful for quickly creating ASM based unit tests
• IRLoader
  • Used for running FEX's IR based unit tests
  • Only runs through FEX and checks the results
  • Minimal number of tests
• UnitTestGenerator
  • This is supposed to be an x86-64 instruction generator
  • Has an understanding of x86-64 instruction encoding format
  • Generates hundreds of thousands of instructions
  • Used to be used with a lockstep runner to ensure correct behaviour
  • Largely unsupported, avoiding undefined behaviour is difficult and FEX doesn't need to match undefined behaviour
  • Lockstep hostrunner wasn't ever made to be bug free, hard to support
  • Doesn't ensure correct behaviour on AArch64 device