FEX-Emu Wiki - User contributions [en]

Development:Setting up RootFS

2026-01-13T14:47:15Z

Neobrain:

[[Category:Development]]
= Setting up a RootFS =
'''For AArch64 hosts you are required to have an x86-64 rootfs'''. With or without 32bit compatibility libraries.

This isn't strictly required for testing on x86-64 host environments but is useful for ensuring correct behaviour there.

= Quick Setup with FEXRootFSFetcher =
FEX comes with a FEXRootFSFetcher tool that can download an official preconfigured rootfs from the FEX servers.
This tool operates as either a terminal application or a Zenity application depending on if it is launched from terminal or not.

This tool will walk you through selecting a RootFS for FEX:
* If your host OS matches one in the download list then it will recommend that one
* After selecting the RootFS, it will download it
** This downloads to the default location in '''$HOME/.fex-emu/RootFS/'''
* After downloading the RootFS the tool will ask if you want to use the SquashFS file directly or extract it
* It will then ask if you want to set the downloaded RootFS as your default
** This writes to the default configuration in '''$HOME/.fex-emu/Config.json'''

You can skip the rest of the guide if you used FEXRootFSFetcher to download the rootfs.

== Troubleshooting ==
* The squashfs image will fail to extract if you don't have unsquashfs installed
** Install the squashfs-tools package to get this application

* '''SquashFS RootFS file corrupted?'''
** Rerun the tool, select the rootfs from the list, and if the rootfs already exists it will ask to verify
** If verification fails it will ask if you want to redownload
* '''Want to manually check the hash?'''
** Run the rootfs file directly through the tool to get a hash
** '''FEXRootFSFetcher $HOME/.fex-emu/RootFS/Ubuntu_21_10.sqsh'''
** Ubuntu_21_10.sqsh has hash: bf9507fbdbaec2cf
* '''Want to use the Zenity UI but still launch through terminal?'''
** Force Zenity from terminal with '''FEXRootFSFetcher 1> /dev/null'''
* '''What if my Linux distro isn't listed?'''
** Don't worry, select one that is most similar to yours or newest
** This will work in most cases
* '''How can I trust these images?'''
** These images are generated by scripts living in our public repo
** If you want to rebuild them yourself, you are free to do so.
** https://github.com/FEX-Emu/RootFS
* '''I want to add a distro'''
** Follow how our current rootfs is generated and add your own
** Open a pull request to add the rootfs setup
** Contact us on the official FEX-Emu Discord to get the RootFS added to the server.
* '''Where is the raw list of images at?'''
** The image links live on our website at the following link
** https://rootfs.fex-emu.gg/RootFS_links.json
** This file is in a format that our tool understands
** Generally the format is:
*** Distro Name
*** Distro Version
*** Human readable name
*** Raw rootfs link
*** xxhash based file hash

= Making a custom rootfs prerequisites =
An x86-64 PC or FEX-Emu installed for chrooting

= Getting Started =
The easiest way to get started is to just download a prebuilt x86-64 rootfs from your favourite distribution and extract it to where you want.
* Ubuntu - http://cdimage.ubuntu.com/ubuntu-base/releases/
* Debian Debootstrap guide - https://wiki.debian.org/Debootstrap
** Debootstrap is more involved and not explained here.

Using chroot you can then enter the rootfs and install applications that you desire for testing purposes.
It's easiest to set this up on an x86-64 host and then copy it to an AArch64 device.

== Cross architecture chroot ==
'''You need to have binfmt_misc setup for x86-64 support on your host system.'''
* ls /proc/sys/fs/binfmt_misc
** If there are files in this folder then your host supports binfmt_misc
* Once you have binfmt_misc running and FEX-Emu installed in to it, you can chroot in to an x86-64 rootfs

== Chrooting into FEX-provided RootFS ==
* If you're using a FEX-provided RootFS, we provide a script to handle this:
** To enter, use '''./chroot.py chroot''' inside the root directory of the RootFS
** When you finish, make sure to exit the chroot cleanly
If the chroot is working, the remaining steps of this guide can safely be skipped.

* Issues
** As of January 12th, 2026, Fedora packages for FEX-Emu are a few months out of date.
** This causes the chroot script to fail, saying "Host program 'FEX' isn't available and is required!".
** The solution is to copy "/usr/bin/FEXInterpreter" to "/usr/bin/FEX".

== Chrooting into custom RootFS ==
* If you're setting up your own rootfs then you need to follow the binding steps that are following.
** We have a script that if you know what you're doing you can cherry pick parts from [https://github.com/FEX-Emu/RootFS/blob/main/unbreak_chroot.sh here]

= Additional TmpFS folders =
Depending on what you're doing in the chroot, you may need or want some tmpfs folders that applications are expecting

* sudo mount -t proc /proc ${ROOTFS}/proc/
* sudo mount --rbind --make-rslave /sys ${ROOTFS}/sys/
* sudo mount --rbind --make-rslave /dev ${ROOTFS}/dev/
* sudo mount --rbind --make-rslave /dev/pts ${ROOTFS}/dev/pts/
* sudo mount --rbind /lib/ld-linux-aarch64.so.1 ${ROOTFS}/lib/ld-linux-aarch64.so.1
* sudo mount --rbind /lib/aarch64-linux-gnu ${ROOTFS}/lib/aarch64-linux-gnu

Then of course, make sure to unmount these partitions when you're done

* sudo umount ${ROOTFS}/proc/
* sudo umount -R ${ROOTFS}/sys/
* sudo umount -R ${ROOTFS}/dev/pts/
* sudo umount -R ${ROOTFS}/dev/
* sudo umount -R ${ROOTFS}/lib/ld-linux-aarch64.so.1
* sudo umount -R ${ROOTFS}/lib/aarch64-linux-gnu

= Using the RootFS with FEX =

Run the graphical FEXConfig tool to select and change the active RootFS. Any file or folder in $HOME/.fex-emu/RootFS will be detected automatically.

= Additional Steps for building a custom rootfs =
== Running under WSL1 ==
/etc/hosts and /etc/resolv.conf change on the host side. You'll need to copy those over to the rootfs if you want network functionality to work.

== Ubuntu packages to install ==
Sample packages to install in the rootfs
* apt-get install pulseaudio libgles1 libgles2 libglx-mesa0 libgl1-mesa-dri libgl1-mesa-glx mesa-utils mesa-utils-extra
Pulseaudio specifically is required so pulseaudio will work when talking to the host pulseaudio

== File deletion ==
Some files need to be deleted from the rootfs otherwise some applications will fail to work.

Note that deleting these files '''WILL''' break the rootfs and you won't be able to chroot in to it anymore.

A likely better route would be a script that moves these files and folders then brings them back when you want to chroot

=== Files to remove ===
* ${ROOTFS}/etc/hosts
* ${ROOTFS}/etc/resolv.conf
* ${ROOTFS}/etc/timezone
* ${ROOTFS}/etc/localtime
* ${ROOTFS}/etc/passwd

=== Folders to remove ===
* ${ROOTFS}/boot
* ${ROOTFS}/dev
* ${ROOTFS}/home
* ${ROOTFS}/media
* ${ROOTFS}/mnt
* ${ROOTFS}/proc
* ${ROOTFS}/root
* ${ROOTFS}/srv
* ${ROOTFS}/tmp
* ${ROOTFS}/var/cache/apt
* ${ROOTFS}/var/lib/apt
* ${ROOTFS}/sys
* ${ROOTFS}/opt

== Creating a Squashfs rootfs from custom rootfs ==
SquashFS is a great way to do binary distribution of the rootfs. If you have created a rootfs and you want to roll it in to a squashfs like the FEX-Emu provided images then this is information on how to do that.

'''SquashFS is currently only recommended for binary distribution. User should unpack it to not get dangling processes and mounts.'''
* Offers tooling to mount in userspace
* Supports ZSTD compression
* Is a readonly partition that matches FEX's use case

=== Tools that FEX needs to run squashfs ===
* squashfuse
* fusermount

=== Making the squashfs image ===
Making the squashfs image is very straightforward
* mksquashfs ${ROOTFS}/ Squashfs.sqsh -comp zstd

This creates a file called Squashfs.sqsh that contains the original rootfs that we generated above.

=== Using the squashfs image in FEX ===
* Copy the resulting sqsh file in to ${XDG_DATA_HOME}/.fex-emu/RootFS/
** If ${XDG_DATA_HOME} doesn't exist then that will be replaced with ${HOME}
* Use FEXConfig to select the sqsh file as a named rootfs
* or pass the name in to the FEX_ROOTFS argument in the various ways FEX supports
* or the absolute path if you don't want it in the FEX RootFS folder

=== Extracting the squashfs ===
If you don't want to use squashfs for whatever reason then you can extract it in to the FEX RootFS folder and use the directory directly
* unsquashfs -f -d NewRootFS/ Squashfs.sqsh

=== If you want to install packages in the squashfs ===
The squashfs image is immutable and can't be modified.
* Extract the squashfs first
* Change the working directory to the rootfs
* execute `unbreak_chroot.sh`
* `apt-get update`
* `apt-get install <Whatever you want>`
* `exit` to leave the chroot
* Use the new extracted chroot directly, or follow the previous steps to recompress it to a squashfs format.

=== Known problems with squashfs ===
'''FEX automatically mounts your squashfs file and can leave dangling squashfs mounts around in some instances.'''
* '''This is fairly rare but it can occur. Whenever an instance of FEX is run, it registers with the FEXMountDaemon'''
** FEXMountDaemon then watches these instances of FEX to ensure they have exited before unmounting
** There is roughly a ten second grace period of no instances of FEX running before unmounting
** If an instance of FEX starts running then it will reuse the mount partition and reset the timeout
* To get rid of these dangling mounts if they do manage to manifest, use the following commands
** fusermount -u /tmp/.FEXMount* ; rmdir /tmp/.FEXMount*
** These don't require root to unmount
* In some cases the unmounting can end up returning busy status and not unmount
** Check the output of `mount` for any /tmp/.FEXMount folders
** This is due to either FEXMountDaemon or squashfuse still running in the background
** Can force the application to exit
** step 1: kill `pidof squashfuse`
** This will attempt to send SIGTERM to all squashfuse applications, which will unmount the partition on terminate
** Failing that, you can try to be more abusive
** kill -9 `pidof FEXMountDaemon`
** This will forcibly terminate FEXMountDaemon which should also take out the child mounted process
* If any instances of the mount still exists then you may need to kill more FEX processes.
** Any ''FEX'' process will end up keeping the folder mounted and keep it busy

'''strace breaks squashfs'''
* If you run FEX with `strace -f` then the process that mounts your rootfs loses its setuid bit.
** This means you then can't mount the rootFS
* The only workaround is to not use a squashfs in this instance.
* Alternatively you can spin up a FEXBash instance to force squashfs to be mounted. Then strace a different FEX instance which will use the mounted squashfs location.

Development:Adding Config Options

2025-09-22T14:16:24Z

Neobrain: Fix broken list formatting

[[Category:Development]]
Configuration options are programatically generated from json.

This file lives in `FEXCore/Source/Interface/Config/Config.json`

= Options section =
To add a configuration option that can be set from JSON or from environment variables, you need to add the option to this file.

Currently FEX has six groups to group configs
* CPU
* Emulation
* Debug
* Logging
* Hacks
* Misc

== Configuration option object ==
Each JSON object under a group becomes a configuration.

'''Object Layout'''
* Key:
** Becomes the C++ enum. eg. 'Multiblock' becomes `FEXCore::Config::ConfigOption::CONFIG_MULTIBLOCK`
** Also used for the environment variable. Becomes `FEX_MULTIBLOCK`
** If the object type is boolean then a negated argument also is generated. `--no-multiblock`

* Type:
** Can be `[bool, uint8, int32, uint32, uint64, str, strarray]`
** This is what the argument type ends up being once loaded from a configuration backing

* Default:
** Default value that matches this object
** Used to generate code so needs to handle being codified. Compilation will fail otherwise
** If Type is str or strarray then this string will be automatically wrapped with double-quotes when generated

* TextDefault:
** If the `Default` option which is used in code can't be represented by text. Like an enum being used.
** eg. `Core` has a `Default` which is an enum. `TextDefault` is set to `irjit` for the user to handle this case

* Choices:
** Allows us to represent the textual options that a argument can accept
** eg. `Core` supports `[ "irint", "irjit", "host" ]`

* ArgumentHandler:
** If an argument option needs some special handling to be parsed then this points to a C++ function in the argument loader
** `Core` uses it to convert the textual representation in to an enum

* Desc:
** Short description of what this option does
** This is used to generate the text in the man pages

== UnnamedOptions section ==
These are configuration options that can '''NOT''' be set by user means. There won't be any Environment variable or json helpers generated to load these.

These need to exist as a way to have internal configuration options existing in the same global space as user options. Allows us to pull internal configuration options the same way as user configurations.

These can exist under any group config. Currently only `Misc` is used. Only three things need to be set for these
* Key: This is only used to generate the enum in code
* Type: Same supported types as the regular options
* Default: The default value of this configuration

== Getting configurations in FEX code ==
FEX provides a `FEX_CONFIG_OPT` define to help developers get configuration options.
* FEX_CONFIG_OPT(name, enum)
** eg: `FEX_CONFIG_OPT(Cores, THREADS);`
** This will load the THREADS configuration option from the configuration directory and save the value in to local value called `Cores`
* Gotchas:
** Construction of the object queries the the global configuration layer. This can be slow so be careful with its use.
** Upon construction of the `FEXCore::Config::Value<T>` object it will pull the string configuration
** Then it will convert that value in to whatever type the value needs
** In high performance code, keep constructor instantiation outside of the hot code path
** Construct the configuration inside of a class definition so it only pulls the configuration once
* Caveats:
** If the configuration changes then this doesn't pick up the change
** Configuration isn't meant to change live in FEX so this is typically a non-issue
** Frontend object loader needs to be careful with its configuration loading to ensure the meta layer stays in sync
** ELFLoader.cpp for examples here

Development:Debugging Crash

2025-09-22T14:13:23Z

Neobrain: Remove screenshot of outdated FEXConfig interface

== Getting Started ==
* Debug an application with `gdb --args FEX <application full path>`
* Under GDB make sure to do `handle SIGBUS SIGILL SIG63 noprint`
** We use signals for various things, check out [[Development:Debugging_FEX_with_Signals|Here]] for more information

== Crash in emulated/JIT code ==
Walking through debugging a simple test application that is crashing.
$ gdb --args FEX ./sigsegv_test
Reading symbols from FEX...
(gdb) r
Starting program: /usr/bin/FEX ./sigsegv_test
[Thread debugging using libthread_db enabled]
Using host libthread_db library "/lib/aarch64-linux-gnu/libthread_db.so.1".
[New Thread 0x7fccb75f30 (LWP 90107)]

Thread 2 "FEX received signal SIGSEGV, Segmentation fault.
[Switching to Thread 0x7fccb75f30 (LWP 90107)]
0x0000007fccfb9ec8 in ?? ()

* Okay, we have a sigsegv. Let's double check that it is JIT code (aka, guest emulated code)
(gdb) disas $pc,+32
Dump of assembler code from 0x7fe26ebc20 to 0x7fe26ebc40:
=> 0x0000007fe26ebc20: stlrb w22, [x21]
0x0000007fe26ebc24: mov x4, #0x0 // #0
0x0000007fe26ebc28: ldr x10, [x20]
0x0000007fe26ebc2c: add x21, x20, #0x8
0x0000007fe26ebc30: mov x22, #0x0 // #0
0x0000007fe26ebc34: strb w22, [x28, #428]
0x0000007fe26ebc38: mov x22, #0x0 // #0
0x0000007fe26ebc3c: strb w22, [x28, #431]
End of assembler dump.
(gdb) info reg x9
x9 0x0 0
* Looks like JIT code, even doing accesses to x28 which is the FEX CPU state
* Code has no backtrace which reinforces this
* Code is doing an atomic store, which reinforces this is FEX emulating the x86 TSO memory model

* Now that we have checked that we are in the JIT code. Where are we in the guest side?
* Let's dump the FEX CPU state information that is directly pointed to in x28 at all times in JIT code.

(gdb) p/x ((FEXCore::Core::CpuStateFrame*)$x28)->State
$3 = {rip = 0x401110, gregs = {0x416eb0, 0x7fe1e3b640, 0xffffffffffffff70, 0x0, 0x7fe1e3bf30, 0x0, 0x416eb0, 0x7fe1e3ae28, 0x0, 0x7fe1e3b640, 0x8, 0x7fe2054cc0, 0x7ff75ff48e, 0x7ff75ff48f, 0x0, 0x7fe163b000}, xmm = {{0x0, 0x0}, {0x0, 0x0}, {0xdeadbeef, 0xbad0dad1} <repeats 14 times>}, es = 0x0, cs = 0x0, ss = 0x0, ds = 0x0, gs = 0x0, fs = 0x7fe1e3b640, flags = {0x0, 0x1, 0x0,
0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x1, 0x0 <repeats 38 times>}, mm = {{0x0, 0x0}, {0x0, 0x0}, {0x0, 0x0}, {0x0, 0x0}, {0x0, 0x0}, {0x0, 0x0}, {0x0, 0x0}, {0x0, 0x0}}, gdt = {{base = 0x0} <repeats 32 times>}, FCW = 0x37f, FTW = 0xffff}

* Looks like our guest RIP is currently `0x401110`
** Consult the `info proc mappings` again
0x401000 0x402000 0x1000 0x1000 {...}/sigsegv_test
* Yep, we are inside our test application
* For a simple test, let's load the application in gdb-multiarch and disassemble where we are
$ gdb-multiarch ./sigsegv_test
Reading symbols from ./sigsegv_test...
(gdb) set disassembly-flavor intel
(gdb) disas 0x401110
Dump of assembler code for function main(int, char**):
0x0000000000401110 <+0>: push rbp
0x0000000000401111 <+1>: mov rbp,rsp
0x0000000000401114 <+4>: mov DWORD PTR [rbp-0x4],0x0
0x000000000040111b <+11>: mov DWORD PTR [rbp-0x8],edi
0x000000000040111e <+14>: mov QWORD PTR [rbp-0x10],rsi
0x0000000000401122 <+18>: mov rax,QWORD PTR [rbp-0x10]
0x0000000000401126 <+22>: movsxd rcx,DWORD PTR [rbp-0x8]
0x000000000040112a <+26>: mov rax,QWORD PTR [rax+rcx*8]
0x000000000040112e <+30>: mov QWORD PTR [rbp-0x18],rax
0x0000000000401132 <+34>: mov rax,QWORD PTR [rbp-0x18]
0x0000000000401136 <+38>: mov BYTE PTR [rax],0x63
0x0000000000401139 <+41>: xor eax,eax
0x000000000040113b <+43>: pop rbp
0x000000000040113c <+44>: ret
End of assembler dump.
(gdb)

* Okay, not super helpful since FEX translates instructions in to blocks, `0x401110` is just our starting address
** It's in this code somewhere, let's change some FEX settings to get a clearer picture
* Set block size to one instruction and disable multiblock
* Now rerun our test application and find the new RIP

(gdb) p/x ((FEXCore::Core::CpuStateFrame*)$x28)->State.rip
$2 = 0x401136

* Alright, now we know the RIP is exactly at `0x401136`
* Back in gdb-multiarch
(gdb) disas 0x401136,+1
Dump of assembler code from 0x401136 to 0x401137:
0x0000000000401136 <main(int, char**)+38>: mov BYTE PTR [rax],0x63

* Looks like something in main is storing 0x63 to a nullptr
* In this simple case we can now take a look at the test application's source and find the problem.
** We know the problem is in the first block of main()
** We know the exact instruction that it is at
** We know it's something storing a byte to memory
* For more complex cases it is likely necessary to use reverse engineering tools
** BinaryNinja, Ghidra, IDA, and Hopper are all examples of tools like this

== What to do from here ==
Now it becomes a lot harder. You don't get a typical debugging environment or even clean backtraces.

FEX's gdbserver integration is sorely lacking so you can't even use a remote gdb server connecting to FEX right now.

If you enable thunks you can get better backtraces here. [[Development:Debugging_Crash_In_Thunks|Debugging_Crash_In_Thunks]]

== Attempting to use FEX-Emu's gdbserver implementation ==
'''Here be dragons'''

FEX supports gdbserver as an integration. It's implementation is significantly limited but can still be used for debugging and getting some backtraces.
* Currently hardcodes the port to use as `8086` and if you have multiple gdbserver processes running then it will encounter problems.
* Currently does not follow processes through fork/execve at all. No multiprocess support
** This means you '''must''' only start the process you're caring about debugging
* Currently starts the process paused and will wait until gdb attach before continuing
** No way to start a FEX instance then attach at some later point
* Ctrl-C to stop the FEX process needs to be done twice
** Maybe with a small delay inbetween because gdb needs to fetch a bunch of data on pause
** Known bug, unknown why broken at the moment

FEXLoader -G -- <Application> <Args...>

== Double checking if we are in JIT code ==
(gdb) info reg pc
pc 0x7fccfb9ec8 0x7fccfb9ec8

(gdb) info proc mappings
...
0x7fccfb9000 0x7fcdfb9000 0x1000000 0x0

* Looks like FEX JIT mapping, we start out at 16MB but scale up to 128MB
* Depending on version of FEX we can check the base mapping for a unique string
(gdb) p (char*)0x7fccfb9000
$4 = 0x7fccfb9000 "FEXJIT::Arm64JITCore::"

== Getting RIP of current code block ==
FEX sets up an address in our CPU context to get some debug data out.

Currently this isn't exposed in a way that a debugger can see other than manually typing out gdb commands

p/x *(uint64_t*)($x28) = inline block header ptr
p/x *(uint32_t*)(*(uint64_t*)($x28)) = OffsetToBlockTail
p/x *(uint64_t*)($x28) + *(uint32_t*)(*(uint64_t*)($x28))

p/x *(unsigned long long*)((*(unsigned long long*)($x28) + *(unsigned int*)(*(unsigned long long*)($x28)))+8) = RIP of block

disas *(unsigned long long*)($x28),+*(unsigned long long*)(*(unsigned long long*)($x28) + *(unsigned int*)(*(unsigned long long*)($x28))) = Disassemble this block of code.

While hard to decipher here is basically what is happening.
- x28 is the CPU register that FEX keeps in the JIT for context accesses
- offset 184 is the offset of the `InlineJITBlockHeader` member inside of that context.
- As long as FEX is in a JIT block that offset will be valid to point to the current RIP that the block is operating on.

Getting the stack can also be very useful

x/64wx ((FEXCore::Core::CPUState*)$x28)->gregs[FEXCore::X86State::REG_RSP]

Doing raw pointer math here means that this works even when gdb fails to find symbols for the CPUState object, which happens very frequently for some reason.

Development:Setting up RootFS

2025-09-22T14:11:50Z

Neobrain: Update for FEXConfig changes

[[Category:Development]]
= Setting up a RootFS =
'''For AArch64 hosts you are required to have an x86-64 rootfs'''. With or without 32bit compatibility libraries.

This isn't strictly required for testing on x86-64 host environments but is useful for ensuring correct behaviour there.

= Quick Setup with FEXRootFSFetcher =
FEX comes with a FEXRootFSFetcher tool that can download an official preconfigured rootfs from the FEX servers.
This tool operates as either a terminal application or a Zenity application depending on if it is launched from terminal or not.

This tool will walk you through selecting a RootFS for FEX:
* If your host OS matches one in the download list then it will recommend that one
* After selecting the RootFS, it will download it
** This downloads to the default location in '''$HOME/.fex-emu/RootFS/'''
* After downloading the RootFS the tool will ask if you want to use the SquashFS file directly or extract it
* It will then ask if you want to set the downloaded RootFS as your default
** This writes to the default configuration in '''$HOME/.fex-emu/Config.json'''

You can skip the rest of the guide if you used FEXRootFSFetcher to download the rootfs.

== Troubleshooting ==
* The squashfs image will fail to extract if you don't have unsquashfs installed
** Install the squashfs-tools package to get this application

* '''SquashFS RootFS file corrupted?'''
** Rerun the tool, select the rootfs from the list, and if the rootfs already exists it will ask to verify
** If verification fails it will ask if you want to redownload
* '''Want to manually check the hash?'''
** Run the rootfs file directly through the tool to get a hash
** '''FEXRootFSFetcher $HOME/.fex-emu/RootFS/Ubuntu_21_10.sqsh'''
** Ubuntu_21_10.sqsh has hash: bf9507fbdbaec2cf
* '''Want to use the Zenity UI but still launch through terminal?'''
** Force Zenity from terminal with '''FEXRootFSFetcher 1> /dev/null'''
* '''What if my Linux distro isn't listed?'''
** Don't worry, select one that is most similar to yours or newest
** This will work in most cases
* '''How can I trust these images?'''
** These images are generated by scripts living in our public repo
** If you want to rebuild them yourself, you are free to do so.
** https://github.com/FEX-Emu/RootFS
* '''I want to add a distro'''
** Follow how our current rootfs is generated and add your own
** Open a pull request to add the rootfs setup
** Contact us on the official FEX-Emu Discord to get the RootFS added to the server.
* '''Where is the raw list of images at?'''
** The image links live on our website at the following link
** https://rootfs.fex-emu.gg/RootFS_links.json
** This file is in a format that our tool understands
** Generally the format is:
*** Distro Name
*** Distro Version
*** Human readable name
*** Raw rootfs link
*** xxhash based file hash

= Making a custom rootfs prerequisites =
An x86-64 PC or FEX-Emu installed for chrooting

= Getting Started =
The easiest way to get started is to just download a prebuilt x86-64 rootfs from your favourite distribution and extract it to where you want.
* Ubuntu - http://cdimage.ubuntu.com/ubuntu-base/releases/
* Debian Debootstrap guide - https://wiki.debian.org/Debootstrap
** Debootstrap is more involved and not explained here.

Using chroot you can then enter the rootfs and install applications that you desire for testing purposes.
It's easiest to set this up on an x86-64 host and then copy it to an AArch64 device.

== Cross architecture chroot ==
'''You need to have binfmt_misc setup for x86-64 support on your host system.'''
* ls /proc/sys/fs/binfmt_misc
** If there are files in this folder then your host supports binfmt_misc
* Once you have binfmt_misc running and FEX-Emu installed in to it, you can chroot in to an x86-64 rootfs

== Chrooting in to rootfs with FEX-Emu Rootfs ==
* If you're using a FEX-Emu provided rootfs then we provide two scripts at the start to handle this
** '''unbreak_chroot.sh''' - This sets up the rootfs and chroots inside of it
** '''break_chroot.sh''' - This sets up the rootfs for using with FEX generally.
*** '''Make sure to run this after leaving the chroot!'''
*** The '''unbreak_chroot.sh''' script will run this if you exit safely.
** The remaining steps in this guide can be skipped

== Chrooting in to custom rootfs with FEX-Emu ==
* If you're setting up your own rootfs then you need to follow the binding steps that are following.
** We have a script that if you know what you're doing you can cherry pick parts from [https://github.com/FEX-Emu/RootFS/blob/main/unbreak_chroot.sh here]

= Additional TmpFS folders =
Depending on what you're doing in the chroot, you may need or want some tmpfs folders that applications are expecting

* sudo mount -t proc /proc ${ROOTFS}/proc/
* sudo mount --rbind --make-rslave /sys ${ROOTFS}/sys/
* sudo mount --rbind --make-rslave /dev ${ROOTFS}/dev/
* sudo mount --rbind --make-rslave /dev/pts ${ROOTFS}/dev/pts/
* sudo mount --rbind /lib/ld-linux-aarch64.so.1 ${ROOTFS}/lib/ld-linux-aarch64.so.1
* sudo mount --rbind /lib/aarch64-linux-gnu ${ROOTFS}/lib/aarch64-linux-gnu

Then of course, make sure to unmount these partitions when you're done

* sudo umount ${ROOTFS}/proc/
* sudo umount -R ${ROOTFS}/sys/
* sudo umount -R ${ROOTFS}/dev/pts/
* sudo umount -R ${ROOTFS}/dev/
* sudo umount -R ${ROOTFS}/lib/ld-linux-aarch64.so.1
* sudo umount -R ${ROOTFS}/lib/aarch64-linux-gnu

= Using the RootFS with FEX =

Run the graphical FEXConfig tool to select and change the active RootFS. Any file or folder in $HOME/.fex-emu/RootFS will be detected automatically.

= Additional Steps for building a custom rootfs =
== Running under WSL1 ==
/etc/hosts and /etc/resolv.conf change on the host side. You'll need to copy those over to the rootfs if you want network functionality to work.

== Ubuntu packages to install ==
Sample packages to install in the rootfs
* apt-get install pulseaudio libgles1 libgles2 libglx-mesa0 libgl1-mesa-dri libgl1-mesa-glx mesa-utils mesa-utils-extra
Pulseaudio specifically is required so pulseaudio will work when talking to the host pulseaudio

== File deletion ==
Some files need to be deleted from the rootfs otherwise some applications will fail to work.

Note that deleting these files '''WILL''' break the rootfs and you won't be able to chroot in to it anymore.

A likely better route would be a script that moves these files and folders then brings them back when you want to chroot

=== Files to remove ===
* ${ROOTFS}/etc/hosts
* ${ROOTFS}/etc/resolv.conf
* ${ROOTFS}/etc/timezone
* ${ROOTFS}/etc/localtime
* ${ROOTFS}/etc/passwd

=== Folders to remove ===
* ${ROOTFS}/boot
* ${ROOTFS}/dev
* ${ROOTFS}/home
* ${ROOTFS}/media
* ${ROOTFS}/mnt
* ${ROOTFS}/proc
* ${ROOTFS}/root
* ${ROOTFS}/srv
* ${ROOTFS}/tmp
* ${ROOTFS}/var/cache/apt
* ${ROOTFS}/var/lib/apt
* ${ROOTFS}/sys
* ${ROOTFS}/opt

== Creating a Squashfs rootfs from custom rootfs ==
SquashFS is a great way to do binary distribution of the rootfs. If you have created a rootfs and you want to roll it in to a squashfs like the FEX-Emu provided images then this is information on how to do that.

'''SquashFS is currently only recommended for binary distribution. User should unpack it to not get dangling processes and mounts.'''
* Offers tooling to mount in userspace
* Supports ZSTD compression
* Is a readonly partition that matches FEX's use case

=== Tools that FEX needs to run squashfs ===
* squashfuse
* fusermount

=== Making the squashfs image ===
Making the squashfs image is very straightforward
* mksquashfs ${ROOTFS}/ Squashfs.sqsh -comp zstd

This creates a file called Squashfs.sqsh that contains the original rootfs that we generated above.

=== Using the squashfs image in FEX ===
* Copy the resulting sqsh file in to ${XDG_DATA_HOME}/.fex-emu/RootFS/
** If ${XDG_DATA_HOME} doesn't exist then that will be replaced with ${HOME}
* Use FEXConfig to select the sqsh file as a named rootfs
* or pass the name in to the FEX_ROOTFS argument in the various ways FEX supports
* or the absolute path if you don't want it in the FEX RootFS folder

=== Extracting the squashfs ===
If you don't want to use squashfs for whatever reason then you can extract it in to the FEX RootFS folder and use the directory directly
* unsquashfs -f -d NewRootFS/ Squashfs.sqsh

=== If you want to install packages in the squashfs ===
The squashfs image is immutable and can't be modified.
* Extract the squashfs first
* Change the working directory to the rootfs
* execute `unbreak_chroot.sh`
* `apt-get update`
* `apt-get install <Whatever you want>`
* `exit` to leave the chroot
* Use the new extracted chroot directly, or follow the previous steps to recompress it to a squashfs format.

=== Known problems with squashfs ===
'''FEX automatically mounts your squashfs file and can leave dangling squashfs mounts around in some instances.'''
* '''This is fairly rare but it can occur. Whenever an instance of FEX is run, it registers with the FEXMountDaemon'''
** FEXMountDaemon then watches these instances of FEX to ensure they have exited before unmounting
** There is roughly a ten second grace period of no instances of FEX running before unmounting
** If an instance of FEX starts running then it will reuse the mount partition and reset the timeout
* To get rid of these dangling mounts if they do manage to manifest, use the following commands
** fusermount -u /tmp/.FEXMount* ; rmdir /tmp/.FEXMount*
** These don't require root to unmount
* In some cases the unmounting can end up returning busy status and not unmount
** Check the output of `mount` for any /tmp/.FEXMount folders
** This is due to either FEXMountDaemon or squashfuse still running in the background
** Can force the application to exit
** step 1: kill `pidof squashfuse`
** This will attempt to send SIGTERM to all squashfuse applications, which will unmount the partition on terminate
** Failing that, you can try to be more abusive
** kill -9 `pidof FEXMountDaemon`
** This will forcibly terminate FEXMountDaemon which should also take out the child mounted process
* If any instances of the mount still exists then you may need to kill more FEX processes.
** Any ''FEX'' process will end up keeping the folder mounted and keep it busy

'''strace breaks squashfs'''
* If you run FEX with `strace -f` then the process that mounts your rootfs loses its setuid bit.
** This means you then can't mount the rootFS
* The only workaround is to not use a squashfs in this instance.
* Alternatively you can spin up a FEXBash instance to force squashfs to be mounted. Then strace a different FEX instance which will use the mounted squashfs location.

Development:Setting up RootFS

2025-09-22T14:02:51Z

Neobrain: FEXInterpreter -> FEX

[[Category:Development]]
= Setting up a RootFS =
'''For AArch64 hosts you are required to have an x86-64 rootfs'''. With or without 32bit compatibility libraries.

This isn't strictly required for testing on x86-64 host environments but is useful for ensuring correct behaviour there.

= Quick Setup with FEXRootFSFetcher =
FEX comes with a FEXRootFSFetcher tool that can download an official preconfigured rootfs from the FEX servers.
This tool operates as either a terminal application or a Zenity application depending on if it is launched from terminal or not.

This tool will walk you through selecting a RootFS for FEX:
* If your host OS matches one in the download list then it will recommend that one
* After selecting the RootFS, it will download it
** This downloads to the default location in '''$HOME/.fex-emu/RootFS/'''
* After downloading the RootFS the tool will ask if you want to use the SquashFS file directly or extract it
* It will then ask if you want to set the downloaded RootFS as your default
** This writes to the default configuration in '''$HOME/.fex-emu/Config.json'''

You can skip the rest of the guide if you used FEXRootFSFetcher to download the rootfs.

== Troubleshooting ==
* The squashfs image will fail to extract if you don't have unsquashfs installed
** Install the squashfs-tools package to get this application

* '''SquashFS RootFS file corrupted?'''
** Rerun the tool, select the rootfs from the list, and if the rootfs already exists it will ask to verify
** If verification fails it will ask if you want to redownload
* '''Want to manually check the hash?'''
** Run the rootfs file directly through the tool to get a hash
** '''FEXRootFSFetcher $HOME/.fex-emu/RootFS/Ubuntu_21_10.sqsh'''
** Ubuntu_21_10.sqsh has hash: bf9507fbdbaec2cf
* '''Want to use the Zenity UI but still launch through terminal?'''
** Force Zenity from terminal with '''FEXRootFSFetcher 1> /dev/null'''
* '''What if my Linux distro isn't listed?'''
** Don't worry, select one that is most similar to yours or newest
** This will work in most cases
* '''How can I trust these images?'''
** These images are generated by scripts living in our public repo
** If you want to rebuild them yourself, you are free to do so.
** https://github.com/FEX-Emu/RootFS
* '''I want to add a distro'''
** Follow how our current rootfs is generated and add your own
** Open a pull request to add the rootfs setup
** Contact us on the official FEX-Emu Discord to get the RootFS added to the server.
* '''Where is the raw list of images at?'''
** The image links live on our website at the following link
** https://rootfs.fex-emu.gg/RootFS_links.json
** This file is in a format that our tool understands
** Generally the format is:
*** Distro Name
*** Distro Version
*** Human readable name
*** Raw rootfs link
*** xxhash based file hash

= Making a custom rootfs prerequisites =
An x86-64 PC or FEX-Emu installed for chrooting

= Getting Started =
The easiest way to get started is to just download a prebuilt x86-64 rootfs from your favourite distribution and extract it to where you want.
* Ubuntu - http://cdimage.ubuntu.com/ubuntu-base/releases/
* Debian Debootstrap guide - https://wiki.debian.org/Debootstrap
** Debootstrap is more involved and not explained here.

Using chroot you can then enter the rootfs and install applications that you desire for testing purposes.
It's easiest to set this up on an x86-64 host and then copy it to an AArch64 device.

== Cross architecture chroot ==
'''You need to have binfmt_misc setup for x86-64 support on your host system.'''
* ls /proc/sys/fs/binfmt_misc
** If there are files in this folder then your host supports binfmt_misc
* Once you have binfmt_misc running and FEX-Emu installed in to it, you can chroot in to an x86-64 rootfs

== Chrooting in to rootfs with FEX-Emu Rootfs ==
* If you're using a FEX-Emu provided rootfs then we provide two scripts at the start to handle this
** '''unbreak_chroot.sh''' - This sets up the rootfs and chroots inside of it
** '''break_chroot.sh''' - This sets up the rootfs for using with FEX generally.
*** '''Make sure to run this after leaving the chroot!'''
*** The '''unbreak_chroot.sh''' script will run this if you exit safely.
** The remaining steps in this guide can be skipped

== Chrooting in to custom rootfs with FEX-Emu ==
* If you're setting up your own rootfs then you need to follow the binding steps that are following.
** We have a script that if you know what you're doing you can cherry pick parts from [https://github.com/FEX-Emu/RootFS/blob/main/unbreak_chroot.sh here]

= Additional TmpFS folders =
Depending on what you're doing in the chroot, you may need or want some tmpfs folders that applications are expecting

* sudo mount -t proc /proc ${ROOTFS}/proc/
* sudo mount --rbind --make-rslave /sys ${ROOTFS}/sys/
* sudo mount --rbind --make-rslave /dev ${ROOTFS}/dev/
* sudo mount --rbind --make-rslave /dev/pts ${ROOTFS}/dev/pts/
* sudo mount --rbind /lib/ld-linux-aarch64.so.1 ${ROOTFS}/lib/ld-linux-aarch64.so.1
* sudo mount --rbind /lib/aarch64-linux-gnu ${ROOTFS}/lib/aarch64-linux-gnu

Then of course, make sure to unmount these partitions when you're done

* sudo umount ${ROOTFS}/proc/
* sudo umount -R ${ROOTFS}/sys/
* sudo umount -R ${ROOTFS}/dev/pts/
* sudo umount -R ${ROOTFS}/dev/
* sudo umount -R ${ROOTFS}/lib/ld-linux-aarch64.so.1
* sudo umount -R ${ROOTFS}/lib/aarch64-linux-gnu

= Using the rootfs with FEX =
If you're using FEXLoader then you can pass the full path of the rootfs with the -R option.
Additionally you can put the rootfs folder inside of $HOME/.fex-emu/RootFS/<RootFS name>/ and FEX supports a "named" rootfs option
* eq: -R Ubuntu_2104 will use rootfs $HOME/.fex-emu/RootfS/Ubuntu_2104
* Read the Fex man page with man fex for more information

== Setting RootFS with FEXConfig ==
[[File:FEXConfig Load Default.png|frameless|Load the default options on first run]]
On the first run of FEXConfig, select the "Load Default Options" configuration.
* Subsequent runs should select "Open Default"

[[File:FEXConfig RootFS Selection.png|frameless|Select the RootFS from the FEXConfig location]]

FEXConfig will autopopulate the rootfs list with rootfs folders that exist in $HOME/.fex-emu/RootFS/

You can select one from the list or provide a full path in the RootFS: line to set it manually.

''' Make sure to select the File->Save option after the fact to save'''

= Additional Steps for building a custom rootfs =
== Running under WSL1 ==
/etc/hosts and /etc/resolv.conf change on the host side. You'll need to copy those over to the rootfs if you want network functionality to work.

== Ubuntu packages to install ==
Sample packages to install in the rootfs
* apt-get install pulseaudio libgles1 libgles2 libglx-mesa0 libgl1-mesa-dri libgl1-mesa-glx mesa-utils mesa-utils-extra
Pulseaudio specifically is required so pulseaudio will work when talking to the host pulseaudio

== File deletion ==
Some files need to be deleted from the rootfs otherwise some applications will fail to work.

Note that deleting these files '''WILL''' break the rootfs and you won't be able to chroot in to it anymore.

A likely better route would be a script that moves these files and folders then brings them back when you want to chroot

=== Files to remove ===
* ${ROOTFS}/etc/hosts
* ${ROOTFS}/etc/resolv.conf
* ${ROOTFS}/etc/timezone
* ${ROOTFS}/etc/localtime
* ${ROOTFS}/etc/passwd

=== Folders to remove ===
* ${ROOTFS}/boot
* ${ROOTFS}/dev
* ${ROOTFS}/home
* ${ROOTFS}/media
* ${ROOTFS}/mnt
* ${ROOTFS}/proc
* ${ROOTFS}/root
* ${ROOTFS}/srv
* ${ROOTFS}/tmp
* ${ROOTFS}/var/cache/apt
* ${ROOTFS}/var/lib/apt
* ${ROOTFS}/sys
* ${ROOTFS}/opt

== Creating a Squashfs rootfs from custom rootfs ==
SquashFS is a great way to do binary distribution of the rootfs. If you have created a rootfs and you want to roll it in to a squashfs like the FEX-Emu provided images then this is information on how to do that.

'''SquashFS is currently only recommended for binary distribution. User should unpack it to not get dangling processes and mounts.'''
* Offers tooling to mount in userspace
* Supports ZSTD compression
* Is a readonly partition that matches FEX's use case

=== Tools that FEX needs to run squashfs ===
* squashfuse
* fusermount

=== Making the squashfs image ===
Making the squashfs image is very straightforward
* mksquashfs ${ROOTFS}/ Squashfs.sqsh -comp zstd

This creates a file called Squashfs.sqsh that contains the original rootfs that we generated above.

=== Using the squashfs image in FEX ===
* Copy the resulting sqsh file in to ${XDG_DATA_HOME}/.fex-emu/RootFS/
** If ${XDG_DATA_HOME} doesn't exist then that will be replaced with ${HOME}
* Use FEXConfig to select the sqsh file as a named rootfs
* or pass the name in to the FEX_ROOTFS argument in the various ways FEX supports
* or the absolute path if you don't want it in the FEX RootFS folder

=== Extracting the squashfs ===
If you don't want to use squashfs for whatever reason then you can extract it in to the FEX RootFS folder and use the directory directly
* unsquashfs -f -d NewRootFS/ Squashfs.sqsh

=== If you want to install packages in the squashfs ===
The squashfs image is immutable and can't be modified.
* Extract the squashfs first
* Change the working directory to the rootfs
* execute `unbreak_chroot.sh`
* `apt-get update`
* `apt-get install <Whatever you want>`
* `exit` to leave the chroot
* Use the new extracted chroot directly, or follow the previous steps to recompress it to a squashfs format.

=== Known problems with squashfs ===
'''FEX automatically mounts your squashfs file and can leave dangling squashfs mounts around in some instances.'''
* '''This is fairly rare but it can occur. Whenever an instance of FEX is run, it registers with the FEXMountDaemon'''
** FEXMountDaemon then watches these instances of FEX to ensure they have exited before unmounting
** There is roughly a ten second grace period of no instances of FEX running before unmounting
** If an instance of FEX starts running then it will reuse the mount partition and reset the timeout
* To get rid of these dangling mounts if they do manage to manifest, use the following commands
** fusermount -u /tmp/.FEXMount* ; rmdir /tmp/.FEXMount*
** These don't require root to unmount
* In some cases the unmounting can end up returning busy status and not unmount
** Check the output of `mount` for any /tmp/.FEXMount folders
** This is due to either FEXMountDaemon or squashfuse still running in the background
** Can force the application to exit
** step 1: kill `pidof squashfuse`
** This will attempt to send SIGTERM to all squashfuse applications, which will unmount the partition on terminate
** Failing that, you can try to be more abusive
** kill -9 `pidof FEXMountDaemon`
** This will forcibly terminate FEXMountDaemon which should also take out the child mounted process
* If any instances of the mount still exists then you may need to kill more FEX processes.
** Any ''FEX'' process will end up keeping the folder mounted and keep it busy

'''strace breaks squashfs'''
* If you run FEX with `strace -f` then the process that mounts your rootfs loses its setuid bit.
** This means you then can't mount the rootFS
* The only workaround is to not use a squashfs in this instance.
* Alternatively you can spin up a FEXBash instance to force squashfs to be mounted. Then strace a different FEX instance which will use the mounted squashfs location.

Config

2025-09-22T14:02:30Z

Neobrain: FEXLoader was removed

FEX-Emu has a relatively complex configuration system. This is due to the fact that it is an overlay based configuration system and also it is technically two configuration systems in one.
== Main application configuration ==
This is the primary configuration that is the bulk of the implementation. This falls under the "Config" section of the json configuration files.

=== Overlay system ===
This configuration system is based on layers that either override or merge with the layer under it.

=== Loadable layers ===
From highest priority to lowest priority. Higher priority layers can override options in lower-priority layers.

* Meta layer (Top most)
** This layer is the final merging of all the layers under it.
* Environment variables
** This layer loads options from the environment variable prefixed with FEX_
* User-writable app config
* User-writable steam app id
* Global installed app config
* Global installed Steam App config
* User-writable main config
* Global main config

=== Environment variables ===
These come from environment variables prefixed with '''FEX_'''. Check the man page for FEX to know what environment variables correspond to which options.

=== User-writable app config ===
Just like the global app config file, this takes the form of '''<AppName>.json'''

This can be loaded from multiple paths highest priority to lowest priority

* '''$FEX_APP_CONFIG'''
** Usually not set. Direct JSON filepath
* '''$XDG_CONFIG_HOME/.fex-emu/AppConfig/<AppName>.json'''
** Depends on environment, not always set on all distributions
* '''$FEX_APP_CONFIG_LOCATION/AppConfig/<AppName>.json'''
** Usually not set
* '''$HOME/.fex-emu/AppConfig/<AppName>.json'''
** The regular config location

=== User-writable steam app id ===
Just like the global steam app config file, this takes the form of '''Steam_$SteamAppId_<AppName>.json'''

This can be loaded from multiple paths highest priority to lowest priority

* '''$XDG_CONFIG_HOME/.fex-emu/AppConfig/Steam_$SteamAppId_<AppName>.json'''
** Depends on environment, not always set on all distributions
* '''$FEX_APP_CONFIG_LOCATION/AppConfig/Steam_$SteamAppId_<AppName>.json'''
** Usually not set
* '''$HOME/.fex-emu/AppConfig/Steam_$SteamAppId_<AppName>.json'''
** The regular config location

=== Global installed app config ===
This file is loaded from '''/usr/share/fex-emu/AppConfig/<AppName>.json'''

* Usually doesn't exist
* FEX-Emu ships a handful of these

=== Global steam app config ===
This file is loaded from '''/usr/share/fex-emu/AppConfig/Steam_$SteamAppId_<AppName>.json'''

* Usually doesn't exist

=== User-writable main config ===
This can come from a couple locations. From highest priority to lowest.

Will not fallback to loading later configs in the list if one fails.

* '''$XDG_CONFIG_HOME/.fex-emu/Config.json'''
** Depends on environment, not always set on all distributions
* '''$FEX_APP_CONFIG_LOCATION/Config.json'''
** Usually not set
* '''$HOME/.fex-emu/Config.json'''
** The regular config location

=== Global main config ===
This file is loaded from '''/usr/share/fex-emu/Config.json'''
* Usually doesn't exist

== Thunks configuration system ==
This system is disjoint from the other configuration system, but overlaps slightly as we will see. This system loads the '''ThunksDB''' section of the json configuration files.

This system is disjoint from the other configuration system because it only affects thunking configuration options.

=== Loadable Layers ===
The layers from highest priority to lowest priority. Higher priority layers can enable or disable options from the lower priority layers.
* User-writable app config
* User-writable steam app config
* Global installed app config
* Global installed Steam App config
* FEX ThunkConfig option path
* User-writable main config
* Global main config

=== User-writable installed app config ===
* Exactly the same as the above user-writable installed app config

=== User-writable steam app config ===
* Exactly the same as the above user-writable installed steam app config

=== Global installed app config ===
* Exactly the same as the above global installed app config

=== Global installed Steam App config ===
* Exactly the same as the above global installed steam app config

=== FEX ThunkConfig Option path ===
'''ThunkConfig''' is an option in FEX's '''Config''' section. If this filepath exists then it is loaded.
* This means it can be overridden with '''FEX_THUNKCONFIG''' environment variable, which is handy.

=== User-writable main config ===
* Exactly the same as the above user-writable main config

=== Global main config ===
* Exactly the same as the above Global main config

Development:Setting up FEX

2025-09-22T14:01:48Z

Neobrain: FEXInterpreter -> FEX

[[Category:Development]]
== Build dependencies ==
=== Debian/Ubuntu dependencies ===
* git
* cmake
* ninja-build
* pkgconf
* ccache (Recommended if you're rebuilding constantly)
* clang
* llvm
* lld (For ThinLTO)
* binfmt-support (For AArch64 binfmt_misc support)
* libssl-dev
* python-setuptools (For build scripts)
* g++-x86-64-linux-gnu (For building thunks)
* libgcc-XX-dev-i386-cross (For building thunks, version should match your g++-x86-64-linux-gnu package)
* libgcc-XX-dev-amd64-cross (For building thunks, version should match your g++-x86-64-linux-gnu package)
* nasm (only if building tests)
* python3-clang (for struct verifier)
* libstdc++-12-dev-i386-cross (for struct verifier)
* libstdc++-12-dev-amd64-cross (for struct verifier)
* libstdc++-12-dev-arm64-cross (for struct verifier)
* squashfs-tools (For squashfs rootfs support)
* squashfuse (For mounting the squashfs)
* libc-bin or glibc-tools (For unittests, needs catchsegv from this)
* libc6-dev-i386-amd64-cross (For FEXLinuxTests)
* lib32stdc++-XX-dev-amd64-cross (For FEXLinuxTests, version should match your g++-x86-64-linux-gnu package)
* qtdeclarative5-dev (For GUI)
* qml-module-qtquick-controls (For GUI)
* qml-module-qtquick-controls2 (For GUI)
* qml-module-qtquick-dialogs (For GUI)

=== Gentoo dependencies ===
* dev-vcs/git
* cmake
* ninja
* ccache
* clang
* lld
* nasm

'''Use gubbins'''
* >=media-libs/libglvnd-1.3.4 X

=== Fedora dependencies ===
* git
* cmake
* ninja-build
* pkg-config
* ccache
* clang
* lld
* llvm
* llvm-devel
* openssl-devel
* nasm
* python3-clang
* python3-setuptools
* squashfs-tools
* squashfuse
* erofs-fuse
* erofs-utils
* qt5-qtdeclarative-devel
* qt5-qtquickcontrols
* qt5-qtquickcontrols2

== Build Configuration ==
* Ensure release mode is enabled
* Disable test building
* Enable LTO in a release build
* Enable LLD to make LTO builds faster

git clone --recurse-submodules https://github.com/FEX-Emu/FEX.git
cd FEX
mkdir Build
cd Build
CC=clang CXX=clang++ cmake -DCMAKE_INSTALL_PREFIX=/usr -DCMAKE_BUILD_TYPE=Release -DUSE_LINKER=lld -DENABLE_LTO=True -DBUILD_TESTING=False -DENABLE_ASSERTIONS=False -G Ninja ..
ninja

== Installation ==
sudo ninja install

You can install a binfmt_misc handler for both 32bit and 64bit x86 execution directly from the environment.
If you already have box86's 32bit binfmt_misc handler installed then make sure to uninstall their's first.
Make sure to have run install prior to this, otherwise binfmt_misc will install an old handler even if the executable has been updated.

sudo ninja binfmt_misc

=== Quirks ===

==== binfmt_misc problems ====
* Double check that binfmt_misc has worked
** `ls /usr/share/binfmts/FEX*` should return files
** `ls /var/lib/binfmts/FEX*` should return files
** `/proc/sys/fs/binfmt_misc/status` should exist
*** If it doesn't exist then make sure your kernel has `CONFIG_BINFMT_MISC=y`
*** If it still doesn't exist then make sure systemd isn't disabling it `systemctl enable binfmt-support`

==== DESTDIR quirks ====
* When using DESTDIR, make sure the path is absolute instead of relative
** FEX's thunks use subprojects because it needs to cross-compile
** If DESTDIR is relative then the install paths of the subprojects go to the wrong location
** Using $(pwd)/install works well

== Runtime Configuration ==
FEXConfig is the application to use for runtime configuration setup.

CTLR+S saves the configuration.

=== Things to ensure to set ===
On AArch64 host, a rootfs path is mandatory, while it is optional on x86-64 host

* Core: JIT
* RootFS
* Silent Logging (Useful to get FEX information spam out of the way)
* Block Size (500 is a good default. Optimization passes may break things though)

More Details at: Follow the instructions [[Development:Configuring_FEX|here]]

== RootFS generation ==
AArch64 hosts require a rootfs for running applications. FEX provides various x86 images that can be installed through [[Development:Setting_up_RootFS#Quick_Setup_with_FEXRootFSFetcher|FEXRootFSFetcher]]

For more detailed instructions see [[Development:Setting_up_RootFS]]

== Setting up library forwarding ==

Follow the instructions [[Development:Setting_up_Library_Forwarding|here]].

== Binaries ==
* FEX
** This is the emulator executable used to run x86 programs
* FEXInterpreter
** This is a (deprecated) alias for ''FEX'', provided for backwards compatibility
* FEXConfig
** Used for configuring emulation from a Qt-based GUI
* FEXBash
** Can be useful for starting a bash instance that is running under emulation
** It's not a chroot! Don't use sudo inside of this environment!
* FEXMountDaemon
** This is a mount daemon that will run in the background when you are using a squashfs based rootfs
** Usually one one instance open at any given moment
** It usually closes when FEX exits, but not always. See the problems section in setting up a rootfs
*** Has a ten second timeout window in case a another instance of FEX opens and uses it.

=== Debug binaries ===
* TestHarnessRunner
** Used for running FEX's ASM unit tests
** Runs some binary in a host x86_64 host environment or FEX and checks results
** Useful for quickly creating ASM based unit tests
* IRLoader
** Used for running FEX's IR based unit tests
** Only runs through FEX and checks the results
** Minimal number of tests
* UnitTestGenerator
** This is supposed to be an x86-64 instruction generator
** Has an understanding of x86-64 instruction encoding format
** Generates hundreds of thousands of instructions
** Used to be used with a lockstep runner to ensure correct behaviour
** Largely unsupported, avoiding undefined behaviour is difficult and FEX doesn't need to match undefined behaviour
** Lockstep hostrunner wasn't ever made to be bug free, hard to support
** Doesn't ensure correct behaviour on AArch64 device

FEXBash

2025-09-22T14:01:24Z

Neobrain: FEXInterpreter -> FEX

[[Category:Utility_Type_Application]]
FEXBash is a utility program provided by FEX-emu for quickly spinning up a bash instance.
== What isn't FEXBash ==
FEXBash is '''not''':
* A virtual machine
** FEX is never a virtual machine.
* A chroot
** See [[Development:Setting_up_RootFS#Chrooting_in_to_rootfs_with_FEX-Emu_Rootfs|this wiki page]] if you want to chroot in to the rootfs.
* A way to install packages inside the x86-64 rootfs
** See [[Development:Setting_up_RootFS#If_you_want_to_install_packages_in_the_squashfs|this wiki page]] if you want to add packages to the rootfs.
** Trying to install packages while running under FEXBash will not work. The rootfs is effectively read-only.

== What is FEXBash ==
FEXBash is a wrapper program that executes ''FEX'' to run x86-64 '''/bin/sh''' or '''/bin/bash'''. Which application is executed depends on the number of arguments passed in

== Usage ==

=== Zero arguments ===
If the user passes in no arguments then FEXBash will execute x86-64 '''/bin/bash''' to give an emulated shell environment for tinkering around in.
ryanh@ubuntu:~$ uname -a
Linux ubuntu 5.15.0-rc1+ #106 SMP Tue Aug 2 19:43:23 PDT 2022 aarch64 aarch64 aarch64 GNU/Linux
ryanh@ubuntu:~$ FEXBash
FEXBash-ryanh@ubuntu:~> uname -a
Linux ubuntu 5.15.0 #FEX-2208-11-ga2f4f494a SMP Aug 13 2022 09:47:20 x86_64 x86_64 x86_64 GNU/Linux

This is not a virtual machine. It is just an emulated environment that overlays the AArch64 host environment. You can still execute AArch64 programs from the host and also x86-64 programs.

=== Any arguments ===
If any arguments are passed in then FEXBash executes '''/bin/sh -c''' with the provided arguments passed along. One must be careful with this as '''sh -c''' is usually expected to have its arguments wrapped in double quotes.
ryanh@ubuntu:~$ sh -c uname -a
Linux
ryanh@ubuntu:~$ sh -c "uname -a"
Linux ubuntu 5.15.0-rc1+ #106 SMP Tue Aug 2 19:43:23 PDT 2022 aarch64 aarch64 aarch64 GNU/Linux
ryanh@ubuntu:~$ FEXBash uname -a
Linux
ryanh@ubuntu:~$ FEXBash "uname -a"
Linux ubuntu 5.15.0 #FEX-2208-11-ga2f4f494a SMP Aug 13 2022 09:47:20 x86_64 x86_64 x86_64 GNU/Linux
As one can see, without the double quotes you effectively lose the '''-a''' argument. This behaves the same as '''/bin/sh -c''' behaviour without FEXBash.

This is usually used for some sort of scripting wanting to enter the FEX emulated environment.

Development:Debugging Crash

2025-09-22T14:00:56Z

Neobrain: FEXInterpreter -> FEX

== Getting Started ==
* Debug an application with `gdb --args FEX <application full path>`
* Under GDB make sure to do `handle SIGBUS SIGILL SIG63 noprint`
** We use signals for various things, check out [[Development:Debugging_FEX_with_Signals|Here]] for more information

== Crash in emulated/JIT code ==
Walking through debugging a simple test application that is crashing.
$ gdb --args FEX ./sigsegv_test
Reading symbols from FEX...
(gdb) r
Starting program: /usr/bin/FEX ./sigsegv_test
[Thread debugging using libthread_db enabled]
Using host libthread_db library "/lib/aarch64-linux-gnu/libthread_db.so.1".
[New Thread 0x7fccb75f30 (LWP 90107)]

Thread 2 "FEX received signal SIGSEGV, Segmentation fault.
[Switching to Thread 0x7fccb75f30 (LWP 90107)]
0x0000007fccfb9ec8 in ?? ()

* Okay, we have a sigsegv. Let's double check that it is JIT code (aka, guest emulated code)
(gdb) disas $pc,+32
Dump of assembler code from 0x7fe26ebc20 to 0x7fe26ebc40:
=> 0x0000007fe26ebc20: stlrb w22, [x21]
0x0000007fe26ebc24: mov x4, #0x0 // #0
0x0000007fe26ebc28: ldr x10, [x20]
0x0000007fe26ebc2c: add x21, x20, #0x8
0x0000007fe26ebc30: mov x22, #0x0 // #0
0x0000007fe26ebc34: strb w22, [x28, #428]
0x0000007fe26ebc38: mov x22, #0x0 // #0
0x0000007fe26ebc3c: strb w22, [x28, #431]
End of assembler dump.
(gdb) info reg x9
x9 0x0 0
* Looks like JIT code, even doing accesses to x28 which is the FEX CPU state
* Code has no backtrace which reinforces this
* Code is doing an atomic store, which reinforces this is FEX emulating the x86 TSO memory model

* Now that we have checked that we are in the JIT code. Where are we in the guest side?
* Let's dump the FEX CPU state information that is directly pointed to in x28 at all times in JIT code.

(gdb) p/x ((FEXCore::Core::CpuStateFrame*)$x28)->State
$3 = {rip = 0x401110, gregs = {0x416eb0, 0x7fe1e3b640, 0xffffffffffffff70, 0x0, 0x7fe1e3bf30, 0x0, 0x416eb0, 0x7fe1e3ae28, 0x0, 0x7fe1e3b640, 0x8, 0x7fe2054cc0, 0x7ff75ff48e, 0x7ff75ff48f, 0x0, 0x7fe163b000}, xmm = {{0x0, 0x0}, {0x0, 0x0}, {0xdeadbeef, 0xbad0dad1} <repeats 14 times>}, es = 0x0, cs = 0x0, ss = 0x0, ds = 0x0, gs = 0x0, fs = 0x7fe1e3b640, flags = {0x0, 0x1, 0x0,
0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x1, 0x0 <repeats 38 times>}, mm = {{0x0, 0x0}, {0x0, 0x0}, {0x0, 0x0}, {0x0, 0x0}, {0x0, 0x0}, {0x0, 0x0}, {0x0, 0x0}, {0x0, 0x0}}, gdt = {{base = 0x0} <repeats 32 times>}, FCW = 0x37f, FTW = 0xffff}

* Looks like our guest RIP is currently `0x401110`
** Consult the `info proc mappings` again
0x401000 0x402000 0x1000 0x1000 {...}/sigsegv_test
* Yep, we are inside our test application
* For a simple test, let's load the application in gdb-multiarch and disassemble where we are
$ gdb-multiarch ./sigsegv_test
Reading symbols from ./sigsegv_test...
(gdb) set disassembly-flavor intel
(gdb) disas 0x401110
Dump of assembler code for function main(int, char**):
0x0000000000401110 <+0>: push rbp
0x0000000000401111 <+1>: mov rbp,rsp
0x0000000000401114 <+4>: mov DWORD PTR [rbp-0x4],0x0
0x000000000040111b <+11>: mov DWORD PTR [rbp-0x8],edi
0x000000000040111e <+14>: mov QWORD PTR [rbp-0x10],rsi
0x0000000000401122 <+18>: mov rax,QWORD PTR [rbp-0x10]
0x0000000000401126 <+22>: movsxd rcx,DWORD PTR [rbp-0x8]
0x000000000040112a <+26>: mov rax,QWORD PTR [rax+rcx*8]
0x000000000040112e <+30>: mov QWORD PTR [rbp-0x18],rax
0x0000000000401132 <+34>: mov rax,QWORD PTR [rbp-0x18]
0x0000000000401136 <+38>: mov BYTE PTR [rax],0x63
0x0000000000401139 <+41>: xor eax,eax
0x000000000040113b <+43>: pop rbp
0x000000000040113c <+44>: ret
End of assembler dump.
(gdb)

* Okay, not super helpful since FEX translates instructions in to blocks, `0x401110` is just our starting address
** It's in this code somewhere, let's change some FEX settings to get a clearer picture
* Set block size to one instruction and disable multiblock
** See the image in FEXConfig to the right
[[File:FEXConfig 1Inst NoMulti.png|thumb]]
* Now rerun our test application and find the new RIP

(gdb) p/x ((FEXCore::Core::CpuStateFrame*)$x28)->State.rip
$2 = 0x401136

* Alright, now we know the RIP is exactly at `0x401136`
* Back in gdb-multiarch
(gdb) disas 0x401136,+1
Dump of assembler code from 0x401136 to 0x401137:
0x0000000000401136 <main(int, char**)+38>: mov BYTE PTR [rax],0x63

* Looks like something in main is storing 0x63 to a nullptr
* In this simple case we can now take a look at the test application's source and find the problem.
** We know the problem is in the first block of main()
** We know the exact instruction that it is at
** We know it's something storing a byte to memory
* For more complex cases it is likely necessary to use reverse engineering tools
** BinaryNinja, Ghidra, IDA, and Hopper are all examples of tools like this

== What to do from here ==
Now it becomes a lot harder. You don't get a typical debugging environment or even clean backtraces.

FEX's gdbserver integration is sorely lacking so you can't even use a remote gdb server connecting to FEX right now.

If you enable thunks you can get better backtraces here. [[Development:Debugging_Crash_In_Thunks|Debugging_Crash_In_Thunks]]

== Attempting to use FEX-Emu's gdbserver implementation ==
'''Here be dragons'''

FEX supports gdbserver as an integration. It's implementation is significantly limited but can still be used for debugging and getting some backtraces.
* Currently hardcodes the port to use as `8086` and if you have multiple gdbserver processes running then it will encounter problems.
* Currently does not follow processes through fork/execve at all. No multiprocess support
** This means you '''must''' only start the process you're caring about debugging
* Currently starts the process paused and will wait until gdb attach before continuing
** No way to start a FEX instance then attach at some later point
* Ctrl-C to stop the FEX process needs to be done twice
** Maybe with a small delay inbetween because gdb needs to fetch a bunch of data on pause
** Known bug, unknown why broken at the moment

FEXLoader -G -- <Application> <Args...>

== Double checking if we are in JIT code ==
(gdb) info reg pc
pc 0x7fccfb9ec8 0x7fccfb9ec8

(gdb) info proc mappings
...
0x7fccfb9000 0x7fcdfb9000 0x1000000 0x0

* Looks like FEX JIT mapping, we start out at 16MB but scale up to 128MB
* Depending on version of FEX we can check the base mapping for a unique string
(gdb) p (char*)0x7fccfb9000
$4 = 0x7fccfb9000 "FEXJIT::Arm64JITCore::"

== Getting RIP of current code block ==
FEX sets up an address in our CPU context to get some debug data out.

Currently this isn't exposed in a way that a debugger can see other than manually typing out gdb commands

p/x *(uint64_t*)($x28) = inline block header ptr
p/x *(uint32_t*)(*(uint64_t*)($x28)) = OffsetToBlockTail
p/x *(uint64_t*)($x28) + *(uint32_t*)(*(uint64_t*)($x28))

p/x *(unsigned long long*)((*(unsigned long long*)($x28) + *(unsigned int*)(*(unsigned long long*)($x28)))+8) = RIP of block

disas *(unsigned long long*)($x28),+*(unsigned long long*)(*(unsigned long long*)($x28) + *(unsigned int*)(*(unsigned long long*)($x28))) = Disassemble this block of code.

While hard to decipher here is basically what is happening.
- x28 is the CPU register that FEX keeps in the JIT for context accesses
- offset 184 is the offset of the `InlineJITBlockHeader` member inside of that context.
- As long as FEX is in a JIT block that offset will be valid to point to the current RIP that the block is operating on.

Getting the stack can also be very useful

x/64wx ((FEXCore::Core::CPUState*)$x28)->gregs[FEXCore::X86State::REG_RSP]

Doing raw pointer math here means that this works even when gdb fails to find symbols for the CPUState object, which happens very frequently for some reason.

Development:Debugging Crash In Thunks

2025-09-22T14:00:29Z

Neobrain: FEXInterpreter -> FEX

[[Category:Development]]
== Once Thunks are enabled ==
* Once you've verified your crash is in a thunk then this becomes quite a bit easier

gdb --args FEX /usr/bin/vulkaninfo
Thread 1 "FEX" received signal SIGSEGV, Segmentation fault.
0x0000007fc82240f4 in tu_CreateDevice () from /lib/aarch64-linux-gnu/libvulkan_freedreno.so
(gdb) bt
#0 0x0000007fc82240f4 in tu_CreateDevice () from /lib/aarch64-linux-gnu/libvulkan_freedreno.so
#1 0x0000007fc8642b94 in function_traits<decltype (&(fexfn_unpack_libvulkan_freedreno_vkCreateDevice(fexfn_packed_args_libvulkan_freedreno_vkCreateDevice*)))>::result_t fexfn_type_erased_unpack<&(fexfn_unpack_libvulkan_freedreno_vkCreateDevice(fexfn_packed_args_libvulkan_freedreno_vkCreateDevice*))>(void*) () from /usr/lib/fex-emu/HostThunks//libvulkan_freedreno-host.so
#2 0x0000007fe25e4028 in ?? ()
Backtrace stopped: previous frame inner to this frame (corrupt stack?)

* Okay, that's a direct crash in Turnip, let's rebuild Mesa with a debug build and try again
Thread 1 "FEX" received signal SIGSEGV, Segmentation fault.
0x0000007fc80bc600 in tu_CreateDevice (physicalDevice=0x5559947aa0, pCreateInfo=0x7ff75fda20, pAllocator=0x0, pDevice=0x7fcbe7b358) at ../src/freedreno/vulkan/tu_device.c:1632
1632 memset(ptr, 0x42, 16);
(gdb) bt
#0 0x0000007fc80bc600 in tu_CreateDevice (physicalDevice=0x5559947aa0, pCreateInfo=0x7ff75fda20, pAllocator=0x0, pDevice=0x7fcbe7b358) at ../src/freedreno/vulkan/tu_device.c:1632
#1 0x0000007fc80eab9c in vk_tramp_CreateDevice (physicalDevice=0x5559947aa0, pCreateInfo=0x7ff75fda20, pAllocator=0x0, pDevice=0x7fcbe7b358) at src/vulkan/runtime/vk_dispatch_trampolines.c:78
#2 0x0000007fc8642b94 in function_traits<decltype (&(fexfn_unpack_libvulkan_freedreno_vkCreateDevice(fexfn_packed_args_libvulkan_freedreno_vkCreateDevice*)))>::result_t fexfn_type_erased_unpack<&(fexfn_unpack_libvulkan_freedreno_vkCreateDevice(fexfn_packed_args_libvulkan_freedreno_vkCreateDevice*))>(void*) () from /usr/lib/fex-emu/HostThunks//libvulkan_freedreno-host.so
#3 0x0000007fe25e426c in ?? ()

* Okay the crash is at.. `tu_device:1632`, let's go and debug that line

1631 void* ptr = (void*)1;
1632 memset(ptr, 0x42, 16);

* Who put that in there?!

Development:Setting up FEX

2025-09-22T14:00:03Z

Neobrain: FEXInterpreter -> FEX

[[Category:Development]]
== Build dependencies ==
=== Debian/Ubuntu dependencies ===
* git
* cmake
* ninja-build
* pkgconf
* ccache (Recommended if you're rebuilding constantly)
* clang
* llvm
* lld (For ThinLTO)
* binfmt-support (For AArch64 binfmt_misc support)
* libssl-dev
* python-setuptools (For build scripts)
* g++-x86-64-linux-gnu (For building thunks)
* libgcc-XX-dev-i386-cross (For building thunks, version should match your g++-x86-64-linux-gnu package)
* libgcc-XX-dev-amd64-cross (For building thunks, version should match your g++-x86-64-linux-gnu package)
* nasm (only if building tests)
* python3-clang (for struct verifier)
* libstdc++-12-dev-i386-cross (for struct verifier)
* libstdc++-12-dev-amd64-cross (for struct verifier)
* libstdc++-12-dev-arm64-cross (for struct verifier)
* squashfs-tools (For squashfs rootfs support)
* squashfuse (For mounting the squashfs)
* libc-bin or glibc-tools (For unittests, needs catchsegv from this)
* libc6-dev-i386-amd64-cross (For FEXLinuxTests)
* lib32stdc++-XX-dev-amd64-cross (For FEXLinuxTests, version should match your g++-x86-64-linux-gnu package)
* qtdeclarative5-dev (For GUI)
* qml-module-qtquick-controls (For GUI)
* qml-module-qtquick-controls2 (For GUI)
* qml-module-qtquick-dialogs (For GUI)

=== Gentoo dependencies ===
* dev-vcs/git
* cmake
* ninja
* ccache
* clang
* lld
* nasm

'''Use gubbins'''
* >=media-libs/libglvnd-1.3.4 X

=== Fedora dependencies ===
* git
* cmake
* ninja-build
* pkg-config
* ccache
* clang
* lld
* llvm
* llvm-devel
* openssl-devel
* nasm
* python3-clang
* python3-setuptools
* squashfs-tools
* squashfuse
* erofs-fuse
* erofs-utils
* qt5-qtdeclarative-devel
* qt5-qtquickcontrols
* qt5-qtquickcontrols2

== Build Configuration ==
* Ensure release mode is enabled
* Disable test building
* Enable LTO in a release build
* Enable LLD to make LTO builds faster

git clone --recurse-submodules https://github.com/FEX-Emu/FEX.git
cd FEX
mkdir Build
cd Build
CC=clang CXX=clang++ cmake -DCMAKE_INSTALL_PREFIX=/usr -DCMAKE_BUILD_TYPE=Release -DUSE_LINKER=lld -DENABLE_LTO=True -DBUILD_TESTING=False -DENABLE_ASSERTIONS=False -G Ninja ..
ninja

== Installation ==
sudo ninja install

You can install a binfmt_misc handler for both 32bit and 64bit x86 execution directly from the environment.
If you already have box86's 32bit binfmt_misc handler installed then make sure to uninstall their's first.
Make sure to have run install prior to this, otherwise binfmt_misc will install an old handler even if the executable has been updated.

sudo ninja binfmt_misc

=== Quirks ===

==== binfmt_misc problems ====
* Double check that binfmt_misc has worked
** `ls /usr/share/binfmts/FEX*` should return files
** `ls /var/lib/binfmts/FEX*` should return files
** `/proc/sys/fs/binfmt_misc/status` should exist
*** If it doesn't exist then make sure your kernel has `CONFIG_BINFMT_MISC=y`
*** If it still doesn't exist then make sure systemd isn't disabling it `systemctl enable binfmt-support`

==== DESTDIR quirks ====
* When using DESTDIR, make sure the path is absolute instead of relative
** FEX's thunks use subprojects because it needs to cross-compile
** If DESTDIR is relative then the install paths of the subprojects go to the wrong location
** Using $(pwd)/install works well

== Runtime Configuration ==
FEXConfig is the application to use for runtime configuration setup.

CTLR+S saves the configuration.

=== Things to ensure to set ===
On AArch64 host, a rootfs path is mandatory, while it is optional on x86-64 host

* Core: JIT
* RootFS
* Silent Logging (Useful to get FEX information spam out of the way)
* Block Size (500 is a good default. Optimization passes may break things though)

More Details at: Follow the instructions [[Development:Configuring_FEX|here]]

== RootFS generation ==
AArch64 hosts require a rootfs for running applications. FEX provides various x86 images that can be installed through [[Development:Setting_up_RootFS#Quick_Setup_with_FEXRootFSFetcher|FEXRootFSFetcher]]

For more detailed instructions see [[Development:Setting_up_RootFS]]

== Setting up library forwarding ==

Follow the instructions [[Development:Setting_up_Library_Forwarding|here]].

== Binaries ==
* FEX
** This is the emulator executable used to run x86 programs
* FEXInterpreter
** This is a (deprecated) alias for ''FEX'', provided for backwards compatibility
* FEXConfig
** Used for configuring emulation from a Qt-based GUI
* FEXBash
** Can be useful for starting a bash instance that is running under emulation
** It's not a chroot! Don't use sudo inside of this environment!
* FEXMountDaemon
** This is a mount daemon that will run in the background when you are using a squashfs based rootfs
** Usually one one instance open at any given moment
** It usually closes when the FEXInterpreter exits, but not always. See the problems section in setting up a rootfs
*** Has a ten second timeout window in case a another instance of FEX opens and uses it.

=== Debug binaries ===
* TestHarnessRunner
** Used for running FEX's ASM unit tests
** Runs some binary in a host x86_64 host environment or FEX and checks results
** Useful for quickly creating ASM based unit tests
* IRLoader
** Used for running FEX's IR based unit tests
** Only runs through FEX and checks the results
** Minimal number of tests
* UnitTestGenerator
** This is supposed to be an x86-64 instruction generator
** Has an understanding of x86-64 instruction encoding format
** Generates hundreds of thousands of instructions
** Used to be used with a lockstep runner to ensure correct behaviour
** Largely unsupported, avoiding undefined behaviour is difficult and FEX doesn't need to match undefined behaviour
** Lockstep hostrunner wasn't ever made to be bug free, hard to support
** Doesn't ensure correct behaviour on AArch64 device

Development:Adding Config Options

2025-09-13T08:18:53Z

Neobrain: FEXLoader was removed

[[Category:Development]]
Configuration options are programatically generated from json.

This file lives in `FEXCore/Source/Interface/Config/Config.json`

= Options section =
To add a configuration option that can be set from JSON or from environment variables, you need to add the option to this file.

Currently FEX has six groups to group configs
* CPU
* Emulation
* Debug
* Logging
* Hacks
* Misc

== Configuration option object ==
Each JSON object under a group becomes a configuration.

'''Object Layout'''
* Key:
* Becomes the C++ enum. eg. 'Multiblock' becomes `FEXCore::Config::ConfigOption::CONFIG_MULTIBLOCK`
* Also used for the environment variable. Becomes `FEX_MULTIBLOCK`
* If the object type is boolean then a negated argument also is generated. `--no-multiblock`

* Type:
* Can be `[bool, uint8, int32, uint32, uint64, str, strarray]`
* This is what the argument type ends up being once loaded from a configuration backing

* Default:
* Default value that matches this object
* Used to generate code so needs to handle being codified. Compilation will fail otherwise
* If Type is str or strarray then this string will be automatically wrapped with double-quotes when generated

* TextDefault:
* If the `Default` option which is used in code can't be represented by text. Like an enum being used.
* eg. `Core` has a `Default` which is an enum. `TextDefault` is set to `irjit` for the user to handle this case

* Choices:
* Allows us to represent the textual options that a argument can accept
* eg. `Core` supports `[ "irint", "irjit", "host" ]`

* ArgumentHandler:
* If an argument option needs some special handling to be parsed then this points to a C++ function in the argument loader
* `Core` uses it to convert the textual representation in to an enum

* Desc:
* Short description of what this option does
* This is used to generate the text in the man pages

== UnnamedOptions section ==
These are configuration options that can '''NOT''' be set by user means. There won't be any Environment variable or json helpers generated to load these.

These need to exist as a way to have internal configuration options existing in the same global space as user options. Allows us to pull internal configuration options the same way as user configurations.

These can exist under any group config. Currently only `Misc` is used. Only three things need to be set for these
* Key: This is only used to generate the enum in code
* Type: Same supported types as the regular options
* Default: The default value of this configuration

== Getting configurations in FEX code ==
FEX provides a `FEX_CONFIG_OPT` define to help developers get configuration options.
* FEX_CONFIG_OPT(name, enum)
* eg: `FEX_CONFIG_OPT(Cores, THREADS);`
* This will load the THREADS configuration option from the configuration directory and save the value in to local value called `Cores`
* Gotchas:
* Construction of the object queries the the global configuration layer. This can be slow so be careful with its use.
* Upon construction of the `FEXCore::Config::Value<T>` object it will pull the string configuration
* Then it will convert that value in to whatever type the value needs
* In high performance code, keep constructor instantiation outside of the hot code path
* Construct the configuration inside of a class definition so it only pulls the configuration once
* Caveats:
* If the configuration changes then this doesn't pick up the change
* Configuration isn't meant to change live in FEX so this is typically a non-issue
* Frontend object loader needs to be careful with its configuration loading to ensure the meta layer stays in sync
* ELFLoader.cpp for examples here

Development:Configuration

2025-09-13T08:14:17Z

Neobrain: FEXLoader was removed

FEX-Emu supports a layered configuration system that can be confusing to understand at face value. This is page to hopefully explain how FEX's configuration system works.

= FEX Configuration layers =
FEX supports multiple layers to its configuration. Many of which are available for users to modify to their liking.

'''List of layers - Lower layers overwriting configurations from upper layers'''
* Global Primary configuration - '''Read-only'''
* User Primary configuration
* User Arguments configuration
* Global application configuration - '''Read-only'''
* User application configuration
* User environment variable configuration

== Global Primary configuration ==
'''/usr/share/fex-emu/Config.json'''

This configuration file is a global read-only configuration that is only used in very select circumstances. It is not expected that a user will need or want this since it is read-only.

This is FEX's standard JSON configuration format.

== User Primary configuration ==
'''$HOME/.fex-emu/Config.json'''

This is the primary configuration file that is expected to be changed most frequently by the user. If the user is modifying configuration with FEXConfig, this is the file that should be saved.

This is FEX's standard JSON configuration format.

== Global application configuration ==
'''/usr/share/fex-emu/AppConfig/<Application Name>.json'''

This is a read-only configuration for application specific options. These are usually workarounds or trivial workarounds that FEX needs for specific applications. It's not expected that a user will need to modify these.

This is FEX's standard JSON configuration format.

== User application configuration ==
'''$HOME/.fex-emu/AppConfig/<Application Name>.json'''

This is a user provided application configuration. This is useful for users to set their own options for specific applications.
* Very useful for only enabling Thunks for specific applications. Where global enabling can cause too many crashes.

This is FEX's standard JSON configuration format.

== User environment variable configuration ==
'''Environment variables starting with FEX_'''

These are useful for testing various options. While in most cases the environment variables will pass through to child processes. It still isn't guaranteed if the application is modifying the environment before executing a child process.

These can be found under '''man FEX''' under the '''ENVIRONMENT''' section. The environment name is just '''FEX_''' prepended to an all capitalized json key name.

Development:Setting up FEX

2025-09-13T08:13:13Z

Neobrain: FEXLoader was removed

[[Category:Development]]
== Build dependencies ==
=== Debian/Ubuntu dependencies ===
* git
* cmake
* ninja-build
* pkgconf
* ccache (Recommended if you're rebuilding constantly)
* clang
* llvm
* lld (For ThinLTO)
* binfmt-support (For AArch64 binfmt_misc support)
* libssl-dev
* python-setuptools (For build scripts)
* g++-x86-64-linux-gnu (For building thunks)
* libgcc-XX-dev-i386-cross (For building thunks, version should match your g++-x86-64-linux-gnu package)
* libgcc-XX-dev-amd64-cross (For building thunks, version should match your g++-x86-64-linux-gnu package)
* nasm (only if building tests)
* python3-clang (for struct verifier)
* libstdc++-12-dev-i386-cross (for struct verifier)
* libstdc++-12-dev-amd64-cross (for struct verifier)
* libstdc++-12-dev-arm64-cross (for struct verifier)
* squashfs-tools (For squashfs rootfs support)
* squashfuse (For mounting the squashfs)
* libc-bin or glibc-tools (For unittests, needs catchsegv from this)
* libc6-dev-i386-amd64-cross (For FEXLinuxTests)
* lib32stdc++-XX-dev-amd64-cross (For FEXLinuxTests, version should match your g++-x86-64-linux-gnu package)
* qtdeclarative5-dev (For GUI)
* qml-module-qtquick-controls (For GUI)
* qml-module-qtquick-controls2 (For GUI)
* qml-module-qtquick-dialogs (For GUI)

=== Gentoo dependencies ===
* dev-vcs/git
* cmake
* ninja
* ccache
* clang
* lld
* nasm

'''Use gubbins'''
* >=media-libs/libglvnd-1.3.4 X

=== Fedora dependencies ===
* git
* cmake
* ninja-build
* pkg-config
* ccache
* clang
* lld
* llvm
* llvm-devel
* openssl-devel
* nasm
* python3-clang
* python3-setuptools
* squashfs-tools
* squashfuse
* erofs-fuse
* erofs-utils
* qt5-qtdeclarative-devel
* qt5-qtquickcontrols
* qt5-qtquickcontrols2

== Build Configuration ==
* Ensure release mode is enabled
* Disable test building
* Enable LTO in a release build
* Enable LLD to make LTO builds faster

git clone --recurse-submodules https://github.com/FEX-Emu/FEX.git
cd FEX
mkdir Build
cd Build
CC=clang CXX=clang++ cmake -DCMAKE_INSTALL_PREFIX=/usr -DCMAKE_BUILD_TYPE=Release -DUSE_LINKER=lld -DENABLE_LTO=True -DBUILD_TESTING=False -DENABLE_ASSERTIONS=False -G Ninja ..
ninja

== Installation ==
sudo ninja install

You can install a binfmt_misc handler for both 32bit and 64bit x86 execution directly from the environment.
If you already have box86's 32bit binfmt_misc handler installed then make sure to uninstall their's first.
Make sure to have run install prior to this, otherwise binfmt_misc will install an old handler even if the executable has been updated.

sudo ninja binfmt_misc

=== Quirks ===

==== binfmt_misc problems ====
* Double check that binfmt_misc has worked
** `ls /usr/share/binfmts/FEX*` should return files
** `ls /var/lib/binfmts/FEX*` should return files
** `/proc/sys/fs/binfmt_misc/status` should exist
*** If it doesn't exist then make sure your kernel has `CONFIG_BINFMT_MISC=y`
*** If it still doesn't exist then make sure systemd isn't disabling it `systemctl enable binfmt-support`

==== DESTDIR quirks ====
* When using DESTDIR, make sure the path is absolute instead of relative
** FEX's thunks use subprojects because it needs to cross-compile
** If DESTDIR is relative then the install paths of the subprojects go to the wrong location
** Using $(pwd)/install works well

== Runtime Configuration ==
FEXConfig is the application to use for runtime configuration setup.

CTLR+S saves the configuration.

=== Things to ensure to set ===
On AArch64 host, a rootfs path is mandatory, while it is optional on x86-64 host

* Core: JIT
* RootFS
* Silent Logging (Useful to get FEX information spam out of the way)
* Block Size (500 is a good default. Optimization passes may break things though)

More Details at: Follow the instructions [[Development:Configuring_FEX|here]]

== RootFS generation ==
AArch64 hosts require a rootfs for running applications. FEX provides various x86 images that can be installed through [[Development:Setting_up_RootFS#Quick_Setup_with_FEXRootFSFetcher|FEXRootFSFetcher]]

For more detailed instructions see [[Development:Setting_up_RootFS]]

== Setting up library forwarding ==

Follow the instructions [[Development:Setting_up_Library_Forwarding|here]].

== Binaries ==
* FEXInterpreter
** This is the emulator executable used to run x86 programs
* FEXConfig
** Used for configuring emulation from a Qt-based GUI
* FEXBash
** Can be useful for starting a bash instance that is running under emulation
** It's not a chroot! Don't use sudo inside of this environment!
* FEXMountDaemon
** This is a mount daemon that will run in the background when you are using a squashfs based rootfs
** Usually one one instance open at any given moment
** It usually closes when the FEXInterpreter exits, but not always. See the problems section in setting up a rootfs
*** Has a ten second timeout window in case a another instance of FEX opens and uses it.

=== Debug binaries ===
* TestHarnessRunner
** Used for running FEX's ASM unit tests
** Runs some binary in a host x86_64 host environment or FEX and checks results
** Useful for quickly creating ASM based unit tests
* IRLoader
** Used for running FEX's IR based unit tests
** Only runs through FEX and checks the results
** Minimal number of tests
* UnitTestGenerator
** This is supposed to be an x86-64 instruction generator
** Has an understanding of x86-64 instruction encoding format
** Generates hundreds of thousands of instructions
** Used to be used with a lockstep runner to ensure correct behaviour
** Largely unsupported, avoiding undefined behaviour is difficult and FEX doesn't need to match undefined behaviour
** Lockstep hostrunner wasn't ever made to be bug free, hard to support
** Doesn't ensure correct behaviour on AArch64 device

Development:Setting up FEX

2025-09-11T12:12:09Z

Neobrain: BUILD_TESTS -> BUILD_TESTING

[[Category:Development]]
== Build dependencies ==
=== Debian/Ubuntu dependencies ===
* git
* cmake
* ninja-build
* pkgconf
* ccache (Recommended if you're rebuilding constantly)
* clang
* llvm
* lld (For ThinLTO)
* binfmt-support (For AArch64 binfmt_misc support)
* libssl-dev
* python-setuptools (For build scripts)
* g++-x86-64-linux-gnu (For building thunks)
* libgcc-XX-dev-i386-cross (For building thunks, version should match your g++-x86-64-linux-gnu package)
* libgcc-XX-dev-amd64-cross (For building thunks, version should match your g++-x86-64-linux-gnu package)
* nasm (only if building tests)
* python3-clang (for struct verifier)
* libstdc++-12-dev-i386-cross (for struct verifier)
* libstdc++-12-dev-amd64-cross (for struct verifier)
* libstdc++-12-dev-arm64-cross (for struct verifier)
* squashfs-tools (For squashfs rootfs support)
* squashfuse (For mounting the squashfs)
* libc-bin or glibc-tools (For unittests, needs catchsegv from this)
* libc6-dev-i386-amd64-cross (For FEXLinuxTests)
* lib32stdc++-XX-dev-amd64-cross (For FEXLinuxTests, version should match your g++-x86-64-linux-gnu package)
* qtdeclarative5-dev (For GUI)
* qml-module-qtquick-controls (For GUI)
* qml-module-qtquick-controls2 (For GUI)
* qml-module-qtquick-dialogs (For GUI)

=== Gentoo dependencies ===
* dev-vcs/git
* cmake
* ninja
* ccache
* clang
* lld
* nasm

'''Use gubbins'''
* >=media-libs/libglvnd-1.3.4 X

=== Fedora dependencies ===
* git
* cmake
* ninja-build
* pkg-config
* ccache
* clang
* lld
* llvm
* llvm-devel
* openssl-devel
* nasm
* python3-clang
* python3-setuptools
* squashfs-tools
* squashfuse
* erofs-fuse
* erofs-utils
* qt5-qtdeclarative-devel
* qt5-qtquickcontrols
* qt5-qtquickcontrols2

== Build Configuration ==
* Ensure release mode is enabled
* Disable test building
* Enable LTO in a release build
* Enable LLD to make LTO builds faster

git clone --recurse-submodules https://github.com/FEX-Emu/FEX.git
cd FEX
mkdir Build
cd Build
CC=clang CXX=clang++ cmake -DCMAKE_INSTALL_PREFIX=/usr -DCMAKE_BUILD_TYPE=Release -DUSE_LINKER=lld -DENABLE_LTO=True -DBUILD_TESTING=False -DENABLE_ASSERTIONS=False -G Ninja ..
ninja

== Installation ==
sudo ninja install

You can install a binfmt_misc handler for both 32bit and 64bit x86 execution directly from the environment.
If you already have box86's 32bit binfmt_misc handler installed then make sure to uninstall their's first.
Make sure to have run install prior to this, otherwise binfmt_misc will install an old handler even if the executable has been updated.

sudo ninja binfmt_misc

=== Quirks ===

==== binfmt_misc problems ====
* Double check that binfmt_misc has worked
** `ls /usr/share/binfmts/FEX*` should return files
** `ls /var/lib/binfmts/FEX*` should return files
** `/proc/sys/fs/binfmt_misc/status` should exist
*** If it doesn't exist then make sure your kernel has `CONFIG_BINFMT_MISC=y`
*** If it still doesn't exist then make sure systemd isn't disabling it `systemctl enable binfmt-support`

==== DESTDIR quirks ====
* When using DESTDIR, make sure the path is absolute instead of relative
** FEX's thunks use subprojects because it needs to cross-compile
** If DESTDIR is relative then the install paths of the subprojects go to the wrong location
** Using $(pwd)/install works well

== Runtime Configuration ==
FEXConfig is the application to use for runtime configuration setup.

CTLR+S saves the configuration.

=== Things to ensure to set ===
On AArch64 host, a rootfs path is mandatory, while it is optional on x86-64 host

* Core: JIT
* RootFS
* Silent Logging (Useful to get FEX information spam out of the way)
* Block Size (500 is a good default. Optimization passes may break things though)

More Details at: Follow the instructions [[Development:Configuring_FEX|here]]

== RootFS generation ==
AArch64 hosts require a rootfs for running applications. FEX provides various x86 images that can be installed through [[Development:Setting_up_RootFS#Quick_Setup_with_FEXRootFSFetcher|FEXRootFSFetcher]]

For more detailed instructions see [[Development:Setting_up_RootFS]]

== Setting up library forwarding ==

Follow the instructions [[Development:Setting_up_Library_Forwarding|here]].

== Binaries ==
* FEXLoader
** This is useful for development purposes but is generally recommended against using.
** Allows passing arguments to FEX for configuration purposes, which can break under some execve situations
* FEXInterpreter
** This is a hardlink to FEXLoader which refuses to treat arguments as configuration options
** Is recommended to use over FEXLoader; Using FEXConfig to set configuration
** Requires being a hardlink or executable copy, execve logic sees through a softlink
** Is the executable used for binfmt_misc for interpreters
* FEXConfig
** Used for configuring emulation from a Qt-based GUI
* FEXBash
** Can be useful for starting a bash instance that is running under emulation
** It's not a chroot! Don't use sudo inside of this environment!
* FEXMountDaemon
** This is a mount daemon that will run in the background when you are using a squashfs based rootfs
** Usually one one instance open at any given moment
** It usually closes when the FEXInterpreter exits, but not always. See the problems section in setting up a rootfs
*** Has a ten second timeout window in case a another instance of FEX opens and uses it.

=== Debug binaries ===
* TestHarnessRunner
** Used for running FEX's ASM unit tests
** Runs some binary in a host x86_64 host environment or FEX and checks results
** Useful for quickly creating ASM based unit tests
* IRLoader
** Used for running FEX's IR based unit tests
** Only runs through FEX and checks the results
** Minimal number of tests
* UnitTestGenerator
** This is supposed to be an x86-64 instruction generator
** Has an understanding of x86-64 instruction encoding format
** Generates hundreds of thousands of instructions
** Used to be used with a lockstep runner to ensure correct behaviour
** Largely unsupported, avoiding undefined behaviour is difficult and FEX doesn't need to match undefined behaviour
** Lockstep hostrunner wasn't ever made to be bug free, hard to support
** Doesn't ensure correct behaviour on AArch64 device

Development:InstCountCI

2025-09-11T12:11:54Z

Neobrain: BUILD_TESTS -> BUILD_TESTING

[[Category:Development]]
InstCountCI is a continuous integration tool that FEX-Emu uses to ensure that instruction implementations aren't getting worse over time.
== Getting Started ==
Make sure to follow [[Development:Setting_up_FEX]] to get an initial build environment set up.

=== What you need ===
* An Arm64 Linux device that can build FEX
** An x86-64 device using the VIXL simulator can be used as a substitute.

=== Additional cmake options ===
Some additional cmake options need to be passed to the FEX-Emu cmake options to get the tests building.

* -DBUILD_TESTING=True
* -DENABLE_VIXL_DISASSEMBLER=True

=== Quality of life improvements ===
Add these cmake options to make iteration time faster and have debug assertions to catch problems.
* -DENABLE_LTO=False
* -DCMAKE_BUILD_TYPE=RelWithDebInfo
* -DENABLE_ASSERTIONS=True

=== Running InstCountCI ===
First thing you need to build the tests. This step will parse all the json files inside of '''unittests/InstructionCountCI/''' and set up running CI in the next step.
* ninja instcountci_test_files

Next you need to actually run the tests. This will run all the instructions declared in '''unittests/InstructionCountCI/*.json'''. If this step fails, that is okay since that just means that either an instruction translation has gotten worse, or if the test crashed then something catastrophic happened.
* ninja instcountci_tests

The next step is to take the data generated from the previous step and modify the resulting json that is tracked by git.
* ninja instcountci_update_tests

Now to see how the implementations have changed, you can just run '''git diff''' to see how the json files in '''unittests/InstructionCountCI/''' have changed.

=== Example ===
Minor improvement when optimizing a move instruction. From [https://github.com/FEX-Emu/FEX/pull/2972 This] pull request.
"movd mm0, eax": {
- "ExpectedInstructionCount": 3,
+ "ExpectedInstructionCount": 2,
"Comment": "0x0f 0x6e",
"ExpectedArm64ASM": [
- "ubfx x20, x4, #0, #32",
- "fmov s4, w20",
+ "fmov s4, w4",
"str d4, [x28, #752]"
]
},
* Reset the files with '''git checkout -- unittests/InstructionCountCI/*.json''' if the changes weren't desired.

== What classifies as an optimal translation? ==
Nothing classifies if an instruction implementation is considered optimal or not. This is left up to the human to try and understand if the translation is optimal for that particular instruction.
* In general the ''Optimal'' tag is just a guideline since humans could have made a mistake and the instruction could be further optimized.
* Alternatively ARM could introduce a new set of instructions that improve how optimal an instruction could be.
* Further an instruction could be considered optimal, but the reviewer ignored something like flag generation around it since that is a systemic FEX-Emu issue.
* Or of course, just human error and it was misunderstood as an optimal implementation but someone found a way to do it better.

This is more just a tag to help humans when they are doing code auditing so they don't need to pay as much attention to the ones that are classified as such. It's still good to periodically go over these implementations and see if things could be done better.

== Diving deeper in to the assembly ==
=== Manually run instcountci result ===
A useful first step might be to run the json tests directly in the code size validation program. This can be done from the build directory.

* eg:
./Bin/CodeSizeValidation unittests/InstructionCountCI/FEXOpt/libnss.json.instcountci

=== Run the test through the TestHarnessRunner ===
While the instruction count CI is good at showing the final result, it isn't the best at showing what FEX did to get to that result. This is where the assembly test harness can come in handy.
* Create a file in '''unittests/ASM/Test.asm'''
* Add the following data:
%ifdef CONFIG
{
}
%endif
addps xmm0, xmm1
hlt
* Recompile the asm tests with `ninja asm_files`
* Run the assembly test manually now with '''FEX_DUMPIR=stderr FEX_DISASSEMBLE=blocks ./Bin/TestHarnessRunner -c irjit -n 1 -g ./unittests/ASM/Test.asm.bin ./unittests/ASM/Test.asm.config.bin'''
** This will dump both FEX's internal IR and the disassembly of the code for each instruction
** The second code block for the '''hlt''' can be ignored. It is just necessary for this test harness to run.

The resulting output will be:
IR-post 0x10000:
(%0) IRHeader %2, #65536, #0, #1
(%2) CodeBlock %3, %10
(%3 i0) BeginBlock %2(Invalid)
%4(FPRFixed1) i128 = LoadRegister #0x0, #0xd0, FPR, FPRFixed, u8:Tmp:Size
%5(FPRFixed0) i128 = LoadRegister #0x0, #0xc0, FPR, FPRFixed, u8:Tmp:Size
%6(FPRFixed0) i32v4 = VFAdd u8:Tmp:RegisterSize, u8:Tmp:ElementSize, %5(FPRFixed0) i128, %4(FPRFixed1) i128
(%7 i128) StoreRegister %6(FPRFixed0) i32v4, #0x0, #0xc0, FPR, FPRFixed, u8:Tmp:Size
(%8 i64) InlineEntrypointOffset #0x3, u8:Tmp:RegisterSize
(%9 i64) ExitFunction %8(Invalid)
(%10 i0) EndBlock %2(Invalid)
@@@@@
[INFO] Disassemble Begin
[INFO] adr x0, #-0x4 (addr 0xffff6fa00018)
[INFO] str x0, [x28, #184]
[INFO] fadd v16.4s, v16.4s, v17.4s
[INFO] ldr x0, pc+8 (addr 0xffff6fa00030)
[INFO] blr x0
[INFO] unallocated (Unallocated)
[INFO] udf #0xffff
[INFO] unallocated (Unallocated)
[INFO] udf #0x0
[INFO] Disassemble End

* The disassembly has some instructions at the start and end which are necessary for the JIT to run
** InstCountCI strips this code out automatically.
* In a vacuum of a single instruction, the code block header and tail can dominate the code size.
** It's recommended to become familiar with what the header and tail look like and ignore it in the resulting code generation.
** Currently the header is the first two instructions '''adr+str'''
** Currently the tail starts with the '''ldr+blr''' after the '''fadd''' and continues with some metadata afterwards.

=== Diving Deeper ===
I would recommend looking at the man page for FEX to see additional options that can be useful
* Specifically the options for '''FEX_PASSMANAGERDUMPIR''' to get more IR dumping options and '''FEX_HOSTFEATURES''' to fake CPU feature support.
* Enabling the vixl simulator with the cmake option '''-DENABLE_VIXL_SIMULATOR=True''' can be useful to test features your CPU doesn't support!

Development:ARM64EC

2025-09-11T12:11:37Z

Neobrain: BUILD_TESTS -> BUILD_TESTING

[[Category:Development]]
=== Building ===
These instructions expect you to be building on an ARM64 host.

==== Toolchain ====
Scripts and instructions to build an ARM64EC toolchain can be found [https://github.com/bylaws/llvm-mingw here], with prebuilt binaries for ARM64/x86 hosts on the [https://github.com/bylaws/llvm-mingw/releases/tag/20240929 releases page]. Once built/downloaded, add it to your environment with
export PATH="<path to toolchain>/bin:$PATH"
note the toolchain must come before all other path entries as it needs to override the host ar binary.
* Make sure to grab the tar.gz aarch64 files when compiling on Linux. The zip files are for a Windows toolchain which doesn't work for these instructions.

==== Wine ====
[https://gitlab.winehq.org/wine/wine Upstream] wine has various things missing for full FEX/ARM64EC support, while it can be used, [https://github.com/bylaws/wine/tree/upstream-arm64ec my fork] will give the best results for now. Make sure to install the deps listed [https://salsa.debian.org/wine-team/wine/-/blob/debian/9.0_repack-4/debian/control.in?ref_type=tags#L13 here] before building with the ARM64EC toolchain.

./configure --enable-archs=arm64ec,aarch64,i386 --prefix=/usr --with-mingw=clang --disable-tests
make -j$(nproc)
sudo --preserve-env=PATH make install -j$(nproc)

==== FEX ====
FEX must be built twice, once for the x86 (WOW64) emulation module and once for x86_64 emulation module (ARM64EC)

FEX's official [https://launchpad.net/~fex-emu/+archive/ubuntu/fex Ubuntu PPA] now provides a fex-emu-wine package of the latest release. This package can be downloaded and the DLL files can be extracted if compiling isn't desired.

===== Build directions =====
mkdir build-arm64ec
cd build-arm64ec
cmake -GNinja -DCMAKE_BUILD_TYPE=Release -DCMAKE_TOOLCHAIN_FILE=../Data/CMake/toolchain_mingw.cmake -DCMAKE_INSTALL_LIBDIR=/usr/lib/wine/aarch64-windows -DENABLE_LTO=False -DMINGW_TRIPLE=arm64ec-w64-mingw32 -DBUILD_TESTING=False -DENABLE_JEMALLOC_GLIBC_ALLOC=False -DCMAKE_INSTALL_PREFIX=/usr ..
ninja
sudo ninja install
cd ..
mkdir build-wow64
cd build-wow64
cmake -GNinja -DCMAKE_BUILD_TYPE=Release -DCMAKE_TOOLCHAIN_FILE=../Data/CMake/toolchain_mingw.cmake -DCMAKE_INSTALL_LIBDIR=/usr/lib/wine/aarch64-windows -DENABLE_LTO=False -DMINGW_TRIPLE=aarch64-w64-mingw32 -DBUILD_TESTING=False -DENABLE_JEMALLOC_GLIBC_ALLOC=False -DCMAKE_INSTALL_PREFIX=/usr ..
ninja
sudo ninja install
cd ..

==== dxvk/vkd3d-proton ====

Both can be built and installed as instructed on their respective github pages, except that a path to the following cross file should be passed to meson with --cross-file:
[binaries]
ar = 'arm64ec-w64-mingw32-ar'
c = 'arm64ec-w64-mingw32-gcc'
cpp = 'arm64ec-w64-mingw32-g++'
ld = 'arm64ec-w64-mingw32-ld'
windres = 'arm64ec-w64-mingw32-windres'
strip = 'strip'
widl = 'arm64ec-w64-mingw32-widl'
pkgconfig = 'aarch64-linux-gnu-pkg-config'
[host_machine]
system = 'windows'
cpu_family = 'aarch64'
cpu = 'aarch64'
endian = 'little'

=== Windows ===
ARM64EC FEX is usable as a drop-in replacement for xtajit64 on native WOA, build FEX with -DUSE_PDB_DEBUGINFO=True and copy the libarm64ecfex.{dll,pdb} binaries into C:/Windows/System32. To actually use FEX create an empty registry key at HKLM\\Software\\Microsoft\\Wow64\\amd64 and set its value to libarm64ecfex.dll, all subsequently launched programs will then use FEX. Logs/configuration is stored in %localappdata%.

=== Info ===
Unlike Linux FEX, the Windows FEX modules purely handle CPU emulation, with syscall translation being done on either the wine side (WOW64) or ABI (ARM64EC). This is similarly the case for wine's unixcalls and as a result native ARM64 variants can be used for all .so libraries (Vulkan, wine, etc.) dropping the need for an x86 rootfs and FEX-side thunking.

WOW64 (Windows 32-bit On Windows 64-bit) is the mechanism to run 32-bit applications on 64-bit platforms: All system libraries are built as x86 and emulated. Any 32-bit syscalls they use are adapted on-the-fly to 64-bit equivalents by injecting thunks.

ARM64EC (Emulation Compatible) on the other hand deals with 64-bit applications and avoids emulation overhead by using a dedicated set of system libraries compiled for ARM64 with the custom ARM64EC ABI. This ABI is designed to be callable from recompiled x86_64 code in order to allow a mix-and-match of native and emulated code. To avoid needing both a set of ARM64EC and ARM64 system libraries, ARM64X is introduced: It uses linker relocations to combine ARM64EC code and regular ARM64 code in a single binary file, which can be used both by standard Windows-on-Arm applications and emulated x86_64 ones.

The following pages serve as a good reference:

https://learn.microsoft.com/en-us/windows/arm/arm64ec

https://learn.microsoft.com/en-us/windows/arm/arm64ec-abi

https://gitlab.winehq.org/wine/wine/-/wikis/ARM64EC-Toolchain

Of note is the fact that FEX needs to run in a very reduced Windows environment - due to how early it is loaded it can only depend on ntdll and not say the system CRT. This is worked around by implementing/stubbing the subset of the CRT/Windows APIs necessary for FEX to run/link purely ontop of ntdll (see Common/Windows/{WinAPI,CRT}).

=== Building problems ===
==== `bytes of instructions in range, but .seh directives corresponding to` ====
This is seemingly caused by the `-mcpu` tune option, pass to cmake `-DTUNE_CPU=none`

Development:Setting up FEX

2025-07-14T10:12:19Z

Neobrain:

[[Category:Development]]
== Build dependencies ==
=== Debian/Ubuntu dependencies ===
* git
* cmake
* ninja-build
* pkgconf
* ccache (Recommended if you're rebuilding constantly)
* clang
* llvm
* lld (For ThinLTO)
* binfmt-support (For AArch64 binfmt_misc support)
* libssl-dev
* python-setuptools (For build scripts)
* g++-x86-64-linux-gnu (For building thunks)
* libgcc-XX-dev-i386-cross (For building thunks, version should match your g++-x86-64-linux-gnu package)
* libgcc-XX-dev-amd64-cross (For building thunks, version should match your g++-x86-64-linux-gnu package)
* nasm (only if building tests)
* python3-clang (for struct verifier)
* libstdc++-12-dev-i386-cross (for struct verifier)
* libstdc++-12-dev-amd64-cross (for struct verifier)
* libstdc++-12-dev-arm64-cross (for struct verifier)
* squashfs-tools (For squashfs rootfs support)
* squashfuse (For mounting the squashfs)
* libc-bin or glibc-tools (For unittests, needs catchsegv from this)
* libc6-dev-i386-amd64-cross (For FEXLinuxTests)
* lib32stdc++-XX-dev-amd64-cross (For FEXLinuxTests, version should match your g++-x86-64-linux-gnu package)
* qtdeclarative5-dev (For GUI)
* qml-module-qtquick-controls (For GUI)
* qml-module-qtquick-controls2 (For GUI)
* qml-module-qtquick-dialogs (For GUI)

=== Gentoo dependencies ===
* dev-vcs/git
* cmake
* ninja
* ccache
* clang
* lld
* nasm

'''Use gubbins'''
* >=media-libs/libglvnd-1.3.4 X

=== Fedora dependencies ===
* git
* cmake
* ninja-build
* pkg-config
* ccache
* clang
* lld
* llvm
* llvm-devel
* openssl-devel
* nasm
* python3-clang
* python3-setuptools
* squashfs-tools
* squashfuse
* erofs-fuse
* erofs-utils
* qt5-qtdeclarative-devel
* qt5-qtquickcontrols
* qt5-qtquickcontrols2

== Build Configuration ==
* Ensure release mode is enabled
* Disable test building
* Enable LTO in a release build
* Enable LLD to make LTO builds faster

git clone --recurse-submodules https://github.com/FEX-Emu/FEX.git
cd FEX
mkdir Build
cd Build
CC=clang CXX=clang++ cmake -DCMAKE_INSTALL_PREFIX=/usr -DCMAKE_BUILD_TYPE=Release -DUSE_LINKER=lld -DENABLE_LTO=True -DBUILD_TESTS=False -DENABLE_ASSERTIONS=False -G Ninja ..
ninja

== Installation ==
sudo ninja install -j4

You can install a binfmt_misc handler for both 32bit and 64bit x86 execution directly from the environment.
If you already have box86's 32bit binfmt_misc handler installed then make sure to uninstall their's first.
Make sure to have run install prior to this, otherwise binfmt_misc will install an old handler even if the executable has been updated.

sudo ninja binfmt_misc_32
sudo ninja binfmt_misc_64

=== Quirks ===

==== binfmt_misc problems ====
* Double check that binfmt_misc has worked
** `ls /usr/share/binfmts/FEX*` should return files
** `ls /var/lib/binfmts/FEX*` should return files
** `/proc/sys/fs/binfmt_misc/status` should exist
*** If it doesn't exist then make sure your kernel has `CONFIG_BINFMT_MISC=y`
*** If it still doesn't exist then make sure systemd isn't disabling it `systemctl enable binfmt-support`

==== DESTDIR quirks ====
* When using DESTDIR, make sure the path is absolute instead of relative
** FEX's thunks use subprojects because it needs to cross-compile
** If DESTDIR is relative then the install paths of the subprojects go to the wrong location
** Using $(pwd)/install works well

== Runtime Configuration ==
FEXConfig is the application to use for runtime configuration setup.

CTLR+S saves the configuration.

=== Things to ensure to set ===
On AArch64 host, a rootfs path is mandatory, while it is optional on x86-64 host

* Core: JIT
* RootFS
* Silent Logging (Useful to get FEX information spam out of the way)
* Block Size (500 is a good default. Optimization passes may break things though)

More Details at: Follow the instructions [[Development:Configuring_FEX|here]]

== RootFS generation ==
AArch64 hosts require a rootfs for running applications. FEX provides various x86 images that can be installed through [[Development:Setting_up_RootFS#Quick_Setup_with_FEXRootFSFetcher|FEXRootFSFetcher]]

For more detailed instructions see [[Development:Setting_up_RootFS]]

== Setting up library forwarding ==

Follow the instructions [[Development:Setting_up_Library_Forwarding|here]].

== Binaries ==
* FEXLoader
** This is useful for development purposes but is generally recommended against using.
** Allows passing arguments to FEX for configuration purposes, which can break under some execve situations
* FEXInterpreter
** This is a hardlink to FEXLoader which refuses to treat arguments as configuration options
** Is recommended to use over FEXLoader; Using FEXConfig to set configuration
** Requires being a hardlink or executable copy, execve logic sees through a softlink
** Is the executable used for binfmt_misc for interpreters
* FEXConfig
** Used for configuring emulation from a Qt-based GUI
* FEXBash
** Can be useful for starting a bash instance that is running under emulation
** It's not a chroot! Don't use sudo inside of this environment!
* FEXMountDaemon
** This is a mount daemon that will run in the background when you are using a squashfs based rootfs
** Usually one one instance open at any given moment
** It usually closes when the FEXInterpreter exits, but not always. See the problems section in setting up a rootfs
*** Has a ten second timeout window in case a another instance of FEX opens and uses it.

=== Debug binaries ===
* TestHarnessRunner
** Used for running FEX's ASM unit tests
** Runs some binary in a host x86_64 host environment or FEX and checks results
** Useful for quickly creating ASM based unit tests
* IRLoader
** Used for running FEX's IR based unit tests
** Only runs through FEX and checks the results
** Minimal number of tests
* UnitTestGenerator
** This is supposed to be an x86-64 instruction generator
** Has an understanding of x86-64 instruction encoding format
** Generates hundreds of thousands of instructions
** Used to be used with a lockstep runner to ensure correct behaviour
** Largely unsupported, avoiding undefined behaviour is difficult and FEX doesn't need to match undefined behaviour
** Lockstep hostrunner wasn't ever made to be bug free, hard to support
** Doesn't ensure correct behaviour on AArch64 device

Development:Setting up FEX

2025-07-14T10:08:57Z

Neobrain: /* Installing Thunks */

[[Category:Development]]
== Build dependencies ==
=== Debian/Ubuntu dependencies ===
* git
* cmake
* ninja-build
* pkgconf
* ccache (Recommended if you're rebuilding constantly)
* clang
* llvm
* lld (For ThinLTO)
* binfmt-support (For AArch64 binfmt_misc support)
* libssl-dev
* python-setuptools (For build scripts)
* g++-x86-64-linux-gnu (For building thunks)
* libgcc-XX-dev-i386-cross (For building thunks, version should match your g++-x86-64-linux-gnu package)
* libgcc-XX-dev-amd64-cross (For building thunks, version should match your g++-x86-64-linux-gnu package)
* nasm (only if building tests)
* python3-clang (for struct verifier)
* libstdc++-12-dev-i386-cross (for struct verifier)
* libstdc++-12-dev-amd64-cross (for struct verifier)
* libstdc++-12-dev-arm64-cross (for struct verifier)
* squashfs-tools (For squashfs rootfs support)
* squashfuse (For mounting the squashfs)
* libc-bin or glibc-tools (For unittests, needs catchsegv from this)
* libc6-dev-i386-amd64-cross (For FEXLinuxTests)
* lib32stdc++-XX-dev-amd64-cross (For FEXLinuxTests, version should match your g++-x86-64-linux-gnu package)
* qtdeclarative5-dev (For GUI)
* qml-module-qtquick-controls (For GUI)
* qml-module-qtquick-controls2 (For GUI)
* qml-module-qtquick-dialogs (For GUI)

=== Gentoo dependencies ===
* dev-vcs/git
* cmake
* ninja
* ccache
* clang
* lld
* nasm

'''Use gubbins'''
* >=media-libs/libglvnd-1.3.4 X

=== Fedora dependencies ===
* git
* cmake
* ninja-build
* pkg-config
* ccache
* clang
* lld
* llvm
* llvm-devel
* openssl-devel
* nasm
* python3-clang
* python3-setuptools
* squashfs-tools
* squashfuse
* erofs-fuse
* erofs-utils
* qt5-qtdeclarative-devel
* qt5-qtquickcontrols
* qt5-qtquickcontrols2

== Build Configuration ==
* Ensure release mode is enabled
* Disable test building
* Enable LTO in a release build
* Enable LLD to make LTO builds faster

git clone --recurse-submodules https://github.com/FEX-Emu/FEX.git
cd FEX
mkdir Build
cd Build
CC=clang CXX=clang++ cmake -DCMAKE_INSTALL_PREFIX=/usr -DCMAKE_BUILD_TYPE=Release -DUSE_LINKER=lld -DENABLE_LTO=True -DBUILD_TESTS=False -DENABLE_ASSERTIONS=False -G Ninja ..
ninja

== Installation ==
sudo ninja install -j4

You can install a binfmt_misc handler for both 32bit and 64bit x86 execution directly from the environment.
If you already have box86's 32bit binfmt_misc handler installed then make sure to uninstall their's first.
Make sure to have run install prior to this, otherwise binfmt_misc will install an old handler even if the executable has been updated.

sudo ninja binfmt_misc_32
sudo ninja binfmt_misc_64

=== Quirks ===

==== binfmt_misc problems ====
* Double check that binfmt_misc has worked
** `ls /usr/share/binfmts/FEX*` should return files
** `ls /var/lib/binfmts/FEX*` should return files
** `/proc/sys/fs/binfmt_misc/status` should exist
*** If it doesn't exist then make sure your kernel has `CONFIG_BINFMT_MISC=y`
*** If it still doesn't exist then make sure systemd isn't disabling it `systemctl enable binfmt-support`

==== DESTDIR quirks ====
* When using DESTDIR, make sure the path is absolute instead of relative
** FEX's thunks use subprojects because it needs to cross-compile
** If DESTDIR is relative then the install paths of the subprojects go to the wrong location
** Using $(pwd)/install works well

== Runtime Configuration ==
FEXConfig is the application to use for runtime configuration setup.

CTLR+S saves the configuration.

=== Things to ensure to set ===
On AArch64 host, a rootfs path is mandatory, while it is optional on x86-64 host

* Core: JIT
* RootFS
* Silent Logging (Useful to get FEX information spam out of the way)
* Block Size (500 is a good default. Optimization passes may break things though)

More Details at: Follow the instructions [[Development:Configuring_FEX|here]]

== RootFS generation ==
AArch64 hosts require a rootfs for running applications. FEX provides various x86 images that can be installed through [[Development:Setting_up_RootFS#Quick_Setup_with_FEXRootFSFetcher|FEXRootFSFetcher]]

For more detailed instructions see [[Development:Setting_up_RootFS]]

== Setting up library forwarding ==

Follow the instructions [[Development:Setting_up_Library_Forwarding|here]].

== Binaries ==
* FEXLoader
** This is useful for development purposes but is generally recommended against using.
** Allows passing arguments to FEX for configuration purposes, which can break under some execve situations
* FEXInterpreter
** This is a hardlink to FEXLoader which refuses to treat arguments as configuration options
** Is recommended to use over FEXLoader; Using FEXConfig to set configuration
** Requires being a hardlink or executable copy, execve logic sees through a softlink
** Is the executable used for binfmt_misc for interpreters
* FEXConfig
** Used for setting configuration options from a GUI
** Written using Dear ImGui
** Can load/save global configuration and application configuration files
** Application configuration files are compared against the executable filename
* FEXBash
** Can be useful for starting a bash instance that is running under emulation
** It's not a chroot! Don't use sudo inside of this environment!
* FEXMountDaemon
** This is a mount daemon that will run in the background when you are using a squashfs based rootfs
** Usually one one instance open at any given moment
** It usually closes when the FEXInterpreter exits, but not always. See the problems section in setting up a rootfs
*** Has a ten second timeout window in case a another instance of FEX opens and uses it.

=== Debug binaries ===
* TestHarnessRunner
** Used for running FEX's ASM unit tests
** Runs some binary in a host x86_64 host environment or FEX and checks results
** Useful for quickly creating ASM based unit tests
* IRLoader
** Used for running FEX's IR based unit tests
** Only runs through FEX and checks the results
** Minimal number of tests
* UnitTestGenerator
** This is supposed to be an x86-64 instruction generator
** Has an understanding of x86-64 instruction encoding format
** Generates hundreds of thousands of instructions
** Used to be used with a lockstep runner to ensure correct behaviour
** Largely unsupported, avoiding undefined behaviour is difficult and FEX doesn't need to match undefined behaviour
** Lockstep hostrunner wasn't ever made to be bug free, hard to support
** Doesn't ensure correct behaviour on AArch64 device

Main Page

2025-07-14T10:05:46Z

Neobrain: /* Development Topics */

Welcome to the FEX emulator wiki website!

== Account Creation ==
Due to the typical problem of mediawiki sites getting hit by spam bots. Account creation is moderated.

If you create a new account and it doesn't pass through the moderation queue quickly then make sure to ask on IRC or Discord and someone should be able to help.

== Game Categories of interest ==

[[:Category:Game_Type_Application|Games list]]

[[:Category:Vulkan_Render_API|Games that support Vulkan]]
* [[:Category:Vulkan_Thunk_Capable|Vulkan Thunk Capable]]

[[:Category:OpenGL_Render_API|Games that support OpenGL]]
* [[:Category:OpenGL_Thunk_Capable|OpenGL Thunk Capable]]

[[:Category:X86-32|32-bit x86 Games]]

[[:Category:X86-64|64-bit x86_64 Games]]

== Development Topics ==
[[Development:Setting_up_FEX]]

[[Development:Setting_up_RootFS]]

[[Development:Setting_up_Library_Forwarding]]

[[Development:Debugging_FEX_with_Signals]]

[[Development:Debugging_Crash]]

[[Development:Renderdoc]]

[[Development:gfxreconstruct]]

Development:Setting up Library Forwarding

2025-07-14T10:05:07Z

Neobrain:

Library forwarding is a FEX feature that allows redirecting calls to common libraries OpenGL or Vulkan from their emulated x86 builds to their native ARM64 counterparts installed on the host system. This saves emulation overhead and dramatically improves performance for some applications.

= Build instructions =

Install the dependencies below and enable the <code>BUILD_THUNKS</code> CMake option:
* <code>cmake -DBUILD_THUNKS=True .</code>

If your distribution is not listed below, there is a generic fallback that requires Nix or a [https://lix.systems/install/#on-any-other-linuxmacos-system compatible alternative].

=== Ubuntu dependencies ===
* g++-x86-64-linux-gnu (For building thunks)
* pkg-config
* libclang-dev
* libssl-dev
* Development packages for thunked libraries:
** libdrm-dev
** libxcb-present-dev
** libxcb-dri2-0-dev
** libxcb-dri3-dev
** libxcb-glx0-dev
** libxcb-shm0-dev
** libxshmfence-dev

=== Fedora dependencies ===
* libXext-devel
* libXfixes-devel
* libXrender-devel
* libXrandr-devel
* alsa-lib-devel
* libxcb-devel
* libxshmfence-devel
* libdrm-devel
* xorg-x11-server-devel

=== Generic Nix fallback ===

With Nix installed, run <code>../Data/nix/cmake_enable_libfwd.sh</code> from your build folder. This will set up a cross-compiler, install the required dependencies, and enable the required CMake option.

= Configuration =

FEXConfig can be used to enable/disable individual libraries.

=== Per-app overrides ===
* Filename in '''$HOME/.fex-emu/AppConfig/Balatro.exe.json'''
* Change the filename as necessary for your application

{
"ThunksDB": {
"GL": 1,
"Vulkan": 1
}
}

= Advanced configuration =
=== FEX environment variables and command-line arguments ===
* FEX_THUNKHOSTLIBS, -t <Thunk Host Libs Path>, --thunkhostlibs=<Thunk Host Libs path>, JSON: '''ThunkHostLibs:<Path>'''
** Folder to find the host-side thunking libraries.
** FEX by default installs these files to: '''/usr/lib/fex-emu/HostThunks/'''
* FEX_THUNKGUESTLIBS, -j <Thunk Host Libs Path>, --thunkguestlibs=<Thunk Host Libs path>, JSON: '''ThunkGuestLibs:<Path>'''
** Folder to find the guest-side thunking libraries.
** FEX by default installs these files to: '''/usr/share/fex-emu/GuestThunks/'''
* FEX_THUNKCONFIG, -k <Thunk Config json path>, --thunkconfig=<Thunk Config json path>, JSON: '''ThunkConfig:<Path>'''
** A json file specifying where to overlay the thunks.
** If the filename isn't a path then FEX will search for the file in '''$HOME/.fex-emu/ThunkConfigs/'''

===Overriding paths for library wrappers ===
You can can set the three configuration options to custom paths

* The user can likely set '''ThunkHostLibs''' and '''ThunkGuestLibs''' to the global install path
** Unless these are set in the '''ThunkConfig''' file, these will be unused
** Overriding these to another location is useful for application specific hacks if necessary

=== ThunksDB.json schema ===
FEX provides a ThunksDB.json file which allows the user to describe a thunk mapping. '''$prefix/share/fex-emu/ThunksDB.json'''. Normally this does not need to be modified, but an overlay to the default database can be provided at '''$HOME/.fex-emu/ThunksDB.json'''.

Example:
{
"DB": {
"<User Defined Name>": {
"Library": "<Guest Library Name>.so",
"Depends": [
"<Other User Defined Name>"
],
"Overlay": [
"<System Library Path to library>"
]
}
}
}

[[Category:Development]]

Development:Setting up Thunks

2025-07-14T09:42:46Z

Neobrain: Neobrain moved page Development:Setting up Thunks to Development:Setting up Library Forwarding

#REDIRECT [[Development:Setting up Library Forwarding]]

Development:Setting up Library Forwarding

2025-07-14T09:42:46Z

Neobrain: Neobrain moved page Development:Setting up Thunks to Development:Setting up Library Forwarding

Development:ARM64EC

2025-07-14T09:41:50Z

Neobrain:

[[Category:Development]]
=== Building ===
These instructions expect you to be building on an ARM64 host.

==== Toolchain ====
Scripts and instructions to build an ARM64EC toolchain can be found [https://github.com/bylaws/llvm-mingw here], with prebuilt binaries for ARM64/x86 hosts on the [https://github.com/bylaws/llvm-mingw/releases/tag/20240929 releases page]. Once built/downloaded, add it to your environment with
export PATH="<path to toolchain>/bin:$PATH"
note the toolchain must come before all other path entries as it needs to override the host ar binary.
* Make sure to grab the tar.gz aarch64 files when compiling on Linux. The zip files are for a Windows toolchain which doesn't work for these instructions.

==== Wine ====
[https://gitlab.winehq.org/wine/wine Upstream] wine has various things missing for full FEX/ARM64EC support, while it can be used, [https://github.com/bylaws/wine/tree/upstream-arm64ec my fork] will give the best results for now. Make sure to install the deps listed [https://salsa.debian.org/wine-team/wine/-/blob/debian/9.0_repack-4/debian/control.in?ref_type=tags#L13 here] before building with the ARM64EC toolchain.

./configure --enable-archs=arm64ec,aarch64,i386 --prefix=/usr --with-mingw=clang --disable-tests
make -j$(nproc)
sudo --preserve-env=PATH make install -j$(nproc)

==== FEX ====
FEX must be built twice, once for the x86 (WOW64) emulation module and once for x86_64 emulation module (ARM64EC)

FEX's official [https://launchpad.net/~fex-emu/+archive/ubuntu/fex Ubuntu PPA] now provides a fex-emu-wine package of the latest release. This package can be downloaded and the DLL files can be extracted if compiling isn't desired.

===== Build directions =====
mkdir build-arm64ec
cd build-arm64ec
cmake -GNinja -DCMAKE_BUILD_TYPE=Release -DCMAKE_TOOLCHAIN_FILE=../Data/CMake/toolchain_mingw.cmake -DCMAKE_INSTALL_LIBDIR=/usr/lib/wine/aarch64-windows -DENABLE_LTO=False -DMINGW_TRIPLE=arm64ec-w64-mingw32 -DBUILD_TESTS=False -DENABLE_JEMALLOC_GLIBC_ALLOC=False -DCMAKE_INSTALL_PREFIX=/usr ..
ninja
sudo ninja install
cd ..
mkdir build-wow64
cd build-wow64
cmake -GNinja -DCMAKE_BUILD_TYPE=Release -DCMAKE_TOOLCHAIN_FILE=../Data/CMake/toolchain_mingw.cmake -DCMAKE_INSTALL_LIBDIR=/usr/lib/wine/aarch64-windows -DENABLE_LTO=False -DMINGW_TRIPLE=aarch64-w64-mingw32 -DBUILD_TESTS=False -DENABLE_JEMALLOC_GLIBC_ALLOC=False -DCMAKE_INSTALL_PREFIX=/usr ..
ninja
sudo ninja install
cd ..

==== dxvk/vkd3d-proton ====

Both can be built and installed as instructed on their respective github pages, except that a path to the following cross file should be passed to meson with --cross-file:
[binaries]
ar = 'arm64ec-w64-mingw32-ar'
c = 'arm64ec-w64-mingw32-gcc'
cpp = 'arm64ec-w64-mingw32-g++'
ld = 'arm64ec-w64-mingw32-ld'
windres = 'arm64ec-w64-mingw32-windres'
strip = 'strip'
widl = 'arm64ec-w64-mingw32-widl'
pkgconfig = 'aarch64-linux-gnu-pkg-config'
[host_machine]
system = 'windows'
cpu_family = 'aarch64'
cpu = 'aarch64'
endian = 'little'

=== Windows ===
ARM64EC FEX is usable as a drop-in replacement for xtajit64 on native WOA, build FEX with -DUSE_PDB_DEBUGINFO=True and copy the libarm64ecfex.{dll,pdb} binaries into C:/Windows/System32. To actually use FEX create an empty registry key at HKLM\\Software\\Microsoft\\Wow64\\amd64 and set its value to libarm64ecfex.dll, all subsequently launched programs will then use FEX. Logs/configuration is stored in %localappdata%.

=== Info ===
Unlike Linux FEX, the Windows FEX modules purely handle CPU emulation, with syscall translation being done on either the wine side (WOW64) or ABI (ARM64EC). This is similarly the case for wine's unixcalls and as a result native ARM64 variants can be used for all .so libraries (Vulkan, wine, etc.) dropping the need for an x86 rootfs and FEX-side thunking.

WOW64 (Windows 32-bit On Windows 64-bit) is the mechanism to run 32-bit applications on 64-bit platforms: All system libraries are built as x86 and emulated. Any 32-bit syscalls they use are adapted on-the-fly to 64-bit equivalents by injecting thunks.

ARM64EC (Emulation Compatible) on the other hand deals with 64-bit applications and avoids emulation overhead by using a dedicated set of system libraries compiled for ARM64 with the custom ARM64EC ABI. This ABI is designed to be callable from recompiled x86_64 code in order to allow a mix-and-match of native and emulated code. To avoid needing both a set of ARM64EC and ARM64 system libraries, ARM64X is introduced: It uses linker relocations to combine ARM64EC code and regular ARM64 code in a single binary file, which can be used both by standard Windows-on-Arm applications and emulated x86_64 ones.

The following pages serve as a good reference:

https://learn.microsoft.com/en-us/windows/arm/arm64ec

https://learn.microsoft.com/en-us/windows/arm/arm64ec-abi

https://gitlab.winehq.org/wine/wine/-/wikis/ARM64EC-Toolchain

Of note is the fact that FEX needs to run in a very reduced Windows environment - due to how early it is loaded it can only depend on ntdll and not say the system CRT. This is worked around by implementing/stubbing the subset of the CRT/Windows APIs necessary for FEX to run/link purely ontop of ntdll (see Common/Windows/{WinAPI,CRT}).

=== Building problems ===
==== `bytes of instructions in range, but .seh directives corresponding to` ====
This is seemingly caused by the `-mcpu` tune option, pass to cmake `-DTUNE_CPU=none`

Development:ARM64EC

2025-06-27T08:19:10Z

Neobrain:

[[Category:Development]]
=== Building ===
These instructions expect you to be building on an ARM64 host.

==== Toolchain ====
Scripts and instructions to build an ARM64EC toolchain can be found [https://github.com/bylaws/llvm-mingw here], with prebuilt binaries for ARM64/x86 hosts on the [https://github.com/bylaws/llvm-mingw/releases/tag/20240929 releases page]. Once built/downloaded, add it to your environment with
export PATH="<path to toolchain>/bin:$PATH"
note the toolchain must come before all other path entries as it needs to override the host ar binary.
* Make sure to grab the tar.gz aarch64 files when compiling on Linux. The zip files are for a Windows toolchain which doesn't work for these instructions.

==== Wine ====
[https://gitlab.winehq.org/wine/wine Upstream] wine has various things missing for full FEX/ARM64EC support, while it can be used, [https://github.com/bylaws/wine/tree/upstream-arm64ec my fork] will give the best results for now. Make sure to install the deps listed [https://salsa.debian.org/wine-team/wine/-/blob/debian/9.0_repack-4/debian/control.in?ref_type=tags#L13 here] before building with the ARM64EC toolchain.

./configure --enable-archs=arm64ec,aarch64,i386 --prefix=/usr --with-mingw=clang --disable-tests
make -j$(nproc)
sudo --preserve-env=PATH make install -j$(nproc)

==== FEX ====
FEX must be built twice, once for the x86 (WOW64) emulation module and once for x86_64 emulation module (ARM64EC)

FEX's official [https://launchpad.net/~fex-emu/+archive/ubuntu/fex Ubuntu PPA] now provides a fex-emu-wine package of the latest release. This package can be downloaded and the DLL files can be extracted if compiling isn't desired.

===== Build directions =====
mkdir build-arm64ec
cd build-arm64ec
cmake -GNinja -DCMAKE_BUILD_TYPE=Release -DCMAKE_TOOLCHAIN_FILE=../toolchain_mingw.cmake -DCMAKE_INSTALL_LIBDIR=/usr/lib/wine/aarch64-windows -DENABLE_LTO=False -DMINGW_TRIPLE=arm64ec-w64-mingw32 -DBUILD_TESTS=False -DENABLE_JEMALLOC_GLIBC_ALLOC=False -DCMAKE_INSTALL_PREFIX=/usr ..
ninja
sudo ninja install
cd ..
mkdir build-wow64
cd build-wow64
cmake -GNinja -DCMAKE_BUILD_TYPE=Release -DCMAKE_TOOLCHAIN_FILE=../toolchain_mingw.cmake -DCMAKE_INSTALL_LIBDIR=/usr/lib/wine/aarch64-windows -DENABLE_LTO=False -DMINGW_TRIPLE=aarch64-w64-mingw32 -DBUILD_TESTS=False -DENABLE_JEMALLOC_GLIBC_ALLOC=False -DCMAKE_INSTALL_PREFIX=/usr ..
ninja
sudo ninja install
cd ..

==== dxvk/vkd3d-proton ====

Both can be built and installed as instructed on their respective github pages, except that a path to the following cross file should be passed to meson with --cross-file:
[binaries]
ar = 'arm64ec-w64-mingw32-ar'
c = 'arm64ec-w64-mingw32-gcc'
cpp = 'arm64ec-w64-mingw32-g++'
ld = 'arm64ec-w64-mingw32-ld'
windres = 'arm64ec-w64-mingw32-windres'
strip = 'strip'
widl = 'arm64ec-w64-mingw32-widl'
pkgconfig = 'aarch64-linux-gnu-pkg-config'
[host_machine]
system = 'windows'
cpu_family = 'aarch64'
cpu = 'aarch64'
endian = 'little'

=== Windows ===
ARM64EC FEX is usable as a drop-in replacement for xtajit64 on native WOA, build FEX with -DUSE_PDB_DEBUGINFO=True and copy the libarm64ecfex.{dll,pdb} binaries into C:/Windows/System32. To actually use FEX create an empty registry key at HKLM\\Software\\Microsoft\\Wow64\\amd64 and set its value to libarm64ecfex.dll, all subsequently launched programs will then use FEX. Logs/configuration is stored in %localappdata%.

=== Info ===
Unlike Linux FEX, the Windows FEX modules purely handle CPU emulation, with syscall translation being done on either the wine side (WOW64) or ABI (ARM64EC). This is similarly the case for wine's unixcalls and as a result native ARM64 variants can be used for all .so libraries (Vulkan, wine, etc.) dropping the need for an x86 rootfs and FEX-side thunking.

WOW64 is the mechanism to run 32-bit applications on 64-bit platforms: Almost all Windows system libraries are emulated from their original x86 builds here. Only {unix,sys}calls are adapted on-the-fly by injecting thunks. For developers, Microsoft distributes native ARM64 builds of the system libraries as well, but that isn't a concern for FEX.

ARM64EC on the other hand deals with 64-bit applications and avoids emulation overhead by using a dedicated set of system libraries compiled for ARM64 with a custom ABI. This ARM64EC ABI is designed to be callable from recompiled x86_64 code, so that application binaries can use the non-emulated system libraries directly. To ease application porting, Microsoft extended the PE format ("ARM64X") to combine ARM64EC code with regular ARM64 code in a single binary file (leveraging linker relocations for function prologues/epilogues to avoid duplicating any code). This extension allows a single set of system libraries to be used both for native ARM64 applications and for emulated x86 ones, without needing to distribute any of the original x86 system libraries at all.

The following pages serve as a good reference:

https://learn.microsoft.com/en-us/windows/arm/arm64ec

https://learn.microsoft.com/en-us/windows/arm/arm64ec-abi

https://gitlab.winehq.org/wine/wine/-/wikis/ARM64EC-Toolchain

Of note is the fact that FEX needs to run in a very reduced Windows environment - due to how early it is loaded it can only depend on ntdll and not say the system CRT. This is worked around by implementing/stubbing the subset of the CRT/Windows APIs necessary for FEX to run/link purely ontop of ntdll (see Common/Windows/{WinAPI,CRT}).

Development:Configuring FEX

2025-02-16T17:02:43Z

Neobrain:

== Configuration via GUI ==

The FEXConfig tool provides a graphical user interface to edit FEX configuration. On first use, it will load default settings and save them to .fex-emu/Config.json.

=== Setup Rootfs ===
The ''General'' tab in FEXConfig contains the rootfs configuration.
* Named RootFS folders option shows RootFS installed in '''$HOME/.fex-emu/RootFS/'''
** sqsh files and folders are selectable here
* Selecting an option in the Named RootFS option will set the name in RootFS
** If your x86 RootFS is in an absolute folder elsewhere, you can place the absolute path in the RootFS option

== Manual Configuration ==
The FEX main configuration file lives at $HOME/.fex-emu/Config.json
* or $XDG_CONFIG_HOME/.fex-emu/Config.json if you use XDG).

This is a JSON file that holds all of our settings.
The basic structure is as follows:
{
"Config": {
"Threads": "0",
"MaxInst": "500",
"RootFS": "Ubuntu_21_10"
}
}

There are a fair number of configuration options to tinker with. You can see a full suite of configuration option in the FEX man pages.
* man FEX - To see the options
** Under the ENVIRONMENT section there is each option's ENVIRONMENT override
** Additionally each option has a "JSON key" name that shows the Key inside of the Config.json file
** Every option is a string inside of the config and will be reinterpreted to their programmatic type at load

[[Category:Development]]

Development:Setting up Library Forwarding

2025-02-16T16:57:14Z

Neobrain:

Development:ARM64EC

2025-02-02T16:39:06Z

Neobrain:

[[Category:Development]]
=== Building ===
These instructions expect you to be building on an ARM64 host.

==== Toolchain ====
Scripts and instructions to build an ARM64EC toolchain can be found [https://github.com/bylaws/llvm-mingw here], with prebuilt binaries for ARM64/x86 hosts on the [https://github.com/bylaws/llvm-mingw/releases/tag/20240929 releases page]. Once built/downloaded, add it to your environment with
export PATH="<path to toolchain>/bin:$PATH"
note the toolchain must come before all other path entries as it needs to override the host ar binary.
* Make sure to grab the tar.gz aarch64 files when compiling on Linux. The zip files are for a Windows toolchain which doesn't work for these instructions.

==== Wine ====
[https://gitlab.winehq.org/wine/wine Upstream] wine has various things missing for full FEX/ARM64EC support, while it can be used, [https://github.com/bylaws/wine/tree/upstream-arm64ec my fork] will give the best results for now. Make sure to install the deps listed [https://salsa.debian.org/wine-team/wine/-/blob/debian/9.0_repack-4/debian/control.in?ref_type=tags#L13 here] before building with the ARM64EC toolchain.

./configure --enable-archs=arm64ec,aarch64,i386 --with-mingw=clang --disable-tests
make -j$(nproc)
sudo --preserve-env=PATH make install -j$(nproc)

If using FEX from the ppa then make sure to append '''--prefix=/usr''' to the wine configure commands. Both FEX and Wine's prefix need to match if they are to find each other!

==== FEX ====
FEX must be built twice, once for the x86 (WOW64) emulation module and once for x86_64 emulation module (ARM64EC)

FEX's official [https://launchpad.net/~fex-emu/+archive/ubuntu/fex Ubuntu PPA] now provides a fex-emu-wine package of the latest release. This package can be downloaded and the DLL files can be extracted if compiling isn't desired.

===== Build directions =====
mkdir build-arm64ec
cd build-arm64ec
cmake -GNinja -DCMAKE_BUILD_TYPE=<desire build type> -DCMAKE_TOOLCHAIN_FILE=/data/Proton/FEX/toolchain_mingw.cmake -DCMAKE_INSTALL_LIBDIR=/usr/local/lib/wine/aarch64-windows -DENABLE_LTO=False -DMINGW_TRIPLE=arm64ec-w64-mingw32 -DBUILD_TESTS=False -DENABLE_JEMALLOC_GLIBC_ALLOC=False <path to fex>
ninja
sudo ninja install
cd ..
mkdir build-wow64
cd build-wow64
cmake -GNinja -DCMAKE_BUILD_TYPE=<desire build type> -DCMAKE_TOOLCHAIN_FILE=/data/Proton/FEX/toolchain_mingw.cmake -DCMAKE_INSTALL_LIBDIR=/usr/local/lib/wine/aarch64-windows -DENABLE_LTO=False -DMINGW_TRIPLE=aarch64-w64-mingw32 -DBUILD_TESTS=False -DENABLE_JEMALLOC_GLIBC_ALLOC=False <path to fex>
ninja
sudo ninja install
cd ..

==== dxvk/vkd3d-proton ====

Both can be built and installed as instructed on their respective github pages, except that a path to the following cross file should be passed to meson with --cross-file:
[binaries]
ar = 'arm64ec-w64-mingw32-ar'
c = 'arm64ec-w64-mingw32-gcc'
cpp = 'arm64ec-w64-mingw32-g++'
ld = 'arm64ec-w64-mingw32-ld'
windres = 'arm64ec-w64-mingw32-windres'
strip = 'strip'
widl = 'arm64ec-w64-mingw32-widl'
pkgconfig = 'aarch64-linux-gnu-pkg-config'
[host_machine]
system = 'windows'
cpu_family = 'aarch64'
cpu = 'aarch64'
endian = 'little'

=== Windows ===
ARM64EC FEX is usable as a drop-in replacement for xtajit64 on native WOA, build FEX with -DUSE_PDB_DEBUGINFO=True and copy the libarm64ecfex.{dll,pdb} binaries into C:/Windows/System32. To actually use FEX create an empty registry key at HKLM\\Software\\Microsoft\\Wow64\\amd64 and set its value to libarm64ecfex.dll, all subsequently launched programs will then use FEX. Logs/configuration is stored in %localappdata%.

=== Info ===
Unlike Linux FEX, the Windows FEX modules purely handle CPU emulation, with syscall translation being done on either the wine side (WOW64) or ABI (ARM64EC). This is similarly the case for wine's unixcalls and as a result native ARM64 variants can be used for all .so libraries (Vulkan, wine, etc.) dropping the need for an x86 rootfs and FEX-side thunking.

WOW64 is fairly simple, every DLL is compiled with both x86 and ARM64 variants and in an x86 process only the x86 variants are loaded (with ARM64 FEX and ntdll) the only thunking done is for {unix,sys}calls so almost all Windows code is emulated.

ARM64EC on the otherhand avoids duplicating DLLs at all, instead choosing to introduce a new ARM64 ABI allowing for transparent compiler-aided interop with any existing x86_64 code. The following pages serve as a good introduction:

https://learn.microsoft.com/en-us/windows/arm/arm64ec

https://learn.microsoft.com/en-us/windows/arm/arm64ec-abi

https://gitlab.winehq.org/wine/wine/-/wikis/ARM64EC-Toolchain

Of note is the fact that FEX needs to run in a very reduced Windows environment - due to how early it is loaded it can only depend on ntdll and not say the system CRT. This is worked around by implementing/stubbing the subset of the CRT/Windows APIs necessary for FEX to run/link purely ontop of ntdll (see Common/Windows/{WinAPI,CRT}).

Development:Setting up FEX

2022-09-15T08:17:34Z

Neobrain: Add standard library dependencies for cross-compiling

[[Category:Development]]
== Build dependencies ==
* git
* cmake
* ninja-build
* pkg-config
* ccache (Recommended if you're rebuilding constantly)
* clang
* lld (For ThinLTO)
* binfmt-support (For AArch64 binfmt_misc support)
* libsdl2-dev (For GUI)
* libepoxy-dev (For GUI)
* g++-x86-64-linux-gnu (For building thunks)
* nasm (only if building tests)
* python3-clang (for struct verifier)
* libstdc++-10-dev-i386-cross (for struct verifier)
* libstdc++-10-dev-amd64-cross (for struct verifier)
* libstdc++-10-dev-arm64-cross (for struct verifier)
* squashfs-tools (For squashfs rootfs support)
* squashfuse (For mounting the squashfs)
* libc-bin or glibc-tools (For unittests, needs catchsegv from this)
* gcc-multilib (For FEXLinuxTests)
* g++-multilib (For FEXLinuxTests)
* libc6-dev-i386-amd64-cross (For FEXLinuxTests)
* lib32stdc++-XX-dev-amd64-cross (For FEXLinuxTests, version should match your g++-x86-64-linux-gnu package)

=== Gentoo dependencies ===
* dev-vcs/git
* cmake
* ninja
* ccache
* clang
* lld
* libsdl2
* libepoxy
* nasm

'''Use gubbins'''
* >=media-libs/libglvnd-1.3.4 X
== Build Configuration ==
* Ensure release mode is enabled
* Disable test building
* Enable LTO in a release build
* Enable LLD to make LTO builds faster

git clone --recurse-submodules https://github.com/FEX-Emu/FEX.git
cd FEX
mkdir Build
cd Build
CC=clang CXX=clang++ cmake -DCMAKE_INSTALL_PREFIX=/usr -DCMAKE_BUILD_TYPE=Release -DENABLE_LTO=True -DENABLE_LLD=True -DBUILD_TESTS=False -DENABLE_ASSERTIONS=False -G Ninja ..
ninja

== Installation ==
sudo ninja install

=== On AArch64 hosts ===
You can install a binfmt_misc handler for both 32bit and 64bit x86 execution directly from the environment.
If you already have box86's 32bit binfmt_misc handler installed then I don't recommend installing FEX's until it is useful.
Make sure to have run install prior to this, otherwise binfmt_misc will install an old handler even if the executable has been updated.

sudo ninja binfmt_misc_32
sudo ninja binfmt_misc_64

'''binfmt_misc problems'''
* Double check that binfmt_misc has worked
** `ls /usr/share/binfmts/FEX*` should return files
** `ls /var/lib/binfmts/FEX*` should return files
** `/proc/sys/fs/binfmt_misc/status` should exist
*** If it doesn't exist then make sure your kernel has `CONFIG_BINFMT_MISC=y`
*** If it still doesn't exist then make sure systemd isn't disabling it `systemctl enable binfmt-support`

== Runtime Configuration ==
FEXConfig is the application to use for runtime configuration setup.

CTRL+SHIFT+O will open default "global" configuration.

CTLR+S saves it.

=== Things to ensure to set ===
On AArch64 host, a rootfs path is mandatory, while it is optional on x86-64 host

* Core: JIT
* RootFS
* Silent Logging (Useful to get FEX information spam out of the way)
* Block Size (500 is a good default. Optimization passes may break things though)
* Multiblock: enabled (Can cause more stuttering)

More Details at: Follow the instructions [[Development:Configuring_FEX|here]]
== RootFS generation ==
AArch64 hosts require a rootfs for running applications
Follow the instructions [[Development:Setting_up_RootFS]]

== Installing Thunks ==
Only if you want

Follow the instructions [[Development:Setting_up_Thunks|here]]

Additional details here: https://github.com/FEX-Emu/FEX/blob/main/ThunkLibs/README.md

== Binaries ==
* FEXLoader
** This is useful for development purposes but is generally recommended against using.
** Allows passing arguments to FEX for configuration purposes, which can break under some execve situations
* FEXInterpreter
** This is a hardlink to FEXLoader which refuses to treat arguments as configuration options
** Is recommended to use over FEXLoader; Using FEXConfig to set configuration
** Requires being a hardlink or executable copy, execve logic sees through a softlink
** Is the executable used for binfmt_misc for interpreters
* FEXConfig
** Used for setting configuration options from a GUI
** Written using Dear ImGui
** Can load/save global configuration and application configuration files
** Application configuration files are compared against the executable filename
* FEXBash
** Can be useful for starting a bash instance that is running under emulation
** It's not a chroot! Don't use sudo inside of this environment!
* FEXMountDaemon
** This is a mount daemon that will run in the background when you are using a squashfs based rootfs
** Usually one one instance open at any given moment
** It usually closes when the FEXInterpreter exits, but not always. See the problems section in setting up a rootfs
*** Has a ten second timeout window in case a another instance of FEX opens and uses it.

=== Debug binaries ===
* TestHarnessRunner
** Used for running FEX's ASM unit tests
** Runs some binary in a host x86_64 host environment or FEX and checks results
** Useful for quickly creating ASM based unit tests
* IRLoader
** Used for running FEX's IR based unit tests
** Only runs through FEX and checks the results
** Minimal number of tests
* UnitTestGenerator
** This is supposed to be an x86-64 instruction generator
** Has an understanding of x86-64 instruction encoding format
** Generates hundreds of thousands of instructions
** Used to be used with a lockstep runner to ensure correct behaviour
** Largely unsupported, avoiding undefined behaviour is difficult and FEX doesn't need to match undefined behaviour
** Lockstep hostrunner wasn't ever made to be bug free, hard to support
** Doesn't ensure correct behaviour on AArch64 device

Jupiter Hell

2022-07-09T09:07:53Z

Neobrain: Mention workaround for Vulkan thunks

{{Infobox
|steamappid = 811320
|os = Linux
|engine = U1nique
|arch = x86-64
|cpufeats = Unknown
|compat = Playable
|apptype = Game
|renderapi = Vulkan
}}

== Steps to Run ==
'''No special instructions'''

== Quirks ==

== Problems/Workarounds ==
== Supported Thunks ==
== Unsupported Thunks ==
Vulkan
Ships its own libvulkan loader, resulting in Vulkan thunking not working. As a workaround, removing libvulkan.so.1 from the game folder allows Vulkan thunks to operate properly.

== Testing Results ==




{{testing/start}}
{{testing/entry|revision=FEX-2207|archtitle=x86-64|archhost=AArch64|result=Game runs|tester=Sonicadvance1}}
{{testing/end}}

Development:Setting up Library Forwarding

2022-06-22T14:00:55Z

Neobrain:

= Build instructions =
== Build dependencies ==
* g++-x86-64-linux-gnu (For building thunks)
* pkg-config
* libclang-dev
* libssl-dev
* Development packages for thunked libraries:
** libdrm-dev
** libxcb-present-dev
** libxcb-dri2-0-dev
** libxcb-dri3-dev
** libxcb-glx0-dev
** libxcb-shm0-dev
** libxshmfence-dev
** libvulkan-dev

== Setting CMake to build thunks ==
Pass in to CMake the option to enable thunk building.
* -DBUILD_THUNKS=True

= Configuration =
== FEX thunk options ==
* FEX_THUNKHOSTLIBS, -t <Thunk Host Libs Path>, --thunkhostlibs=<Thunk Host Libs path>, JSON: '''ThunkHostLibs:<Path>'''
** Folder to find the host-side thunking libraries.
** FEX by default installs these files to: '''/usr/lib/fex-emu/HostThunks/'''
* FEX_THUNKGUESTLIBS, -j <Thunk Host Libs Path>, --thunkguestlibs=<Thunk Host Libs path>, JSON: '''ThunkGuestLibs:<Path>'''
** Folder to find the guest-side thunking libraries.
** FEX by default installs these files to: '''/usr/share/fex-emu/GuestThunks/'''
* FEX_THUNKCONFIG, -k <Thunk Config json path>, --thunkconfig=<Thunk Config json path>, JSON: '''ThunkConfig:<Path>'''
** A json file specifying where to overlay the thunks.
** If the filename isn't a path then FEX will search for the file in '''$HOME/.fex-emu/ThunkConfigs/'''

== How to use Thunks ==
Once Thunks are built and installed then you can can set the three configuration options to the correct paths

* The user can likely set '''ThunkHostLibs''' and '''ThunkGuestLibs''' to the global install path
** Unless these are set in the '''ThunkConfig''' file, these will be unused
** Overriding these to another location is useful for application specific hacks if necessary
* Setting the global application profile for '''ThunkConfig''' is useful for testing but not recommended for long term use
** Due to the unstable nature of thunks, it is recommended to use application specific profiles to enable thunks
** This allows the user to test applications first using the global config, then once confirmed it works, generating an application profile
** Recommended to reduce the application profile to the minimum set of options. FEXConfig will save a full config.

== Thunks Database format ==
FEX provides a ThunksDB.json file which allows the user to describe a thunk mapping. '''$prefix/share/fex-emu/ThunksDB.json'''

The user can also provide a '''$HOME/.fex-emu/ThunksDB.json''' which will overlay the system installed path

This allows the user to configure the thunk file mapping once, since it is unlikely to change between applications. The user can also mix and match ThunksDB and regular thunks configuration if a complex configuration is necessary.

* ThunksDB.json example
{
"DB": {
"<User Defined Name>": {
"Library": "<Guest Library Name>.so",
"Depends": [
"<Other User Defined Name>"
],
"Overlay": [
"<System Library Path to library>"
]
}
}
}

Look at the FEX provided library for a real implementation, or just use the provided configuration.

Once the user has a viable ThunksDB provided, now configure FEX with a ThunkConfig filepath with the application thunk configuration.

This thunks configuration file can now enable Thunk definitions using the user defined names.

* ThunkConfig example
{
"ThunksDB": {
"GL": 0,
"xcb": 1,
"xcb-dri2": 1,
"xcb-dri3": 1,
"X11": 0,
"xcb-xfixes": 1,
"xcb-shm": 1,
"xcb-sync": 1,
"xcb-randr": 0,
"xcb-present": 0,
"xcb-glx": 1,
"xshmfence": 1,
"drm": 0,
"Xrender": 0,
"Xext": 0,
"Xfixes": 0,
"Vulkan-lavapipe": 1,
"Vulkan-radeon": 1,
"Vulkan-freedreno": 1,
"Vulkan-intel": 1,
"Vulkan-panfrost": 1,
"Vulkan-virtio": 1,
"asound": 0
}
}

= Further notes =
== Limitations and problems ==
* 32-bit thunks aren't available currently
** Requires kernel patches for ARM64 support
* Thunks aren't fully stable. Expect missing symbols and crashes.

== Legacy ThunkConfig example ==
'''This is legacy path and not recommended.'''

This is just an example for how one could lay out the thunks.
* Save this to a file somewhere and point '''ThunkConfig''' option to it
** '''$HOME/.fex-emu/ThunkConfigs/''' isn't a bad choice for consistency.
{
"thunks": {
"libGL-guest.so": ["/lib/x86_64-linux-gnu/libGL.so.1", "/usr/lib/x86_64-linux-gnu/libGL.so.1"],
"libasound-guest.so": ["/lib/x86_64-linux-gnu/libasound.so.2", "/usr/lib/x86_64-linux-gnu/libasound.so.2"],
"libX11-guest.so": ["/lib/x86_64-linux-gnu/libX11.so.6", "/usr/lib/x86_64-linux-gnu/libX11.so.6"],
"libXrender-guest.so": ["/lib/x86_64-linux-gnu/libXrender.so.1", "/usr/lib/x86_64-linux-gnu/libXrender.so.1"],
"libXext-guest.so": ["/lib/x86_64-linux-gnu/libXext.so.6", "/usr/lib/x86_64-linux-gnu/libXext.so.6"],
"libXfixes-guest.so": ["/lib/x86_64-linux-gnu/libXfixes.so.3", "/usr/lib/x86_64-linux-gnu/libXfixes.so.3"]
}
}
[[Category:Development]]

Development:Configuring FEX

2022-06-22T13:49:19Z

Neobrain: Add development category

== FEXConfig configuration ==
'''To be filled'''
* File-> Load Default Options
** Only do this one time to fill the default Config.json
** File-> Save to save this to .fex-emu/Config.json
* File-> Open Default
** Opens the .fex-emu/Config.json file

=== Setup Rootfs ===
Emulation tab in FEXConfig contains the rootfs configuration.
* Named RootFS folders option shows RootFS installed in '''$HOME/.fex-emu/RootFS/'''
** sqsh files and folders are selectable here
* Selecting an option in the Named RootFS option will set the name in RootFS
** If your x86 RootFS is in an absolute folder elsewhere, you can place the absolute path in the RootFS option

== Manual Configuration ==
The FEX main configuration file lives at $HOME/.fex-emu/Config.json
* or $XDG_CONFIG_HOME/.fex-emu/Config.json if you use XDG).

This is a JSON file that holds all of our settings.
The basic structure is as follows:
{
"Config": {
"Threads": "0",
"MaxInst": "500",
"RootFS": "Ubuntu_21_10"
}
}

There are a fair number of configuration options to tinker with. You can see a full suite of configuration option in the FEX man pages.
* man FEX - To see the options
** Under the ENVIRONMENT section there is each option's ENVIRONMENT override
** Additionally each option has a "JSON key" name that shows the Key inside of the Config.json file
** Every option is a string inside of the config and will be reinterpreted to their programmatic type at load

[[Category:Development]]

Development:StructPackVerifier

2022-06-22T13:46:34Z

Neobrain: Add development category

== Struct packing verifier ==
The struct packing verifier python script allows us to validate struct packings match between aliases and architectures.
This is necessary to ensure auto generated structs match across architectures and handwritten structs match the original architecture's struct.

== Struct annotations ==
The verifier tool looks for structures and unions with specifically crafted annotations to only check the ones we care about.

`__attribute__((annotate("alias-<arch>-<struct name>")))`
* Where arch can be x86_32, x86_64, aarch64, win32, win64
* Struct name is the original structure name that the struct to should alias to
* eg `__attribute__((annotate("alias-x86_32-rusage")))`

`__attribute__((annotate("fex-match")))`
* If a struct should be compared across architectures then we use this to ensure it matches
* FEX will compare that the struct matches between x86-64 and aarch64 for compatibility

== Dependencies ==
* python3
* python3-clang
* clang (required to find default include paths)
* Multilib packages
** 32bit Linux
*** libc6-dev-i386-cross
*** libstdc++6-i386-cross
*** libstdc++-10-dev-i386-cross
*** linux-libc-dev-i386-cross
** 64bit Linux
*** libc6-dev
*** linux-libc-dev
*** libc6-dev-amd64-cross
*** linux-libc-dev-amd64-cross
*** libstdc++6-amd64-cross
*** libstdc++-10-dev-amd64-cross
** AArch64 Linux
*** libc6-dev-arm64-cross
*** linux-libc-dev-arm64-cross
*** libstdc++6-arm64-cross
*** libstdc++-10-dev-arm64-cross
** Mingw for Windows
*** g++-mingw-w64-x86-64
*** g++-mingw-w64-x86-64-win32
*** g++-mingw-w64-x86-64-posix
*** g++-mingw-w64-i686
*** g++-mingw-w64-i686-win32
*** g++-mingw-w64-i686-posix

== Getting a result ==
If the arguments are passed in to the tool correctly then it will return 0 if everything matches, else it returns 1 if something hasn't matched.
You'll then get logging output to compare against

== Bugs ==
The libclang python implementation might not be all encompassing of definitions. So one should be careful to only throw basic struct and union types at the tooling.
Ideally only POD types should be compared with the expectation that structs will be crossing architecture boundaries and need to match.

[[Category:Development]]

Development:Setting up Library Forwarding

2022-06-22T13:42:24Z

Neobrain:

[[Category:Development]]
== Build Dependencies ==
* g++-x86-64-linux-gnu (For building thunks)
* '''Development libraries for various thunks'''
* libdrm-dev
* libxcb-present-dev
* libxcb-dri2-0-dev
* libxcb-dri3-dev
* libxcb-glx0-dev
* libxcb-shm0-dev
* libxshmfence-dev
* libvulkan-dev

== Building vulkan-docs if your host is too old ==
If you're host's libvulkan-dev is too old, which is highly likely then you'll need to install a new set of vulkan headers

* git clone https://github.com/KhronosGroup/Vulkan-Headers.git
* cd Vulkan-Headers
* mkdir Build
* cd Build
* cmake -DCMAKE_INSTALL_PREFIX=/usr ..
* sudo make install

== Setting cmake to build thunks ==
Pass in to cmake the option to enable thunk building.
* -DBUILD_THUNKS=True

== FEX thunk options ==
* FEX_THUNKHOSTLIBS, -t <Thunk Host Libs Path>, --thunkhostlibs=<Thunk Host Libs path>, JSON: '''ThunkHostLibs:<Path>'''
** Folder to find the host-side thunking libraries.
** FEX by default installs these files to: '''/usr/lib/fex-emu/HostThunks/'''
* FEX_THUNKGUESTLIBS, -j <Thunk Host Libs Path>, --thunkguestlibs=<Thunk Host Libs path>, JSON: '''ThunkGuestLibs:<Path>'''
** Folder to find the guest-side thunking libraries.
** FEX by default installs these files to: '''/usr/share/fex-emu/GuestThunks/'''
* FEX_THUNKCONFIG, -k <Thunk Config json path>, --thunkconfig=<Thunk Config json path>, JSON: '''ThunkConfig:<Path>'''
** A json file specifying where to overlay the thunks.
** If the filename isn't a path then FEX will search for the file in '''$HOME/.fex-emu/ThunkConfigs/'''

== How to use Thunks ==
Once Thunks are built and installed then you can can set the three configuration options to the correct paths

* The user can likely set '''ThunkHostLibs''' and '''ThunkGuestLibs''' to the global install path
** Unless these are set in the '''ThunkConfig''' file, these will be unused
** Overriding these to another location is useful for application specific hacks if necessary
* Setting the global application profile for '''ThunkConfig''' is useful for testing but not recommended for long term use
** Due to the unstable nature of thunks, it is recommended to use application specific profiles to enable thunks
** This allows the user to test applications first using the global config, then once confirmed it works, generating an application profile
** Recommended to reduce the application profile to the minimum set of options. FEXConfig will save a full config.

== Thunks Database format ==
FEX provides a ThunksDB.json file which allows the user to describe a thunk mapping. '''$prefix/share/fex-emu/ThunksDB.json'''

The user can also provide a '''$HOME/.fex-emu/ThunksDB.json''' which will overlay the system installed path

This allows the user to configure the thunk file mapping once, since it is unlikely to change between applications. The user can also mix and match ThunksDB and regular thunks configuration if a complex configuration is necessary.

* ThunksDB.json example
{
"DB": {
"<User Defined Name>": {
"Library": "<Guest Library Name>.so",
"Depends": [
"<Other User Defined Name>"
],
"Overlay": [
"<System Library Path to library>"
]
}
}
}

Look at the FEX provided library for a real implementation, or just use the provided configuration.

Once the user has a viable ThunksDB provided, now configure FEX with a ThunkConfig filepath with the application thunk configuration.

This thunks configuration file can now enable Thunk definitions using the user defined names.

* ThunkConfig example
{
"ThunksDB": {
"GL": 0,
"xcb": 1,
"xcb-dri2": 1,
"xcb-dri3": 1,
"X11": 0,
"xcb-xfixes": 1,
"xcb-shm": 1,
"xcb-sync": 1,
"xcb-randr": 0,
"xcb-present": 0,
"xcb-glx": 1,
"xshmfence": 1,
"drm": 0,
"Xrender": 0,
"Xext": 0,
"Xfixes": 0,
"Vulkan-lavapipe": 1,
"Vulkan-radeon": 1,
"Vulkan-freedreno": 1,
"Vulkan-intel": 1,
"Vulkan-panfrost": 1,
"Vulkan-virtio": 1,
"asound": 0
}
}

== Limitations and problems ==
* 32-bit thunks aren't available currently
** Requires kernel patches for ARM64 support
* Thunks aren't fully stable. Expect missing symbols and crashes.

== Legacy ThunkConfig example ==
'''This is legacy path and not recommended.'''

This is just an example for how one could lay out the thunks.
* Save this to a file somewhere and point '''ThunkConfig''' option to it
** '''$HOME/.fex-emu/ThunkConfigs/''' isn't a bad choice for consistency.
{
"thunks": {
"libGL-guest.so": ["/lib/x86_64-linux-gnu/libGL.so.1", "/usr/lib/x86_64-linux-gnu/libGL.so.1"],
"libasound-guest.so": ["/lib/x86_64-linux-gnu/libasound.so.2", "/usr/lib/x86_64-linux-gnu/libasound.so.2"],
"libX11-guest.so": ["/lib/x86_64-linux-gnu/libX11.so.6", "/usr/lib/x86_64-linux-gnu/libX11.so.6"],
"libXrender-guest.so": ["/lib/x86_64-linux-gnu/libXrender.so.1", "/usr/lib/x86_64-linux-gnu/libXrender.so.1"],
"libXext-guest.so": ["/lib/x86_64-linux-gnu/libXext.so.6", "/usr/lib/x86_64-linux-gnu/libXext.so.6"],
"libXfixes-guest.so": ["/lib/x86_64-linux-gnu/libXfixes.so.3", "/usr/lib/x86_64-linux-gnu/libXfixes.so.3"]
}
}

Development:Setting up RootFS

2022-06-22T13:42:17Z

Neobrain:

[[Category:Development]]
= Setting up a RootFS =
'''For AArch64 hosts you are required to have an x86-64 rootfs'''. With or without 32bit compatibility libraries.

This isn't strictly required for testing on x86-64 host environments but is useful for ensuring correct behaviour there.

= Quick Setup with FEXRootFSFetch =
FEX comes with a FEXRootFSFetcher tool that can download an official preconfigured rootfs from the FEX servers.
This tool operates as either a terminal application or a Zenity application depending on if it is launched from terminal or not.

This tool will walk you through selecting a RootFS for FEX:
* If your host OS matches one in the download list then it will recommend that one
* After selecting the RootFS, it will download it
** This downloads to the default location in '''$HOME/.fex-emu/RootFS/'''
* After downloading the RootFS the tool will ask if you want to use the SquashFS file directly or extract it
* It will then ask if you want to set the downloaded RootFS as your default
** This writes to the default configuration in '''$HOME/.fex-emu/Config.json'''

You can skip the rest of the guide if you used FEXRootFSFetcher to download the rootfs.

== Troubleshooting ==
* The squashfs image will fail to extract if you don't have unsquashfs installed
** Install the squashfs-tools package to get this application

* '''SquashFS RootFS file corrupted?'''
** Rerun the tool, select the rootfs from the list, and if the rootfs already exists it will ask to verify
** If verification fails it will ask if you want to redownload
* '''Want to manually check the hash?'''
** Run the rootfs file directly through the tool to get a hash
** '''FEXRootFSFetcher $HOME/.fex-emu/RootFS/Ubuntu_21_10.sqsh'''
** Ubuntu_21_10.sqsh has hash: bf9507fbdbaec2cf
* '''Want to use the Zenity UI but still launch through terminal?'''
** Force Zenity from terminal with '''FEXRootFSFetcher 1> /dev/null'''
* '''What if my Linux distro isn't listed?'''
** Don't worry, select one that is most similar to yours or newest
** This will work in most cases
* '''How can I trust these images?'''
** These images are generated by scripts living in our public repo
** If you want to rebuild them yourself, you are free to do so.
** https://github.com/FEX-Emu/RootFS
* '''I want to add a distro'''
** Follow how our current rootfs is generated and add your own
** Open a pull request to add the rootfs setup
** Contact us on the official FEX-Emu Discord to get the RootFS added to the server.
* '''Where is the raw list of images at?'''
** The image links live on our website at the following link
** https://rootfs.fex-emu.org/file/fex-rootfs/RootFS_links_XXH3.txt
** This file is in a format that our tool understands
** Generally the format is:
*** Distro Name
*** Distro Version
*** Human readable name
*** Raw rootfs link
*** xxhash based file hash

= Making a custom rootfs prerequisites =
An x86-64 PC or FEX-Emu installed for chrooting

= Getting Started =
The easiest way to get started is to just download a prebuilt x86-64 rootfs from your favourite distribution and extract it to where you want.
* Ubuntu - http://cdimage.ubuntu.com/ubuntu-base/releases/
* Debian Debootstrap guide - https://wiki.debian.org/Debootstrap
** Debootstrap is more involved and not explained here.

Using chroot you can then enter the rootfs and install applications that you desire for testing purposes.
It's easiest to set this up on an x86-64 host and then copy it to an AArch64 device.

== Cross architecture chroot ==
'''You need to have binfmt_misc setup for x86-64 support on your host system.'''
* ls /proc/sys/fs/binfmt_misc
** If there are files in this folder then your host supports binfmt_misc
* Once you have binfmt_misc running and FEX-Emu installed in to it, you can chroot in to an x86-64 rootfs

== Chrooting in to rootfs with FEX-Emu Rootfs ==
* If you're using a FEX-Emu provided rootfs then we provide two scripts at the start to handle this
** '''unbreak_chroot.sh''' - This sets up the rootfs and chroots inside of it
** '''break_chroot.sh''' - This sets up the rootfs for using with FEX generally.
*** '''Make sure to run this after leaving the chroot!'''
** The remaining steps in this guide can be skipped

== Chrooting in to custom rootfs with FEX-Emu ==
* If you're setting up your own rootfs then you need to follow the binding steps that are following.
** We have a script that if you know what you're doing you can cherry pick parts from [https://github.com/FEX-Emu/RootFS/blob/main/unbreak_chroot.sh here]

= Additional TmpFS folders =
Depending on what you're doing in the chroot, you may need or want some tmpfs folders that applications are expecting

* sudo mount -t proc /proc ${ROOTFS}/proc/
* sudo mount --rbind --make-rslave /sys ${ROOTFS}/sys/
* sudo mount --rbind --make-rslave /dev ${ROOTFS}/dev/
* sudo mount --rbind --make-rslave /dev/pts ${ROOTFS}/dev/pts/
* sudo mount --rbind /lib/ld-linux-aarch64.so.1 ${ROOTFS}/lib/ld-linux-aarch64.so.1
* sudo mount --rbind /lib/aarch64-linux-gnu ${ROOTFS}/lib/aarch64-linux-gnu

Then of course, make sure to unmount these partitions when you're done

* sudo umount ${ROOTFS}/proc/
* sudo umount -R ${ROOTFS}/sys/
* sudo umount -R ${ROOTFS}/dev/pts/
* sudo umount -R ${ROOTFS}/dev/
* sudo umount -R ${ROOTFS}/lib/ld-linux-aarch64.so.1
* sudo umount -R ${ROOTFS}/lib/aarch64-linux-gnu

= Using the rootfs with FEX =
If you're using FEXLoader then you can pass the full path of the rootfs with the -R option.
Additionally you can put the rootfs folder inside of $HOME/.fex-emu/RootFS/<RootFS name>/ and FEX supports a "named" rootfs option
* eq: -R Ubuntu_2104 will use rootfs $HOME/.fex-emu/RootfS/Ubuntu_2104
* Read the Fex man page with man fex for more information

== Setting RootFS with FEXConfig ==
[[File:FEXConfig Load Default.png|frameless|Load the default options on first run]]
On the first run of FEXConfig, select the "Load Default Options" configuration.
* Subsequent runs should select "Open Default"

[[File:FEXConfig RootFS Selection.png|frameless|Select the RootFS from the FEXConfig location]]

FEXConfig will autopopulate the rootfs list with rootfs folders that exist in $HOME/.fex-emu/RootFS/

You can select one from the list or provide a full path in the RootFS: line to set it manually.

''' Make sure to select the File->Save option after the fact to save'''

= Additional Steps for building a custom rootfs =
== Running under WSL1 ==
/etc/hosts and /etc/resolv.conf change on the host side. You'll need to copy those over to the rootfs if you want network functionality to work.

== Ubuntu packages to install ==
Sample packages to install in the rootfs
* apt-get install pulseaudio libgles1 libgles2 libglx-mesa0 libgl1-mesa-dri libgl1-mesa-glx mesa-utils mesa-utils-extra
Pulseaudio specifically is required so pulseaudio will work when talking to the host pulseaudio

== File deletion ==
Some files need to be deleted from the rootfs otherwise some applications will fail to work.

Note that deleting these files '''WILL''' break the rootfs and you won't be able to chroot in to it anymore.

A likely better route would be a script that moves these files and folders then brings them back when you want to chroot

=== Files to remove ===
* ${ROOTFS}/etc/hosts
* ${ROOTFS}/etc/resolv.conf
* ${ROOTFS}/etc/timezone
* ${ROOTFS}/etc/localtime
* ${ROOTFS}/etc/passwd

=== Folders to remove ===
* ${ROOTFS}/boot
* ${ROOTFS}/dev
* ${ROOTFS}/home
* ${ROOTFS}/media
* ${ROOTFS}/mnt
* ${ROOTFS}/proc
* ${ROOTFS}/root
* ${ROOTFS}/srv
* ${ROOTFS}/tmp
* ${ROOTFS}/var/cache/apt
* ${ROOTFS}/var/lib/apt
* ${ROOTFS}/sys
* ${ROOTFS}/opt

== Creating a Squashfs rootfs from custom rootfs ==
SquashFS is a great way to do binary distribution of the rootfs. If you have created a rootfs and you want to roll it in to a squashfs like the FEX-Emu provided images then this is information on how to do that.

'''SquashFS is currently only recommended for binary distribution. User should unpack it to not get dangling processes and mounts.'''
* Offers tooling to mount in userspace
* Supports ZSTD compression
* Is a readonly partition that matches FEX's use case

=== Tools that FEX needs to run squashfs ===
* squashfuse
* fusermount

=== Making the squashfs image ===
Making the squashfs image is very straightforward
* mksquashfs ${ROOTFS}/ Squashfs.sqsh -comp zstd

This creates a file called Squashfs.sqsh that contains the original rootfs that we generated above.

=== Using the squashfs image in FEX ===
* Copy the resulting sqsh file in to ${XDG_DATA_HOME}/.fex-emu/RootFS/
** If ${XDG_DATA_HOME} doesn't exist then that will be replaced with ${HOME}
* Use FEXConfig to select the sqsh file as a named rootfs
* or pass the name in to the FEX_ROOTFS argument in the various ways FEX supports
* or the absolute path if you don't want it in the FEX RootFS folder

=== Extracting the squashfs ===
If you don't want to use squashfs for whatever reason then you can extract it in to the FEX RootFS folder and use the directory directly
* unsquashfs -f -d NewRootFS/ Squashfs.sqsh

=== If you want to install packages in the squashfs ===
The squashfs image is immutable and can't be modified.
* Extract the squashfs first
* Change the working directory to the rootfs
* execute `unbreak_chroot.sh`
* `apt-get update`
* `apt-get install <Whatever you want>`
* `exit` to leave the chroot
* `break_chroot.sh`
* Use the new extracted chroot directly, or follow the previous steps to recompress it to a squashfs format.

=== Known problems with squashfs ===
'''FEX automatically mounts your squashfs file and can leave dangling squashfs mounts around in some instances.'''
* '''This is fairly rare but it can occur. Whenever an instance of FEX is run, it registers with the FEXMountDaemon'''
** FEXMountDaemon then watches these instances of FEX to ensure they have exited before unmounting
** There is roughly a ten second grace period of no instances of FEX running before unmounting
** If an instance of FEX starts running then it will reuse the mount partition and reset the timeout
* To get rid of these dangling mounts if they do manage to manifest, use the following commands
** fusermount -u /tmp/.FEXMount* ; rmdir /tmp/.FEXMount*
** These don't require root to unmount
* In some cases the unmounting can end up returning busy status and not unmount
** Check the output of `mount` for any /tmp/.FEXMount folders
** This is due to either FEXMountDaemon or squashfuse still running in the background
** Can force the application to exit
** step 1: kill `pidof squashfuse`
** This will attempt to send SIGTERM to all squashfuse applications, which will unmount the partition on terminate
** Failing that, you can try to be more abusive
** kill -9 `pidof FEXMountDaemon`
** This will forcibly terminate FEXMountDaemon which should also take out the child mounted process
* If any instances of the mount still exists then you may need to kill more FEX processes.
** Any FEXInterpreter process will end up keeping the folder mounted and keep it busy

'''strace breaks squashfs'''
* If you run FEX with `strace -f` then the process that mounts your rootfs loses its setuid bit.
** This means you then can't mount the rootFS
* The only workaround is to not use a squashfs in this instance.
* Alternatively you can spin up a FEXBash instance to force squashfs to be mounted. Then strace a different FEX instance which will use the mounted squashfs location.

Development:Setting up FEX

2022-06-22T13:42:11Z

Neobrain:

[[Category:Development]]
== Build dependencies ==
* git
* cmake
* ninja-build
* ccache (Recommended if you're rebuilding constantly)
* clang
* lld (For ThinLTO)
* binfmt-support (For AArch64 binfmt_misc support)
* libsdl2-dev (For GUI)
* libepoxy-dev (For GUI)
* g++-x86-64-linux-gnu (For building thunks)
* nasm (only if building tests)
* python3-clang (for struct verifier)
* libstdc++-10-dev-i386-cross (for struct verifier)
* libstdc++-10-dev-amd64-cross (for struct verifier)
* libstdc++-10-dev-arm64-cross (for struct verifier)
* squashfs-tools (For squashfs rootfs support)
* libc-bin or glibc-tools (For unittests, needs catchsegv from this)

=== Gentoo dependencies ===
* dev-vcs/git
* cmake
* ninja
* ccache
* clang
* lld
* libsdl2
* libepoxy
* nasm

'''Use gubbins'''
* >=media-libs/libglvnd-1.3.4 X
== Build Configuration ==
* Ensure release mode is enabled
* Disable static pie
** Bug with glibc means static-pie needs to be disabled for thunks to work.
** This is necessary for chroot and pressure-vessel, but is currently incompatible with thunks
** If using LLD then you need at least LLD 13.0
** If using LLD then you need at least glibc 2.35
** Recommendation: '''Disable LLD, use binutils ld'''
* Enable LTO - '''Currently bugged in some configurations'''
** Disable if enabling static-pie
* Disable test building

git clone --recurse-submodules https://github.com/FEX-Emu/FEX.git
cd FEX
mkdir Build
cd Build
CC=clang CXX=clang++ cmake -DCMAKE_INSTALL_PREFIX=/usr -DCMAKE_BUILD_TYPE=Release -DENABLE_LTO=False -DENABLE_STATIC_PIE=False -DBUILD_TESTS=False -DENABLE_ASSERTIONS=False -G Ninja ..
ninja

== Installation ==
sudo ninja install

=== On AArch64 hosts ===
You can install a binfmt_misc handler for both 32bit and 64bit x86 execution directly from the environment.
If you already have box86's 32bit binfmt_misc handler installed then I don't recommend installing FEX's until it is useful.
Make sure to have run install prior to this, otherwise binfmt_misc will install an old handler even if the executable has been updated.

sudo ninja binfmt_misc_32
sudo ninja binfmt_misc_64

'''binfmt_misc problems'''
* Double check that binfmt_misc has worked
** `ls /usr/share/binfmts/FEX*` should return files
** `ls /var/lib/binfmts/FEX*` should return files
** `/proc/sys/fs/binfmt_misc/status` should exist
*** If it doesn't exist then make sure your kernel has `CONFIG_BINFMT_MISC=y`
*** If it still doesn't exist then make sure systemd isn't disabling it `systemctl enable binfmt-support`

== Runtime Configuration ==
FEXConfig is the application to use for runtime configuration setup.

CTRL+SHIFT+O will open default "global" configuration.

CTLR+S saves it.

=== Things to ensure to set ===
On AArch64 host, a rootfs path is mandatory, while it is optional on x86-64 host

* Core: JIT
* RootFS
* Silent Logging (Useful to get FEX information spam out of the way)
* Block Size (500 is a good default. Optimization passes may break things though)
* Multiblock: enabled (Can cause more stuttering)

More Details at: Follow the instructions here https://wiki.fex-emu.org/index.php/Development:Configuring_FEX
== RootFS generation ==
AArch64 hosts require a rootfs for running applications
Follow the instructions here https://wiki.fex-emu.org/index.php/Development:Setting_up_RootFS

== Installing Thunks ==
Only if you want

Follow the instructions here: https://wiki.fex-emu.org/index.php/Development:Setting_up_Thunks

Additional details here: https://github.com/FEX-Emu/FEX/blob/main/ThunkLibs/README.md

== Binaries ==
* FEXLoader
** This is useful for development purposes but is generally recommended against using.
** Allows passing arguments to FEX for configuration purposes, which can break under some execve situations
* FEXInterpreter
** This is a hardlink to FEXLoader which refuses to treat arguments as configuration options
** Is recommended to use over FEXLoader; Using FEXConfig to set configuration
** Requires being a hardlink or executable copy, execve logic sees through a softlink
** Is the executable used for binfmt_misc for interpreters
* FEXConfig
** Used for setting configuration options from a GUI
** Written using Dear ImGui
** Can load/save global configuration and application configuration files
** Application configuration files are compared against the executable filename
* FEXBash
** Can be useful for starting a bash instance that is running under emulation
** It's not a chroot! Don't use sudo inside of this environment!
* FEXMountDaemon
** This is a mount daemon that will run in the background when you are using a squashfs based rootfs
** Usually one one instance open at any given moment
** It usually closes when the FEXInterpreter exits, but not always. See the problems section in setting up a rootfs
*** Has a ten second timeout window in case a another instance of FEX opens and uses it.

=== Debug binaries ===
* TestHarnessRunner
** Used for running FEX's ASM unit tests
** Runs some binary in a host x86_64 host environment or FEX and checks results
** Useful for quickly creating ASM based unit tests
* IRLoader
** Used for running FEX's IR based unit tests
** Only runs through FEX and checks the results
** Minimal number of tests
* UnitTestGenerator
** This is supposed to be an x86-64 instruction generator
** Has an understanding of x86-64 instruction encoding format
** Generates hundreds of thousands of instructions
** Used to be used with a lockstep runner to ensure correct behaviour
** Largely unsupported, avoiding undefined behaviour is difficult and FEX doesn't need to match undefined behaviour
** Lockstep hostrunner wasn't ever made to be bug free, hard to support
** Doesn't ensure correct behaviour on AArch64 device

Development:Renderdoc

2022-06-22T13:42:05Z

Neobrain:

[[Category:Development]]
= Getting a capture =
== Without Thunks ==
* Ensure Renderdoc is installed in the rootfs

=== OpenGL ===
* Set LD_PRELOAD=librenderdoc.so as an environment variable
* This can be done in FEXConfig or an application profile to limit its reach

=== Vulkan ===
* Ensure the renderdoc layer is installed
* Set ENABLE_VULKAN_RENDERDOC_CAPTURE=1 as an environment variable
* This can be done in FEXConfig or an application profile to limit its reach

=== Alternatively ===
You can use renderdoccmd to launch an application directly.
* renderdoccmd capture -w <program> <args...>
As long as this application is installed in your rootfs then it should work for both OpenGL and Vulkan applications.

=== Capture results ===
* Once you have captured a frame in an application running renderdoc (F12 typically)
* The results will be in /tmp/RenderDoc/ as *.rdc files
* Technically these are meant to be moved over to the qrenderdoc application but you can rip them directly

== With Thunks ==
Likely to be the same as without thunks, but instead of adding environment variables inside of FEX. One needs to do it outside of FEX instead

Development:gfxreconstruct

2022-06-22T13:42:00Z

Neobrain:

[[Category:Development]]
= gfxreconstruct =
gfxreconstruct is a Vulkan layer that is useful for capturing a frame in a robust way.

This is usually useful for capturing frames that is more portable than renderdoc.

https://github.com/LunarG/gfxreconstruct

= Setting up gfxreconstruct =
First thing you must do is build gfxreconstruct from source. Either for your host or guest rootfs, depending on if you're using thunks or not.

If building for the x86 rootfs then make sure to install it in to the rootfs after building. This can be done by just copying the install files over afterwards.

== Set up environment variables ==
* VK_INSTANCE_LAYERS=VK_LAYER_LUNARG_gfxreconstruct
* GFXRECON_CAPTURE_TRIGGER=F12
* GFXRECON_CAPTURE_FILE=/tmp/gfxrecon_capture.gfxr

Once you have the environment variables set up you can then launch a vulkan game. There will be some additional output in the terminal to know when the layer is capturing. You can then press F12 in the application and you'll get a capture in tmp after a few seconds. Be careful as gfxreconstruct will overwrite the capture on each press, so you will want to move each capture file.

== Replay ==
gfxrecon-replay can be used to replay the capture which is quite useful.

Development:Debugging FEX with Signals

2022-06-22T13:41:52Z

Neobrain:

[[Category:Development]]
= Signals and you =

FEX uses signals in quite a few ways that may cause debugging to be a bit of a pain. This is necessary for correct operation so it is an evil we must live with.

== Signals that FEX uses ==
* SIGBUS - The AArch64 JIT generates unaligned loadstore instructions then captures and backpatches these when they fault
** This is necessary for emulating x86-TSO on AArch64 and can't be dropped without massive performance loses
* SIGILL - All of the CPU backends generate this fault and is necessary for the emulator to return from a guest signal
** Going in to a signal happens frequently, gdb will usually ignore the entry when it is SIGALRM or SIGUSR*
** We could choose a different signal, but this one can be generated in one instruction which makes it nice to use
** Using a signal to punch back through to the guest return path is necessary since we need to save and restore guest registers when guest signals occur.
* SIG63 - We use this for various ancillary functions
** This signal currently tells a thread that it should stop running, to pause its execution, or to resume from its paused frame
** This is another necessary signal since if we want to safely shutdown the emulator and a thread is in a syscall, there is no way for it to leave without a signal.
** The stop signal specifically will have the CPU backend long jump from the signal handler to the exit stage of the ASM dispatcher for safe shutdown
** This slightly increases performance because then there aren't atomic checks per iteration of the execution loop for if we should stop executing

= Debugging FEX =
Read more here [[Development:Debugging_Crash]]

= Profiling FEX code =
You'll want to enable a style of JIT code mark-up for perf to annotate with
* FEX currently has three options for this
* Depending on what you're trying to profile you will want to enable one of these options
** FEX_GLOBALJITNAMING
*** Useful if you want to throw all of FEX JIT space in to a global-ish namespace
** FEX_LIBRARYJITNAMING
*** Useful if you want to clump FEX JIT space in to library specific code segments
*** Can be nice for finding libraries to thunk
** FEX_BLOCKJITNAMING
*** This is likely the one you want if you're profiling your application's code
*** This will give you hot blocks in `perf top`

Enabling one of these will give you some symbols when capturing FEX with `perf top`. The block naming is useful for seeing hot blocks in program space so if something ends up being particularly slow in FEX, it can get more isolated.
* We don't currently have an option to relate JIT code directly to function symbols in the ELF at this point.

= Debugging the guest application =
When FEX is started with the -G argument it will open a gdbstub socket listening on port 8086. This can be connected to with gdbserver, ida, ghidra, or binary ninja.

This is currently only useful for inspection, gdbstub doesn't yet send guest signals over through gdbstub!

Development:Debugging Crash In Thunks

2022-06-22T13:41:44Z

Neobrain:

[[Category:Development]]
== Once Thunks are enabled ==
* Once you've verified your crash is in a thunk then this becomes quite a bit easier

gdb --args FEXInterpreter /usr/bin/vulkaninfo
Thread 1 "FEXInterpreter" received signal SIGSEGV, Segmentation fault.
0x0000007fc82240f4 in tu_CreateDevice () from /lib/aarch64-linux-gnu/libvulkan_freedreno.so
(gdb) bt
#0 0x0000007fc82240f4 in tu_CreateDevice () from /lib/aarch64-linux-gnu/libvulkan_freedreno.so
#1 0x0000007fc8642b94 in function_traits<decltype (&(fexfn_unpack_libvulkan_freedreno_vkCreateDevice(fexfn_packed_args_libvulkan_freedreno_vkCreateDevice*)))>::result_t fexfn_type_erased_unpack<&(fexfn_unpack_libvulkan_freedreno_vkCreateDevice(fexfn_packed_args_libvulkan_freedreno_vkCreateDevice*))>(void*) () from /usr/lib/fex-emu/HostThunks//libvulkan_freedreno-host.so
#2 0x0000007fe25e4028 in ?? ()
Backtrace stopped: previous frame inner to this frame (corrupt stack?)

* Okay, that's a direct crash in Turnip, let's rebuild Mesa with a debug build and try again
Thread 1 "FEXInterpreter" received signal SIGSEGV, Segmentation fault.
0x0000007fc80bc600 in tu_CreateDevice (physicalDevice=0x5559947aa0, pCreateInfo=0x7ff75fda20, pAllocator=0x0, pDevice=0x7fcbe7b358) at ../src/freedreno/vulkan/tu_device.c:1632
1632 memset(ptr, 0x42, 16);
(gdb) bt
#0 0x0000007fc80bc600 in tu_CreateDevice (physicalDevice=0x5559947aa0, pCreateInfo=0x7ff75fda20, pAllocator=0x0, pDevice=0x7fcbe7b358) at ../src/freedreno/vulkan/tu_device.c:1632
#1 0x0000007fc80eab9c in vk_tramp_CreateDevice (physicalDevice=0x5559947aa0, pCreateInfo=0x7ff75fda20, pAllocator=0x0, pDevice=0x7fcbe7b358) at src/vulkan/runtime/vk_dispatch_trampolines.c:78
#2 0x0000007fc8642b94 in function_traits<decltype (&(fexfn_unpack_libvulkan_freedreno_vkCreateDevice(fexfn_packed_args_libvulkan_freedreno_vkCreateDevice*)))>::result_t fexfn_type_erased_unpack<&(fexfn_unpack_libvulkan_freedreno_vkCreateDevice(fexfn_packed_args_libvulkan_freedreno_vkCreateDevice*))>(void*) () from /usr/lib/fex-emu/HostThunks//libvulkan_freedreno-host.so
#3 0x0000007fe25e426c in ?? ()

* Okay the crash is at.. `tu_device:1632`, let's go and debug that line

1631 void* ptr = (void*)1;
1632 memset(ptr, 0x42, 16);

* Who put that in there?!

Development:Debugging Crash

2022-06-22T13:41:39Z

Neobrain:

[[Category:Development]]
== Getting Started ==
* Debug an application with `gdb --args FEXInterpreter <application full path>`
* Under GDB make sure to do `handle SIGBUS SIGILL SIG63 noprint`
** We use signals for various things, check out [[Development:Debugging_FEX_with_Signals|Here]] for more information

== Crash in emulated/JIT code ==
Walking through debugging a simple test application that is crashing.
$ gdb --args FEXInterpreter ./sigsegv_test
Reading symbols from FEXInterpreter...
(gdb) r
Starting program: /usr/bin/FEXInterpreter ./sigsegv_test
[Thread debugging using libthread_db enabled]
Using host libthread_db library "/lib/aarch64-linux-gnu/libthread_db.so.1".
[New Thread 0x7fccb75f30 (LWP 90107)]

Thread 2 "FEXInterpreter" received signal SIGSEGV, Segmentation fault.
[Switching to Thread 0x7fccb75f30 (LWP 90107)]
0x0000007fccfb9ec8 in ?? ()

* Okay, we have a sigsegv. Let's double check that it is JIT code (aka, guest emulated code)
(gdb) disas $pc,+32
Dump of assembler code from 0x7fe26ebc20 to 0x7fe26ebc40:
=> 0x0000007fe26ebc20: stlrb w22, [x21]
0x0000007fe26ebc24: mov x4, #0x0 // #0
0x0000007fe26ebc28: ldr x10, [x20]
0x0000007fe26ebc2c: add x21, x20, #0x8
0x0000007fe26ebc30: mov x22, #0x0 // #0
0x0000007fe26ebc34: strb w22, [x28, #428]
0x0000007fe26ebc38: mov x22, #0x0 // #0
0x0000007fe26ebc3c: strb w22, [x28, #431]
End of assembler dump.
(gdb) info reg x9
x9 0x0 0
* Looks like JIT code, even doing accesses to x28 which is the FEX CPU state
* Code has no backtrace which reinforces this
* Code is doing an atomic store, which reinforces this is FEX emulating the x86 TSO memory model

* Now that we have checked that we are in the JIT code. Where are we in the guest side?
* Let's dump the FEX CPU state information that is directly pointed to in x28 at all times in JIT code.

(gdb) p/x ((FEXCore::Core::CpuStateFrame*)$x28)->State
$3 = {rip = 0x401110, gregs = {0x416eb0, 0x7fe1e3b640, 0xffffffffffffff70, 0x0, 0x7fe1e3bf30, 0x0, 0x416eb0, 0x7fe1e3ae28, 0x0, 0x7fe1e3b640, 0x8, 0x7fe2054cc0, 0x7ff75ff48e, 0x7ff75ff48f, 0x0, 0x7fe163b000}, xmm = {{0x0, 0x0}, {0x0, 0x0}, {0xdeadbeef, 0xbad0dad1} <repeats 14 times>}, es = 0x0, cs = 0x0, ss = 0x0, ds = 0x0, gs = 0x0, fs = 0x7fe1e3b640, flags = {0x0, 0x1, 0x0,
0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x1, 0x0 <repeats 38 times>}, mm = {{0x0, 0x0}, {0x0, 0x0}, {0x0, 0x0}, {0x0, 0x0}, {0x0, 0x0}, {0x0, 0x0}, {0x0, 0x0}, {0x0, 0x0}}, gdt = {{base = 0x0} <repeats 32 times>}, FCW = 0x37f, FTW = 0xffff}

* Looks like our guest RIP is currently `0x401110`
** Consult the `info proc mappings` again
0x401000 0x402000 0x1000 0x1000 {...}/sigsegv_test
* Yep, we are inside our test application
* For a simple test, let's load the application in gdb-multiarch and disassemble where we are
$ gdb-multiarch ./sigsegv_test
Reading symbols from ./sigsegv_test...
(gdb) set disassembly-flavor intel
(gdb) disas 0x401110
Dump of assembler code for function main(int, char**):
0x0000000000401110 <+0>: push rbp
0x0000000000401111 <+1>: mov rbp,rsp
0x0000000000401114 <+4>: mov DWORD PTR [rbp-0x4],0x0
0x000000000040111b <+11>: mov DWORD PTR [rbp-0x8],edi
0x000000000040111e <+14>: mov QWORD PTR [rbp-0x10],rsi
0x0000000000401122 <+18>: mov rax,QWORD PTR [rbp-0x10]
0x0000000000401126 <+22>: movsxd rcx,DWORD PTR [rbp-0x8]
0x000000000040112a <+26>: mov rax,QWORD PTR [rax+rcx*8]
0x000000000040112e <+30>: mov QWORD PTR [rbp-0x18],rax
0x0000000000401132 <+34>: mov rax,QWORD PTR [rbp-0x18]
0x0000000000401136 <+38>: mov BYTE PTR [rax],0x63
0x0000000000401139 <+41>: xor eax,eax
0x000000000040113b <+43>: pop rbp
0x000000000040113c <+44>: ret
End of assembler dump.
(gdb)

* Okay, not super helpful since FEX translates instructions in to blocks, `0x401110` is just our starting address
** It's in this code somewhere, let's change some FEX settings to get a clearer picture
* Set block size to one instruction and disable multiblock
** See the image in FEXConfig to the right
[[File:FEXConfig 1Inst NoMulti.png|thumb]]
* Now rerun our test application and find the new RIP

(gdb) p/x ((FEXCore::Core::CpuStateFrame*)$x28)->State.rip
$2 = 0x401136

* Alright, now we know the RIP is exactly at `0x401136`
* Back in gdb-multiarch
(gdb) disas 0x401136,+1
Dump of assembler code from 0x401136 to 0x401137:
0x0000000000401136 <main(int, char**)+38>: mov BYTE PTR [rax],0x63

* Looks like something in main is storing 0x63 to a nullptr
* In this simple case we can now take a look at the test application's source and find the problem.
** We know the problem is in the first block of main()
** We know the exact instruction that it is at
** We know it's something storing a byte to memory
* For more complex cases it is likely necessary to use reverse engineering tools
** BinaryNinja, Ghidra, IDA, and Hopper are all examples of tools like this

== What to do from here ==
Now it becomes a lot harder. You don't get a typical debugging environment or even clean backtraces.

FEX's gdbserver integration is sorely lacking so you can't even use a remote gdb server connecting to FEX right now.

If you enable thunks you can get better backtraces here. [[Development:Debugging_Crash_In_Thunks|Debugging_Crash_In_Thunks]]

== Attempting to use FEX-Emu's gdbserver implementation ==
'''Here be dragons'''

FEX supports gdbserver as an integration. It's implementation is significantly limited but can still be used for debugging and getting some backtraces.
* Currently hardcodes the port to use as `8086` and if you have multiple gdbserver processes running then it will encounter problems.
* Currently does not follow processes through fork/execve at all. No multiprocess support
** This means you '''must''' only start the process you're caring about debugging
* Currently starts the process paused and will wait until gdb attach before continuing
** No way to start a FEX instance then attach at some later point
* Ctrl-C to stop the FEX process needs to be done twice
** Maybe with a small delay inbetween because gdb needs to fetch a bunch of data on pause
** Known bug, unknown why broken at the moment

FEXLoader -G -- <Application> <Args...>

== Double checking if we are in JIT code ==
(gdb) info reg pc
pc 0x7fccfb9ec8 0x7fccfb9ec8

(gdb) info proc mappings
...
0x7fccfb9000 0x7fcdfb9000 0x1000000 0x0

* Looks like FEX JIT mapping, we start out at 16MB but scale up to 128MB
* Depending on version of FEX we can check the base mapping for a unique string
(gdb) p (char*)0x7fccfb9000
$4 = 0x7fccfb9000 "FEXJIT::Arm64JITCore::"

Development:Adding Config Options

2022-06-22T13:41:16Z

Neobrain:

[[Category:Development]]
Configuration options are programatically generated from json.

This file lives in `FEXCore/Source/Interface/Config/Config.json`

= Options section =
To add a configuration option that can be set from JSON, FEXLoader arguments, or Environment variables you need to add the option to this file.

Currently FEX has six groups to group configs
* CPU
* Emulation
* Debug
* Logging
* Hacks
* Misc

== Configuration option object ==
Each JSON object under a group becomes a configuration.

'''Object Layout'''
* Key:
* Becomes the C++ enum. eg. 'Multiblock' becomes `FEXCore::Config::ConfigOption::CONFIG_MULTIBLOCK`
* Also used for the environment variable. Becomes `FEX_MULTIBLOCK`
* Also becomes the FEXLoader argument. Becomes `--multiblock`
* If the object type is boolean then a negated argument also is generated. `--no-multiblock`

* ShortArg:
* This is the string that is used for FEXLoader short arguments
* eg. Multiblock uses m, which turns in to -m in FEXLoader

* Type:
* Can be `[bool, uint8, int32, uint32, uint64, str, strarray]`
* This is what the argument type ends up being once loaded from a configuration backing

* Default:
* Default value that matches this object
* Used to generate code so needs to handle being codified. Compilation will fail otherwise
* If Type is str or strarray then this string will be automatically wrapped with double-quotes when generated

* TextDefault:
* Used by FEXLoader argument loading
* If the `Default` option which is used in code can't be represented by text. Like an enum being used.
* eg. `Core` has a `Default` which is an enum. `TextDefault` is set to `irjit` for the user to handle this case

* Choices:
* Used by FEXLoader argument loading
* Allows us to represent the textual options that a argument can accept
* FEXLoader validates that a choice is supported otherwise errors out
* eg. `Core` supports `[ "irint", "irjit", "host" ]`

* ArgumentHandler:
* Used by FEXLoader argument loading
* If an argument option needs some special handling to be parsed then this points to a C++ function in the argument loader
* `Core` uses it to convert the textual representation in to an enum
* '''Don't''' use this as a way to modify other configuration options from FEXLoader
* Do fix-ups in the ReloadMetaLayer function in FEXCore/Config.cpp instead

* Desc:
* Short description of what this option does
* This is used to generate the help text with `FEXLoader --help` and also the text in the man pages

== UnnamedOptions section ==
These are configuration options that can '''NOT''' be set by user means. There won't be any FEXLoader, Environment variable, or json helpers generated to load these.

These need to exist as a way to have internal configuration options existing in the same global space as user options. Allows us to pull internal configuration options the same way as user configurations.

These can exist under any group config. Currently only `Misc` is used. Only three things need to be set for these
* Key: This is only used to generate the enum in code
* Type: Same supported types as the regular options
* Default: The default value of this configuration

== Getting configurations in FEX code ==
FEX provides a `FEX_CONFIG_OPT` define to help developers get configuration options.
* FEX_CONFIG_OPT(name, enum)
* eg: `FEX_CONFIG_OPT(Cores, THREADS);`
* This will load the THREADS configuration option from the configuration directory and save the value in to local value called `Cores`
* Gotchas:
* Construction of the object queries the the global configuration layer. This can be slow so be careful with its use.
* Upon construction of the `FEXCore::Config::Value<T>` object it will pull the string configuration
* Then it will convert that value in to whatever type the value needs
* In high performance code, keep constructor instantiation outside of the hot code path
* Construct the configuration inside of a class definition so it only pulls the configuration once
* Caveats:
* If the configuration changes then this doesn't pick up the change
* Configuration isn't meant to change live in FEX so this is typically a non-issue
* Frontend object loader needs to be careful with its configuration loading to ensure the meta layer stays in sync
* ELFLoader.cpp for examples here

Development:Adding Config Options

2022-06-22T13:40:25Z

Neobrain: Add development category

[[Category:Development]]

Configuration options are programatically generated from json.

This file lives in `FEXCore/Source/Interface/Config/Config.json`

= Options section =
To add a configuration option that can be set from JSON, FEXLoader arguments, or Environment variables you need to add the option to this file.

Currently FEX has six groups to group configs
* CPU
* Emulation
* Debug
* Logging
* Hacks
* Misc

== Configuration option object ==
Each JSON object under a group becomes a configuration.

'''Object Layout'''
* Key:
* Becomes the C++ enum. eg. 'Multiblock' becomes `FEXCore::Config::ConfigOption::CONFIG_MULTIBLOCK`
* Also used for the environment variable. Becomes `FEX_MULTIBLOCK`
* Also becomes the FEXLoader argument. Becomes `--multiblock`
* If the object type is boolean then a negated argument also is generated. `--no-multiblock`

* ShortArg:
* This is the string that is used for FEXLoader short arguments
* eg. Multiblock uses m, which turns in to -m in FEXLoader

* Type:
* Can be `[bool, uint8, int32, uint32, uint64, str, strarray]`
* This is what the argument type ends up being once loaded from a configuration backing

* Default:
* Default value that matches this object
* Used to generate code so needs to handle being codified. Compilation will fail otherwise
* If Type is str or strarray then this string will be automatically wrapped with double-quotes when generated

* TextDefault:
* Used by FEXLoader argument loading
* If the `Default` option which is used in code can't be represented by text. Like an enum being used.
* eg. `Core` has a `Default` which is an enum. `TextDefault` is set to `irjit` for the user to handle this case

* Choices:
* Used by FEXLoader argument loading
* Allows us to represent the textual options that a argument can accept
* FEXLoader validates that a choice is supported otherwise errors out
* eg. `Core` supports `[ "irint", "irjit", "host" ]`

* ArgumentHandler:
* Used by FEXLoader argument loading
* If an argument option needs some special handling to be parsed then this points to a C++ function in the argument loader
* `Core` uses it to convert the textual representation in to an enum
* '''Don't''' use this as a way to modify other configuration options from FEXLoader
* Do fix-ups in the ReloadMetaLayer function in FEXCore/Config.cpp instead

* Desc:
* Short description of what this option does
* This is used to generate the help text with `FEXLoader --help` and also the text in the man pages

== UnnamedOptions section ==
These are configuration options that can '''NOT''' be set by user means. There won't be any FEXLoader, Environment variable, or json helpers generated to load these.

These need to exist as a way to have internal configuration options existing in the same global space as user options. Allows us to pull internal configuration options the same way as user configurations.

These can exist under any group config. Currently only `Misc` is used. Only three things need to be set for these
* Key: This is only used to generate the enum in code
* Type: Same supported types as the regular options
* Default: The default value of this configuration

== Getting configurations in FEX code ==
FEX provides a `FEX_CONFIG_OPT` define to help developers get configuration options.
* FEX_CONFIG_OPT(name, enum)
* eg: `FEX_CONFIG_OPT(Cores, THREADS);`
* This will load the THREADS configuration option from the configuration directory and save the value in to local value called `Cores`
* Gotchas:
* Construction of the object queries the the global configuration layer. This can be slow so be careful with its use.
* Upon construction of the `FEXCore::Config::Value<T>` object it will pull the string configuration
* Then it will convert that value in to whatever type the value needs
* In high performance code, keep constructor instantiation outside of the hot code path
* Construct the configuration inside of a class definition so it only pulls the configuration once
* Caveats:
* If the configuration changes then this doesn't pick up the change
* Configuration isn't meant to change live in FEX so this is typically a non-issue
* Frontend object loader needs to be careful with its configuration loading to ensure the meta layer stays in sync
* ELFLoader.cpp for examples here

Development:Setting up Library Forwarding

2022-06-22T13:38:33Z

Neobrain: Add development category

[[Category:Development]]

== Build Dependencies ==
* g++-x86-64-linux-gnu (For building thunks)
* '''Development libraries for various thunks'''
* libdrm-dev
* libxcb-present-dev
* libxcb-dri2-0-dev
* libxcb-dri3-dev
* libxcb-glx0-dev
* libxcb-shm0-dev
* libxshmfence-dev
* libvulkan-dev

== Building vulkan-docs if your host is too old ==
If you're host's libvulkan-dev is too old, which is highly likely then you'll need to install a new set of vulkan headers

* git clone https://github.com/KhronosGroup/Vulkan-Headers.git
* cd Vulkan-Headers
* mkdir Build
* cd Build
* cmake -DCMAKE_INSTALL_PREFIX=/usr ..
* sudo make install

== Setting cmake to build thunks ==
Pass in to cmake the option to enable thunk building.
* -DBUILD_THUNKS=True

== FEX thunk options ==
* FEX_THUNKHOSTLIBS, -t <Thunk Host Libs Path>, --thunkhostlibs=<Thunk Host Libs path>, JSON: '''ThunkHostLibs:<Path>'''
** Folder to find the host-side thunking libraries.
** FEX by default installs these files to: '''/usr/lib/fex-emu/HostThunks/'''
* FEX_THUNKGUESTLIBS, -j <Thunk Host Libs Path>, --thunkguestlibs=<Thunk Host Libs path>, JSON: '''ThunkGuestLibs:<Path>'''
** Folder to find the guest-side thunking libraries.
** FEX by default installs these files to: '''/usr/share/fex-emu/GuestThunks/'''
* FEX_THUNKCONFIG, -k <Thunk Config json path>, --thunkconfig=<Thunk Config json path>, JSON: '''ThunkConfig:<Path>'''
** A json file specifying where to overlay the thunks.
** If the filename isn't a path then FEX will search for the file in '''$HOME/.fex-emu/ThunkConfigs/'''

== How to use Thunks ==
Once Thunks are built and installed then you can can set the three configuration options to the correct paths

* The user can likely set '''ThunkHostLibs''' and '''ThunkGuestLibs''' to the global install path
** Unless these are set in the '''ThunkConfig''' file, these will be unused
** Overriding these to another location is useful for application specific hacks if necessary
* Setting the global application profile for '''ThunkConfig''' is useful for testing but not recommended for long term use
** Due to the unstable nature of thunks, it is recommended to use application specific profiles to enable thunks
** This allows the user to test applications first using the global config, then once confirmed it works, generating an application profile
** Recommended to reduce the application profile to the minimum set of options. FEXConfig will save a full config.

== Thunks Database format ==
FEX provides a ThunksDB.json file which allows the user to describe a thunk mapping. '''$prefix/share/fex-emu/ThunksDB.json'''

The user can also provide a '''$HOME/.fex-emu/ThunksDB.json''' which will overlay the system installed path

This allows the user to configure the thunk file mapping once, since it is unlikely to change between applications. The user can also mix and match ThunksDB and regular thunks configuration if a complex configuration is necessary.

* ThunksDB.json example
{
"DB": {
"<User Defined Name>": {
"Library": "<Guest Library Name>.so",
"Depends": [
"<Other User Defined Name>"
],
"Overlay": [
"<System Library Path to library>"
]
}
}
}

Look at the FEX provided library for a real implementation, or just use the provided configuration.

Once the user has a viable ThunksDB provided, now configure FEX with a ThunkConfig filepath with the application thunk configuration.

This thunks configuration file can now enable Thunk definitions using the user defined names.

* ThunkConfig example
{
"ThunksDB": {
"GL": 0,
"xcb": 1,
"xcb-dri2": 1,
"xcb-dri3": 1,
"X11": 0,
"xcb-xfixes": 1,
"xcb-shm": 1,
"xcb-sync": 1,
"xcb-randr": 0,
"xcb-present": 0,
"xcb-glx": 1,
"xshmfence": 1,
"drm": 0,
"Xrender": 0,
"Xext": 0,
"Xfixes": 0,
"Vulkan-lavapipe": 1,
"Vulkan-radeon": 1,
"Vulkan-freedreno": 1,
"Vulkan-intel": 1,
"Vulkan-panfrost": 1,
"Vulkan-virtio": 1,
"asound": 0
}
}

== Limitations and problems ==
* 32-bit thunks aren't available currently
** Requires kernel patches for ARM64 support
* Thunks aren't fully stable. Expect missing symbols and crashes.

== Legacy ThunkConfig example ==
'''This is legacy path and not recommended.'''

This is just an example for how one could lay out the thunks.
* Save this to a file somewhere and point '''ThunkConfig''' option to it
** '''$HOME/.fex-emu/ThunkConfigs/''' isn't a bad choice for consistency.
{
"thunks": {
"libGL-guest.so": ["/lib/x86_64-linux-gnu/libGL.so.1", "/usr/lib/x86_64-linux-gnu/libGL.so.1"],
"libasound-guest.so": ["/lib/x86_64-linux-gnu/libasound.so.2", "/usr/lib/x86_64-linux-gnu/libasound.so.2"],
"libX11-guest.so": ["/lib/x86_64-linux-gnu/libX11.so.6", "/usr/lib/x86_64-linux-gnu/libX11.so.6"],
"libXrender-guest.so": ["/lib/x86_64-linux-gnu/libXrender.so.1", "/usr/lib/x86_64-linux-gnu/libXrender.so.1"],
"libXext-guest.so": ["/lib/x86_64-linux-gnu/libXext.so.6", "/usr/lib/x86_64-linux-gnu/libXext.so.6"],
"libXfixes-guest.so": ["/lib/x86_64-linux-gnu/libXfixes.so.3", "/usr/lib/x86_64-linux-gnu/libXfixes.so.3"]
}
}

Development:Debugging Crash In Thunks

2022-06-22T13:38:24Z

Neobrain: Add development category

[[Category:Development]]

== Once Thunks are enabled ==
* Once you've verified your crash is in a thunk then this becomes quite a bit easier

gdb --args FEXInterpreter /usr/bin/vulkaninfo
Thread 1 "FEXInterpreter" received signal SIGSEGV, Segmentation fault.
0x0000007fc82240f4 in tu_CreateDevice () from /lib/aarch64-linux-gnu/libvulkan_freedreno.so
(gdb) bt
#0 0x0000007fc82240f4 in tu_CreateDevice () from /lib/aarch64-linux-gnu/libvulkan_freedreno.so
#1 0x0000007fc8642b94 in function_traits<decltype (&(fexfn_unpack_libvulkan_freedreno_vkCreateDevice(fexfn_packed_args_libvulkan_freedreno_vkCreateDevice*)))>::result_t fexfn_type_erased_unpack<&(fexfn_unpack_libvulkan_freedreno_vkCreateDevice(fexfn_packed_args_libvulkan_freedreno_vkCreateDevice*))>(void*) () from /usr/lib/fex-emu/HostThunks//libvulkan_freedreno-host.so
#2 0x0000007fe25e4028 in ?? ()
Backtrace stopped: previous frame inner to this frame (corrupt stack?)

* Okay, that's a direct crash in Turnip, let's rebuild Mesa with a debug build and try again
Thread 1 "FEXInterpreter" received signal SIGSEGV, Segmentation fault.
0x0000007fc80bc600 in tu_CreateDevice (physicalDevice=0x5559947aa0, pCreateInfo=0x7ff75fda20, pAllocator=0x0, pDevice=0x7fcbe7b358) at ../src/freedreno/vulkan/tu_device.c:1632
1632 memset(ptr, 0x42, 16);
(gdb) bt
#0 0x0000007fc80bc600 in tu_CreateDevice (physicalDevice=0x5559947aa0, pCreateInfo=0x7ff75fda20, pAllocator=0x0, pDevice=0x7fcbe7b358) at ../src/freedreno/vulkan/tu_device.c:1632
#1 0x0000007fc80eab9c in vk_tramp_CreateDevice (physicalDevice=0x5559947aa0, pCreateInfo=0x7ff75fda20, pAllocator=0x0, pDevice=0x7fcbe7b358) at src/vulkan/runtime/vk_dispatch_trampolines.c:78
#2 0x0000007fc8642b94 in function_traits<decltype (&(fexfn_unpack_libvulkan_freedreno_vkCreateDevice(fexfn_packed_args_libvulkan_freedreno_vkCreateDevice*)))>::result_t fexfn_type_erased_unpack<&(fexfn_unpack_libvulkan_freedreno_vkCreateDevice(fexfn_packed_args_libvulkan_freedreno_vkCreateDevice*))>(void*) () from /usr/lib/fex-emu/HostThunks//libvulkan_freedreno-host.so
#3 0x0000007fe25e426c in ?? ()

* Okay the crash is at.. `tu_device:1632`, let's go and debug that line

1631 void* ptr = (void*)1;
1632 memset(ptr, 0x42, 16);

* Who put that in there?!

Development:gfxreconstruct

2022-06-22T13:38:04Z

Neobrain: Add development category

[[Category:Development]]

= gfxreconstruct =
gfxreconstruct is a Vulkan layer that is useful for capturing a frame in a robust way.

This is usually useful for capturing frames that is more portable than renderdoc.

https://github.com/LunarG/gfxreconstruct

= Setting up gfxreconstruct =
First thing you must do is build gfxreconstruct from source. Either for your host or guest rootfs, depending on if you're using thunks or not.

If building for the x86 rootfs then make sure to install it in to the rootfs after building. This can be done by just copying the install files over afterwards.

== Set up environment variables ==
* VK_INSTANCE_LAYERS=VK_LAYER_LUNARG_gfxreconstruct
* GFXRECON_CAPTURE_TRIGGER=F12
* GFXRECON_CAPTURE_FILE=/tmp/gfxrecon_capture.gfxr

Once you have the environment variables set up you can then launch a vulkan game. There will be some additional output in the terminal to know when the layer is capturing. You can then press F12 in the application and you'll get a capture in tmp after a few seconds. Be careful as gfxreconstruct will overwrite the capture on each press, so you will want to move each capture file.

== Replay ==
gfxrecon-replay can be used to replay the capture which is quite useful.

Development:Renderdoc

2022-06-22T13:37:53Z

Neobrain: Add development category

[[Category:Development]]

= Getting a capture =
== Without Thunks ==
* Ensure Renderdoc is installed in the rootfs

=== OpenGL ===
* Set LD_PRELOAD=librenderdoc.so as an environment variable
* This can be done in FEXConfig or an application profile to limit its reach

=== Vulkan ===
* Ensure the renderdoc layer is installed
* Set ENABLE_VULKAN_RENDERDOC_CAPTURE=1 as an environment variable
* This can be done in FEXConfig or an application profile to limit its reach

=== Alternatively ===
You can use renderdoccmd to launch an application directly.
* renderdoccmd capture -w <program> <args...>
As long as this application is installed in your rootfs then it should work for both OpenGL and Vulkan applications.

=== Capture results ===
* Once you have captured a frame in an application running renderdoc (F12 typically)
* The results will be in /tmp/RenderDoc/ as *.rdc files
* Technically these are meant to be moved over to the qrenderdoc application but you can rip them directly

== With Thunks ==
Likely to be the same as without thunks, but instead of adding environment variables inside of FEX. One needs to do it outside of FEX instead

Development:Debugging Crash

2022-06-22T13:37:42Z

Neobrain:

[[Category:Development]]

== Getting Started ==
* Debug an application with `gdb --args FEXInterpreter <application full path>`
* Under GDB make sure to do `handle SIGBUS SIGILL SIG63 noprint`
** We use signals for various things, check out [[Development:Debugging_FEX_with_Signals|Here]] for more information

== Crash in emulated/JIT code ==
Walking through debugging a simple test application that is crashing.
$ gdb --args FEXInterpreter ./sigsegv_test
Reading symbols from FEXInterpreter...
(gdb) r
Starting program: /usr/bin/FEXInterpreter ./sigsegv_test
[Thread debugging using libthread_db enabled]
Using host libthread_db library "/lib/aarch64-linux-gnu/libthread_db.so.1".
[New Thread 0x7fccb75f30 (LWP 90107)]

Thread 2 "FEXInterpreter" received signal SIGSEGV, Segmentation fault.
[Switching to Thread 0x7fccb75f30 (LWP 90107)]
0x0000007fccfb9ec8 in ?? ()

* Okay, we have a sigsegv. Let's double check that it is JIT code (aka, guest emulated code)
(gdb) disas $pc,+32
Dump of assembler code from 0x7fe26ebc20 to 0x7fe26ebc40:
=> 0x0000007fe26ebc20: stlrb w22, [x21]
0x0000007fe26ebc24: mov x4, #0x0 // #0
0x0000007fe26ebc28: ldr x10, [x20]
0x0000007fe26ebc2c: add x21, x20, #0x8
0x0000007fe26ebc30: mov x22, #0x0 // #0
0x0000007fe26ebc34: strb w22, [x28, #428]
0x0000007fe26ebc38: mov x22, #0x0 // #0
0x0000007fe26ebc3c: strb w22, [x28, #431]
End of assembler dump.
(gdb) info reg x9
x9 0x0 0
* Looks like JIT code, even doing accesses to x28 which is the FEX CPU state
* Code has no backtrace which reinforces this
* Code is doing an atomic store, which reinforces this is FEX emulating the x86 TSO memory model

* Now that we have checked that we are in the JIT code. Where are we in the guest side?
* Let's dump the FEX CPU state information that is directly pointed to in x28 at all times in JIT code.

(gdb) p/x ((FEXCore::Core::CpuStateFrame*)$x28)->State
$3 = {rip = 0x401110, gregs = {0x416eb0, 0x7fe1e3b640, 0xffffffffffffff70, 0x0, 0x7fe1e3bf30, 0x0, 0x416eb0, 0x7fe1e3ae28, 0x0, 0x7fe1e3b640, 0x8, 0x7fe2054cc0, 0x7ff75ff48e, 0x7ff75ff48f, 0x0, 0x7fe163b000}, xmm = {{0x0, 0x0}, {0x0, 0x0}, {0xdeadbeef, 0xbad0dad1} <repeats 14 times>}, es = 0x0, cs = 0x0, ss = 0x0, ds = 0x0, gs = 0x0, fs = 0x7fe1e3b640, flags = {0x0, 0x1, 0x0,
0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x1, 0x0 <repeats 38 times>}, mm = {{0x0, 0x0}, {0x0, 0x0}, {0x0, 0x0}, {0x0, 0x0}, {0x0, 0x0}, {0x0, 0x0}, {0x0, 0x0}, {0x0, 0x0}}, gdt = {{base = 0x0} <repeats 32 times>}, FCW = 0x37f, FTW = 0xffff}

* Looks like our guest RIP is currently `0x401110`
** Consult the `info proc mappings` again
0x401000 0x402000 0x1000 0x1000 {...}/sigsegv_test
* Yep, we are inside our test application
* For a simple test, let's load the application in gdb-multiarch and disassemble where we are
$ gdb-multiarch ./sigsegv_test
Reading symbols from ./sigsegv_test...
(gdb) set disassembly-flavor intel
(gdb) disas 0x401110
Dump of assembler code for function main(int, char**):
0x0000000000401110 <+0>: push rbp
0x0000000000401111 <+1>: mov rbp,rsp
0x0000000000401114 <+4>: mov DWORD PTR [rbp-0x4],0x0
0x000000000040111b <+11>: mov DWORD PTR [rbp-0x8],edi
0x000000000040111e <+14>: mov QWORD PTR [rbp-0x10],rsi
0x0000000000401122 <+18>: mov rax,QWORD PTR [rbp-0x10]
0x0000000000401126 <+22>: movsxd rcx,DWORD PTR [rbp-0x8]
0x000000000040112a <+26>: mov rax,QWORD PTR [rax+rcx*8]
0x000000000040112e <+30>: mov QWORD PTR [rbp-0x18],rax
0x0000000000401132 <+34>: mov rax,QWORD PTR [rbp-0x18]
0x0000000000401136 <+38>: mov BYTE PTR [rax],0x63
0x0000000000401139 <+41>: xor eax,eax
0x000000000040113b <+43>: pop rbp
0x000000000040113c <+44>: ret
End of assembler dump.
(gdb)

* Okay, not super helpful since FEX translates instructions in to blocks, `0x401110` is just our starting address
** It's in this code somewhere, let's change some FEX settings to get a clearer picture
* Set block size to one instruction and disable multiblock
** See the image in FEXConfig to the right
[[File:FEXConfig 1Inst NoMulti.png|thumb]]
* Now rerun our test application and find the new RIP

(gdb) p/x ((FEXCore::Core::CpuStateFrame*)$x28)->State.rip
$2 = 0x401136

* Alright, now we know the RIP is exactly at `0x401136`
* Back in gdb-multiarch
(gdb) disas 0x401136,+1
Dump of assembler code from 0x401136 to 0x401137:
0x0000000000401136 <main(int, char**)+38>: mov BYTE PTR [rax],0x63

* Looks like something in main is storing 0x63 to a nullptr
* In this simple case we can now take a look at the test application's source and find the problem.
** We know the problem is in the first block of main()
** We know the exact instruction that it is at
** We know it's something storing a byte to memory
* For more complex cases it is likely necessary to use reverse engineering tools
** BinaryNinja, Ghidra, IDA, and Hopper are all examples of tools like this

== What to do from here ==
Now it becomes a lot harder. You don't get a typical debugging environment or even clean backtraces.

FEX's gdbserver integration is sorely lacking so you can't even use a remote gdb server connecting to FEX right now.

If you enable thunks you can get better backtraces here. [[Development:Debugging_Crash_In_Thunks|Debugging_Crash_In_Thunks]]

== Attempting to use FEX-Emu's gdbserver implementation ==
'''Here be dragons'''

FEX supports gdbserver as an integration. It's implementation is significantly limited but can still be used for debugging and getting some backtraces.
* Currently hardcodes the port to use as `8086` and if you have multiple gdbserver processes running then it will encounter problems.
* Currently does not follow processes through fork/execve at all. No multiprocess support
** This means you '''must''' only start the process you're caring about debugging
* Currently starts the process paused and will wait until gdb attach before continuing
** No way to start a FEX instance then attach at some later point
* Ctrl-C to stop the FEX process needs to be done twice
** Maybe with a small delay inbetween because gdb needs to fetch a bunch of data on pause
** Known bug, unknown why broken at the moment

FEXLoader -G -- <Application> <Args...>

== Double checking if we are in JIT code ==
(gdb) info reg pc
pc 0x7fccfb9ec8 0x7fccfb9ec8

(gdb) info proc mappings
...
0x7fccfb9000 0x7fcdfb9000 0x1000000 0x0

* Looks like FEX JIT mapping, we start out at 16MB but scale up to 128MB
* Depending on version of FEX we can check the base mapping for a unique string
(gdb) p (char*)0x7fccfb9000
$4 = 0x7fccfb9000 "FEXJIT::Arm64JITCore::"

Development:Debugging FEX with Signals

2022-06-22T13:37:36Z

Neobrain:

[[Category:Development]]

= Signals and you =

FEX uses signals in quite a few ways that may cause debugging to be a bit of a pain. This is necessary for correct operation so it is an evil we must live with.

== Signals that FEX uses ==
* SIGBUS - The AArch64 JIT generates unaligned loadstore instructions then captures and backpatches these when they fault
** This is necessary for emulating x86-TSO on AArch64 and can't be dropped without massive performance loses
* SIGILL - All of the CPU backends generate this fault and is necessary for the emulator to return from a guest signal
** Going in to a signal happens frequently, gdb will usually ignore the entry when it is SIGALRM or SIGUSR*
** We could choose a different signal, but this one can be generated in one instruction which makes it nice to use
** Using a signal to punch back through to the guest return path is necessary since we need to save and restore guest registers when guest signals occur.
* SIG63 - We use this for various ancillary functions
** This signal currently tells a thread that it should stop running, to pause its execution, or to resume from its paused frame
** This is another necessary signal since if we want to safely shutdown the emulator and a thread is in a syscall, there is no way for it to leave without a signal.
** The stop signal specifically will have the CPU backend long jump from the signal handler to the exit stage of the ASM dispatcher for safe shutdown
** This slightly increases performance because then there aren't atomic checks per iteration of the execution loop for if we should stop executing

= Debugging FEX =
Read more here [[Development:Debugging_Crash]]

= Profiling FEX code =
You'll want to enable a style of JIT code mark-up for perf to annotate with
* FEX currently has three options for this
* Depending on what you're trying to profile you will want to enable one of these options
** FEX_GLOBALJITNAMING
*** Useful if you want to throw all of FEX JIT space in to a global-ish namespace
** FEX_LIBRARYJITNAMING
*** Useful if you want to clump FEX JIT space in to library specific code segments
*** Can be nice for finding libraries to thunk
** FEX_BLOCKJITNAMING
*** This is likely the one you want if you're profiling your application's code
*** This will give you hot blocks in `perf top`

Enabling one of these will give you some symbols when capturing FEX with `perf top`. The block naming is useful for seeing hot blocks in program space so if something ends up being particularly slow in FEX, it can get more isolated.
* We don't currently have an option to relate JIT code directly to function symbols in the ELF at this point.

= Debugging the guest application =
When FEX is started with the -G argument it will open a gdbstub socket listening on port 8086. This can be connected to with gdbserver, ida, ghidra, or binary ninja.

This is currently only useful for inspection, gdbstub doesn't yet send guest signals over through gdbstub!

Development:Setting up RootFS

2022-06-22T13:37:28Z

Neobrain:

[[Category:Development]]

= Setting up a RootFS =
'''For AArch64 hosts you are required to have an x86-64 rootfs'''. With or without 32bit compatibility libraries.

This isn't strictly required for testing on x86-64 host environments but is useful for ensuring correct behaviour there.

= Quick Setup with FEXRootFSFetch =
FEX comes with a FEXRootFSFetcher tool that can download an official preconfigured rootfs from the FEX servers.
This tool operates as either a terminal application or a Zenity application depending on if it is launched from terminal or not.

This tool will walk you through selecting a RootFS for FEX:
* If your host OS matches one in the download list then it will recommend that one
* After selecting the RootFS, it will download it
** This downloads to the default location in '''$HOME/.fex-emu/RootFS/'''
* After downloading the RootFS the tool will ask if you want to use the SquashFS file directly or extract it
* It will then ask if you want to set the downloaded RootFS as your default
** This writes to the default configuration in '''$HOME/.fex-emu/Config.json'''

You can skip the rest of the guide if you used FEXRootFSFetcher to download the rootfs.

== Troubleshooting ==
* The squashfs image will fail to extract if you don't have unsquashfs installed
** Install the squashfs-tools package to get this application

* '''SquashFS RootFS file corrupted?'''
** Rerun the tool, select the rootfs from the list, and if the rootfs already exists it will ask to verify
** If verification fails it will ask if you want to redownload
* '''Want to manually check the hash?'''
** Run the rootfs file directly through the tool to get a hash
** '''FEXRootFSFetcher $HOME/.fex-emu/RootFS/Ubuntu_21_10.sqsh'''
** Ubuntu_21_10.sqsh has hash: bf9507fbdbaec2cf
* '''Want to use the Zenity UI but still launch through terminal?'''
** Force Zenity from terminal with '''FEXRootFSFetcher 1> /dev/null'''
* '''What if my Linux distro isn't listed?'''
** Don't worry, select one that is most similar to yours or newest
** This will work in most cases
* '''How can I trust these images?'''
** These images are generated by scripts living in our public repo
** If you want to rebuild them yourself, you are free to do so.
** https://github.com/FEX-Emu/RootFS
* '''I want to add a distro'''
** Follow how our current rootfs is generated and add your own
** Open a pull request to add the rootfs setup
** Contact us on the official FEX-Emu Discord to get the RootFS added to the server.
* '''Where is the raw list of images at?'''
** The image links live on our website at the following link
** https://rootfs.fex-emu.org/file/fex-rootfs/RootFS_links_XXH3.txt
** This file is in a format that our tool understands
** Generally the format is:
*** Distro Name
*** Distro Version
*** Human readable name
*** Raw rootfs link
*** xxhash based file hash

= Making a custom rootfs prerequisites =
An x86-64 PC or FEX-Emu installed for chrooting

= Getting Started =
The easiest way to get started is to just download a prebuilt x86-64 rootfs from your favourite distribution and extract it to where you want.
* Ubuntu - http://cdimage.ubuntu.com/ubuntu-base/releases/
* Debian Debootstrap guide - https://wiki.debian.org/Debootstrap
** Debootstrap is more involved and not explained here.

Using chroot you can then enter the rootfs and install applications that you desire for testing purposes.
It's easiest to set this up on an x86-64 host and then copy it to an AArch64 device.

== Cross architecture chroot ==
'''You need to have binfmt_misc setup for x86-64 support on your host system.'''
* ls /proc/sys/fs/binfmt_misc
** If there are files in this folder then your host supports binfmt_misc
* Once you have binfmt_misc running and FEX-Emu installed in to it, you can chroot in to an x86-64 rootfs

== Chrooting in to rootfs with FEX-Emu Rootfs ==
* If you're using a FEX-Emu provided rootfs then we provide two scripts at the start to handle this
** '''unbreak_chroot.sh''' - This sets up the rootfs and chroots inside of it
** '''break_chroot.sh''' - This sets up the rootfs for using with FEX generally.
*** '''Make sure to run this after leaving the chroot!'''
** The remaining steps in this guide can be skipped

== Chrooting in to custom rootfs with FEX-Emu ==
* If you're setting up your own rootfs then you need to follow the binding steps that are following.
** We have a script that if you know what you're doing you can cherry pick parts from [https://github.com/FEX-Emu/RootFS/blob/main/unbreak_chroot.sh here]

= Additional TmpFS folders =
Depending on what you're doing in the chroot, you may need or want some tmpfs folders that applications are expecting

* sudo mount -t proc /proc ${ROOTFS}/proc/
* sudo mount --rbind --make-rslave /sys ${ROOTFS}/sys/
* sudo mount --rbind --make-rslave /dev ${ROOTFS}/dev/
* sudo mount --rbind --make-rslave /dev/pts ${ROOTFS}/dev/pts/
* sudo mount --rbind /lib/ld-linux-aarch64.so.1 ${ROOTFS}/lib/ld-linux-aarch64.so.1
* sudo mount --rbind /lib/aarch64-linux-gnu ${ROOTFS}/lib/aarch64-linux-gnu

Then of course, make sure to unmount these partitions when you're done

* sudo umount ${ROOTFS}/proc/
* sudo umount -R ${ROOTFS}/sys/
* sudo umount -R ${ROOTFS}/dev/pts/
* sudo umount -R ${ROOTFS}/dev/
* sudo umount -R ${ROOTFS}/lib/ld-linux-aarch64.so.1
* sudo umount -R ${ROOTFS}/lib/aarch64-linux-gnu

= Using the rootfs with FEX =
If you're using FEXLoader then you can pass the full path of the rootfs with the -R option.
Additionally you can put the rootfs folder inside of $HOME/.fex-emu/RootFS/<RootFS name>/ and FEX supports a "named" rootfs option
* eq: -R Ubuntu_2104 will use rootfs $HOME/.fex-emu/RootfS/Ubuntu_2104
* Read the Fex man page with man fex for more information

== Setting RootFS with FEXConfig ==
[[File:FEXConfig Load Default.png|frameless|Load the default options on first run]]
On the first run of FEXConfig, select the "Load Default Options" configuration.
* Subsequent runs should select "Open Default"

[[File:FEXConfig RootFS Selection.png|frameless|Select the RootFS from the FEXConfig location]]

FEXConfig will autopopulate the rootfs list with rootfs folders that exist in $HOME/.fex-emu/RootFS/

You can select one from the list or provide a full path in the RootFS: line to set it manually.

''' Make sure to select the File->Save option after the fact to save'''

= Additional Steps for building a custom rootfs =
== Running under WSL1 ==
/etc/hosts and /etc/resolv.conf change on the host side. You'll need to copy those over to the rootfs if you want network functionality to work.

== Ubuntu packages to install ==
Sample packages to install in the rootfs
* apt-get install pulseaudio libgles1 libgles2 libglx-mesa0 libgl1-mesa-dri libgl1-mesa-glx mesa-utils mesa-utils-extra
Pulseaudio specifically is required so pulseaudio will work when talking to the host pulseaudio

== File deletion ==
Some files need to be deleted from the rootfs otherwise some applications will fail to work.

Note that deleting these files '''WILL''' break the rootfs and you won't be able to chroot in to it anymore.

A likely better route would be a script that moves these files and folders then brings them back when you want to chroot

=== Files to remove ===
* ${ROOTFS}/etc/hosts
* ${ROOTFS}/etc/resolv.conf
* ${ROOTFS}/etc/timezone
* ${ROOTFS}/etc/localtime
* ${ROOTFS}/etc/passwd

=== Folders to remove ===
* ${ROOTFS}/boot
* ${ROOTFS}/dev
* ${ROOTFS}/home
* ${ROOTFS}/media
* ${ROOTFS}/mnt
* ${ROOTFS}/proc
* ${ROOTFS}/root
* ${ROOTFS}/srv
* ${ROOTFS}/tmp
* ${ROOTFS}/var/cache/apt
* ${ROOTFS}/var/lib/apt
* ${ROOTFS}/sys
* ${ROOTFS}/opt

== Creating a Squashfs rootfs from custom rootfs ==
SquashFS is a great way to do binary distribution of the rootfs. If you have created a rootfs and you want to roll it in to a squashfs like the FEX-Emu provided images then this is information on how to do that.

'''SquashFS is currently only recommended for binary distribution. User should unpack it to not get dangling processes and mounts.'''
* Offers tooling to mount in userspace
* Supports ZSTD compression
* Is a readonly partition that matches FEX's use case

=== Tools that FEX needs to run squashfs ===
* squashfuse
* fusermount

=== Making the squashfs image ===
Making the squashfs image is very straightforward
* mksquashfs ${ROOTFS}/ Squashfs.sqsh -comp zstd

This creates a file called Squashfs.sqsh that contains the original rootfs that we generated above.

=== Using the squashfs image in FEX ===
* Copy the resulting sqsh file in to ${XDG_DATA_HOME}/.fex-emu/RootFS/
** If ${XDG_DATA_HOME} doesn't exist then that will be replaced with ${HOME}
* Use FEXConfig to select the sqsh file as a named rootfs
* or pass the name in to the FEX_ROOTFS argument in the various ways FEX supports
* or the absolute path if you don't want it in the FEX RootFS folder

=== Extracting the squashfs ===
If you don't want to use squashfs for whatever reason then you can extract it in to the FEX RootFS folder and use the directory directly
* unsquashfs -f -d NewRootFS/ Squashfs.sqsh

=== If you want to install packages in the squashfs ===
The squashfs image is immutable and can't be modified.
* Extract the squashfs first
* Change the working directory to the rootfs
* execute `unbreak_chroot.sh`
* `apt-get update`
* `apt-get install <Whatever you want>`
* `exit` to leave the chroot
* `break_chroot.sh`
* Use the new extracted chroot directly, or follow the previous steps to recompress it to a squashfs format.

=== Known problems with squashfs ===
'''FEX automatically mounts your squashfs file and can leave dangling squashfs mounts around in some instances.'''
* '''This is fairly rare but it can occur. Whenever an instance of FEX is run, it registers with the FEXMountDaemon'''
** FEXMountDaemon then watches these instances of FEX to ensure they have exited before unmounting
** There is roughly a ten second grace period of no instances of FEX running before unmounting
** If an instance of FEX starts running then it will reuse the mount partition and reset the timeout
* To get rid of these dangling mounts if they do manage to manifest, use the following commands
** fusermount -u /tmp/.FEXMount* ; rmdir /tmp/.FEXMount*
** These don't require root to unmount
* In some cases the unmounting can end up returning busy status and not unmount
** Check the output of `mount` for any /tmp/.FEXMount folders
** This is due to either FEXMountDaemon or squashfuse still running in the background
** Can force the application to exit
** step 1: kill `pidof squashfuse`
** This will attempt to send SIGTERM to all squashfuse applications, which will unmount the partition on terminate
** Failing that, you can try to be more abusive
** kill -9 `pidof FEXMountDaemon`
** This will forcibly terminate FEXMountDaemon which should also take out the child mounted process
* If any instances of the mount still exists then you may need to kill more FEX processes.
** Any FEXInterpreter process will end up keeping the folder mounted and keep it busy

'''strace breaks squashfs'''
* If you run FEX with `strace -f` then the process that mounts your rootfs loses its setuid bit.
** This means you then can't mount the rootFS
* The only workaround is to not use a squashfs in this instance.
* Alternatively you can spin up a FEXBash instance to force squashfs to be mounted. Then strace a different FEX instance which will use the mounted squashfs location.