Development:Debugging Crash

Getting Started

Debug an application with `gdb --args FEXInterpreter <application full path>`
Under GDB make sure to do `handle SIGBUS SIGILL SIG63 noprint`
- We use signals for various things, check out Here for more information

Crash in emulated/JIT code

Walking through debugging a simple test application that is crashing.

 $ gdb --args FEXInterpreter ./sigsegv_test
 Reading symbols from FEXInterpreter...
 (gdb) r
 Starting program: /usr/bin/FEXInterpreter ./sigsegv_test
 [Thread debugging using libthread_db enabled]
 Using host libthread_db library "/lib/aarch64-linux-gnu/libthread_db.so.1".
 [New Thread 0x7fccb75f30 (LWP 90107)]
 
 Thread 2 "FEXInterpreter" received signal SIGSEGV, Segmentation fault.
 [Switching to Thread 0x7fccb75f30 (LWP 90107)]
 0x0000007fccfb9ec8 in ?? ()

Okay, we have a sigsegv. Let's double check that it is JIT code (aka, guest emulated code)

 (gdb) disas $pc,+32
 Dump of assembler code from 0x7fe26ebc20 to 0x7fe26ebc40:
 => 0x0000007fe26ebc20:  stlrb   w22, [x21]
    0x0000007fe26ebc24:  mov     x4, #0x0                        // #0
    0x0000007fe26ebc28:  ldr     x10, [x20]
    0x0000007fe26ebc2c:  add     x21, x20, #0x8
    0x0000007fe26ebc30:  mov     x22, #0x0                       // #0
    0x0000007fe26ebc34:  strb    w22, [x28, #428]
    0x0000007fe26ebc38:  mov     x22, #0x0                       // #0
    0x0000007fe26ebc3c:  strb    w22, [x28, #431]
 End of assembler dump.
 (gdb) info reg x9
 x9             0x0                 0

Looks like JIT code, even doing accesses to x28 which is the FEX CPU state
Code has no backtrace which reinforces this
Code is doing an atomic store, which reinforces this is FEX emulating the x86 TSO memory model

Now that we have checked that we are in the JIT code. Where are we in the guest side?
Let's dump the FEX CPU state information that is directly pointed to in x28 at all times in JIT code.

 (gdb) p/x ((FEXCore::Core::CpuStateFrame*)$x28)->State
 $3 = {rip = 0x401110, gregs = {0x416eb0, 0x7fe1e3b640, 0xffffffffffffff70, 0x0, 0x7fe1e3bf30, 0x0, 0x416eb0, 0x7fe1e3ae28, 0x0, 0x7fe1e3b640, 0x8, 0x7fe2054cc0, 0x7ff75ff48e, 0x7ff75ff48f, 0x0, 0x7fe163b000}, xmm = {{0x0, 0x0}, {0x0, 0x0}, {0xdeadbeef, 0xbad0dad1} <repeats 14 times>}, es = 0x0, cs = 0x0, ss = 0x0, ds = 0x0, gs = 0x0, fs = 0x7fe1e3b640, flags = {0x0, 0x1, 0x0,
   0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x1, 0x0 <repeats 38 times>}, mm = {{0x0, 0x0}, {0x0, 0x0}, {0x0, 0x0}, {0x0, 0x0}, {0x0, 0x0}, {0x0, 0x0}, {0x0, 0x0}, {0x0, 0x0}}, gdt = {{base = 0x0} <repeats 32 times>}, FCW = 0x37f, FTW = 0xffff}

Looks like our guest RIP is currently `0x401110`
- Consult the `info proc mappings` again

 0x401000           0x402000     0x1000     0x1000 {...}/sigsegv_test

Yep, we are inside our test application
For a simple test, let's load the application in gdb-multiarch and disassemble where we are

 $ gdb-multiarch ./sigsegv_test
 Reading symbols from ./sigsegv_test...
 (gdb) set disassembly-flavor intel
 (gdb) disas 0x401110
 Dump of assembler code for function main(int, char**):
    0x0000000000401110 <+0>:     push   rbp
    0x0000000000401111 <+1>:     mov    rbp,rsp
    0x0000000000401114 <+4>:     mov    DWORD PTR [rbp-0x4],0x0
    0x000000000040111b <+11>:    mov    DWORD PTR [rbp-0x8],edi
    0x000000000040111e <+14>:    mov    QWORD PTR [rbp-0x10],rsi
    0x0000000000401122 <+18>:    mov    rax,QWORD PTR [rbp-0x10]
    0x0000000000401126 <+22>:    movsxd rcx,DWORD PTR [rbp-0x8]
    0x000000000040112a <+26>:    mov    rax,QWORD PTR [rax+rcx*8]
    0x000000000040112e <+30>:    mov    QWORD PTR [rbp-0x18],rax
    0x0000000000401132 <+34>:    mov    rax,QWORD PTR [rbp-0x18]
    0x0000000000401136 <+38>:    mov    BYTE PTR [rax],0x63
    0x0000000000401139 <+41>:    xor    eax,eax
    0x000000000040113b <+43>:    pop    rbp
    0x000000000040113c <+44>:    ret
 End of assembler dump.
 (gdb)

Okay, not super helpful since FEX translates instructions in to blocks, `0x401110` is just our starting address
- It's in this code somewhere, let's change some FEX settings to get a clearer picture
Set block size to one instruction and disable multiblock
- See the image in FEXConfig to the right

Now rerun our test application and find the new RIP

 (gdb) p/x ((FEXCore::Core::CpuStateFrame*)$x28)->State.rip
 $2 = 0x401136

Alright, now we know the RIP is exactly at `0x401136`
Back in gdb-multiarch

 (gdb) disas 0x401136,+1
 Dump of assembler code from 0x401136 to 0x401137:
    0x0000000000401136 <main(int, char**)+38>:   mov    BYTE PTR [rax],0x63

Looks like something in main is storing 0x63 to a nullptr
In this simple case we can now take a look at the test application's source and find the problem.
- We know the problem is in the first block of main()
- We know the exact instruction that it is at
- We know it's something storing a byte to memory
For more complex cases it is likely necessary to use reverse engineering tools
- BinaryNinja, Ghidra, IDA, and Hopper are all examples of tools like this

What to do from here

Now it becomes a lot harder. You don't get a typical debugging environment or even clean backtraces.

FEX's gdbserver integration is sorely lacking so you can't even use a remote gdb server connecting to FEX right now.

If you enable thunks you can get better backtraces here. Debugging_Crash_In_Thunks

Attempting to use FEX-Emu's gdbserver implementation

Here be dragons

FEX supports gdbserver as an integration. It's implementation is significantly limited but can still be used for debugging and getting some backtraces.

Currently hardcodes the port to use as `8086` and if you have multiple gdbserver processes running then it will encounter problems.
Currently does not follow processes through fork/execve at all. No multiprocess support
- This means you must only start the process you're caring about debugging
Currently starts the process paused and will wait until gdb attach before continuing
- No way to start a FEX instance then attach at some later point
Ctrl-C to stop the FEX process needs to be done twice
- Maybe with a small delay inbetween because gdb needs to fetch a bunch of data on pause
- Known bug, unknown why broken at the moment

 FEXLoader -G -- <Application> <Args...>

Double checking if we are in JIT code

 (gdb) info reg pc
 pc             0x7fccfb9ec8        0x7fccfb9ec8
 
 (gdb) info proc mappings
 ...
 0x7fccfb9000       0x7fcdfb9000  0x1000000        0x0

Looks like FEX JIT mapping, we start out at 16MB but scale up to 128MB
Depending on version of FEX we can check the base mapping for a unique string

 (gdb) p (char*)0x7fccfb9000
 $4 = 0x7fccfb9000 "FEXJIT::Arm64JITCore::"

Getting RIP of current code block

FEX sets up an address in our CPU context to get some debug data out.

Currently this isn't exposed in a way that a debugger can see other than manually typing out gdb commands

 p/x *(uint64_t*)($x28+184) = inline block header ptr
 p/x *(uint32_t*)(*(uint64_t*)($x28+184)) = OffsetToBlockTail
 p/x *(uint64_t*)($x28+184) + *(uint32_t*)(*(uint64_t*)($x28+184))

 p/x *(unsigned long long*)((*(unsigned long long*)($x28+184) + *(unsigned int*)(*(unsigned long long*)($x28+184)))+8) = RIP of block

 disas *(unsigned long long*)($x28+184),+*(unsigned long long*)(*(unsigned long long*)($x28+184) + *(unsigned int*)(*(unsigned long long*)($x28+184))) = Disassemble this block of code.

While hard to decipher here is basically what is happening. - x28 is the CPU register that FEX keeps in the JIT for context accesses - offset 184 is the offset of the `InlineJITBlockHeader` member inside of that context. - As long as FEX is in a JIT block that offset will be valid to point to the current RIP that the block is operating on.

Getting the stack can also be very useful

 x/64wx ((FEXCore::Core::CPUState*)$x28)->gregs[FEXCore::X86State::REG_RSP]

Doing raw pointer math here means that this works even when gdb fails to find symbols for the CPUState object, which happens very frequently for some reason.

Getting RIP of current code block - x86 host