Interesting ARM64 Platforms

This is a brain dump of "actively" interesting ARM64 platforms, mostly from Sonicadvance1. It's hard to remember which platforms are actively interesting for FEX development and usage. Definitely some bias in here.

Platforms that can accept a GPU

"Consumer" platforms

Jetson Orin

Behaviour

• PCIe can support a full Radeon GPU without issue
  • Only 55W on the PCIe bus, so a bit underpowered.
  • Needs external power for larger GPUs

Problems

• Has bugged atomics on its Cortex-A78AE cores, so performance isn't good
  • Errata 1951502
• Needs a kernel recompile on upstream Linux to enable the amdgpu kernel module
• Setting up can be a bit of a pain, but boots upstream kernel

Jetson Thor - Upcoming

Not yet shipped. Will begin shipping in 2025

Behaviour

• Going to have Neoverse-V3AE cores, so will be significantly more powerful than Orin

Snapdragon X Elite

Behaviour

• Can have three Thunderbolt ports
• SoC also has 8x PCIe lane PHY
  • Theoretically can accept GPU on at least the 8x PCIe PHY
• Reasonable CPU performance, if a little underwhelming
• Lots of laptop devices to choose from

Problems

• Linux doesn't yet support Thunderbolt on this platform
  • GPU on Thunderbolt might not work
• No platform with 8x PCIe lanes exposed
  • Snapdragon dev kit has a PCIe slot inside of it that isn't wired up
  • Wouldn't work without device tree changes anyway
  • Maybe someone will eventually ship hardware with these PCIe lanes exposed
• Snapdragon Dev kit basically doesn't exist

Server platforms

All of these are very expensive and not really worth buying as a consumer

NVIDIA Grace

Grace-Hopper and Grace-Grace platform has a large number of Neoverse-V2 cores. Very expensive platform, and Grace-Grace not actually available.

Problems

• Really expensive. $40k to $100k+
• Grace-Grace 144 core not really available

AmpereOne

Only just now becoming available in October 2024.

Behaviour

• A96-37X SKU would likely be great in a desktop platform, but not available
• Custom CPU cores, not as high performance as Grace or very latest consumer platforms

Problems

• Untested but PCIe should be fixed

Platforms that can not accept a GPU

Consumer platforms

All Apple Silicon devices

Behaviour

• Highest performing ARMv8/ARMv9 CPUs today
• Parallels/UTM VM available in MacOS for all devices
• Asahi Linux supported for M1/M2 Pro/Max/Ultra for native Linux
  • But breaks FEX because of 16KB pages
  • Big work in progress to get a VM running FEX with 4k pages and virtio-gpu

Problems

• Thunderbolt can't accept GPUs
  • This is due to Dart/IOMMU problems and memory mapping problems.
  • Asahi linux has some documentation on this failure
• Mac Pro PCIe slots have the same issue, so nothing solved on that platform

Server platforms

Ampere Altra

Behaviour

• Has some desktop-like systems created by Avantek.
  • Overpriced a bit, but volumes are low.

Problems

• Has bugged PCIe fabric that requires kernel workarounds to get GPUs working on it
  • Doesn't fully solve the problem, latest Radeons still don't work
• Old CPU cores based on Neoverse-N1
• Not recommended unless you need a lot of cores

Misc Platforms

Raspberry Pi 4/5

• Low-cost
• Low-performance
• Not really worth using
• GPU doesn't have enough features for DXVK/VKD3D

RK3588 boards

• Old CPU cores, Cortex-A76
• Mali GPU which doesn't support enough features for DXVK/VKD3D
  • Mesa drivers don't have enough features.