Initial Minimal Fedora Image for Raspberry Pi 5

I know this has been much awaited because of all the queries on the various forums and direct to me so here we are the first Fedora image that can run a native userspace with a Fedora kernel with some enablement patches.

This image is far from complete and is NOT yet suitable for desktop UXes and related usecases that require a display.

So what works, what doesn’t, and how can you get started?

The things that are working and tested:

  • The original RPi5 rev c0 SoCs: the older 4Gb/8Gb variants
  • Serial console
  • Late boot HDMI0 display output (IE once the kernel has started) via simple DRM/FB
  • The compute Subsystems (CPUs etc) of both SoC revs
  • The micro SD slot – the only supported OS disk ATM
  • Wired ethernet port
  • Wireless network interface
  • USB ports (NOT for OS disks)

The things that don’t work:

  • The new RPi5 rev d0 SoCs: the 2Gb/16Gb, newer 1.1 Rev 4/8Gb variants, the RPi500, CM5 – the kernel will crash on boot
  • Early boot display output
  • Accelerated GPU
  • Basically everything else

What might work:

  • PCIe for HATs via the add-on HATs and related products including NVME. Not currently for boot/root.

For getting started you will need to have a serial console, ATM booting off anything other than the micro SD card won’t work.

We will eventually support booting off USB/NVME etc and display output as well as other HW but unfortunately we’re not there yet. I feel with USB/eth/wifi support the device is now at a point where it’s actually usable for a lot of Fedora users so I decided it’s time to expand this to more people than just me šŸ˜€

Note I don’t have the spare cycles to assist with debugging any issues that have been listed above as explicitly not working, I want to spend my time on making the device more usable (SORRY, but I do this in my personal spare time). I will update when any of this changes.

You can get the image from here and get started in the usual way with either DD or arm-image-installer (update the storage media name):

arm-image-installer --resizefs --target=none --media=/dev/XXX --image=rpi5-250907-fedora-43-minimal-raw-xz-aarch64.raw.xz

I am working to get more HW support enabled, my focus is on the d0 rev boards (2Gb, 16Gb, 500, CM5 and newer 4/8Gb) and PCIe, from there I will look at what else is possible with my available time. These enhancements will land in new kernel updates to my copr repo, or fixes into Fedora proper. which will arrive by either new published images or kernel updates. I will provide updates when particularly useful milestones are passed and new things start to work.

Bug reports are of course welcome but not for desktopsm or RFEs for other HW enablement, especially things I’ve already stated above, I am working to get more features but I am limited to what I can do because of available time, upstream work and access to HW docs so please be patient as I am doing this in my spare time! Of course being downstream of Fedora please don’t file bugs there unless they are a general problem with userspace. The only thing that’s not vanilla Fedora currently is the kernel. I have an original Pi 5 8Gb and a Pi 5 rev d0 model I am testing with, everything else can be considered untested so YMMV!

Building rust UEFI apps on Fedora

As everyone probably knows rust is considered a great language for secure programming and hence a lot of people are looking at it for everything from low level firmware to GPU drivers. In a similar vein it can already be to build UEFI applications.

Upstream in U-Boot we’ve been adding support for UEFI HTTP and HTTPs Boot and it’s now stable enough I am looking to enable this in Fedora. While I was testing the features on various bits of hardware I wanted a small UEFI app I could pull across a network quickly and easily from a web server for testing devices.

Of course adding in display testing as well would be nice…. so enter UEFI nyan cat for a bit of fun!

Thankfully the Fedora rust toolchain already has the UEFI targets built (aarch64 and x86_64) so you don’t need to mess with third party toolchains or repos, it works the same for both targets, just substitute the one you want in the example where necessary.

I did most of this in a container:

$ podman pull fedora
$ podman run --name=nyan_cat --rm -it fedora /bin/bash
# dnf install -y git-core cargo rust-std-static-aarch64-unknown-uefi
# git clone https://github.com/diekmann/uefi_nyan_80x25.git
# cd uefi_nyan_80x25/nyan/
# cargo build --release --target aarch64-unknown-uefi
# ls target/aarch64-unknown-uefi/release/nyan.efi
target/aarch64-unknown-uefi/release/nyan.efi

From outside the container then copy the binary, and add it to the EFI boot menu:

$ podman cp nyan_cat:/uefi_nyan_80x25/nyan/target/aarch64-unknown-uefi/release/nyan.efi ~/
$ sudo mkdir /boot/efi/EFI/nyan/
$ sudo cp ~/nyan.efi /boot/efi/EFI/nyan/nyan-a64.efi
$ sudo efibootmgr --create --disk /dev/nvme0n1 --part 1 --label "nyan" --loader \\EFI\\nyan\\nyan-a64.efi

Reboot and sit back and enjoy some UEFI Nyan Cat!

Any Linux distro on NVIDIA Jetson Orin with JetPack 6

NVIDIA has just released the Jetpack 6 for the NVIDIA Jetson Orin hardware. The thing that is most exciting about this release is they finally support the ability to use upstream kernels and other Linux distributions. This means you can start to use both RHEL (9.3 and later) and Fedora on the Jetson Orin hardware! This has been a LONG time coming, something I’ve been involved with for 5 years!

So while this is a developer preview, AKA public Beta, it’s still very usable and for people that are interested in using other Linux distributions now is the time to get stuck in. Like all things it’s not perfect and there’s still work to be done, but many hands do make light work!

You start by downloading the BSP from there you can follow the following instructions and you should end up with a device you can easily install Fedora 39+ or RHEL 9.3 or other distros with the appropriate bits enabled.

To flash the firmware you need to follow the Orin AGX guide for recovery and cabling, for Orin NX/nano you need to use the HW pins near the mSD card, to put the device in recovery mode and cabling then do the following for Orin AGX:

$ sudo dnf install -y libxml2-utils lz4 usbutils xxd
$ tar zvf Jetson_Linux_R36.2.0_aarch64.tbz2
$ cd Linux_for_Tegra/
$ lsusb|grep -i nv
Bus 003 Device 044: ID 0955:7045 NVIDIA Corp. [unknown]
Bus 003 Device 045: ID 0955:7023 NVIDIA Corp. [unknown]
$ sudo ./flash.sh p3737-0000-p3701-0000-qspi external
removed a lot of output
*** The target generic has been flashed successfully. ***
Make the target filesystem available to the device and reset the board to boot from external external.
$

The command for other Orin devices such as NX and Nano will be similar, you’ll just have to swap the p3737 variable, eg for Orin Nano use: sudo ./flash.sh p3737-0000-p3701-0000-qspi external.

Once the flash completes the device will reboot and you will be able to use the usual mechanisms to install your OS, whether the RHEL or Fedora installers or a Fedora Arm image. I’ve tested running OSes off both the microSD and a NVME card, plus installing off USB, the DisplayPort output should work in EFI console mode. The firmware is based upon the widely known TianoCore/EDK2 so the firmware interface should be straight forward. For those that may need a serial console if it’s not automatically detected you can use console=ttyAMA0,115200, This runs off the microUSB port on /dev/ttyACM2 on the host device.

For hardware vendors that have hardware based on the NVIDIA Orin hardware they will be able to adopt and make this available to their customers that may wish to run distributions other than L4T. If they are unsure feel free to reach out to me in the usual locations.

Getting started with OpenCL using mesa/rusticl

Mesa, the open source low level graphics stack, has featured support for Open Compute Language (OpenCL) for some time via a front end called Clover. The problem was that the GPUs that it supported were limited, it didn’t have Image support, and wasn’t really under active development. Around a year ago Karol Herbst filed a merge request adding rusticl to Mesa 22.3 release, and soon after that I enabled it optionally in Fedora.. What is rusticl? It’s a OpenCL API implementation written in rust for Mesa, it will eventually replace clover. One advantage it has is image support from the outset and also works on much wider range of OpenCL capable GPUs, including some ARM GPUs, and the support is actively improving all the time.

In the year since it landed in a stable mesa release rusticl keeps evolving, faster, more HW support, more features, less crashes. I’ve tinkered with it as I’ve had spare time on the weekends and evenings as well as trying to work out details of how you’d use it to run higher level stacks.

As of writing it works with following gallium drivers: iris (Intel), nouveau (Nvidia), radeonsi/r600 (AMD ATI), and panfrost (Arm MALI). There’s other drivers that are various stages of development but are not yet upstream.

So let’s get the basics up and running on Fedora if you have supported hardware. First install the core software stack:

$ sudo dnf install -y mesa-libOpenCL mesa-dri-drivers spirv-llvm-translator spirv-tools-libs clinfo clpeak

Next we run clinfo and clpeak with required parameters, from the driver names above (in this case an Intel laptop), to enable rusticl. The two commands output a lot of information so I’m not going to post them here but the output shows OpenCL running and some details about what’s supported:

$ RUSTICL_ENABLE=iris clinfo --list
WARNING: OpenCL support via iris+clover is incomplete.
For a complete and conformant OpenCL implementation, use
https://github.com/intel/compute-runtime instead
Platform #0: rusticl
 `-- Device #0: Mesa Intel(R) Xe Graphics (TGL GT2)
Platform #1: Portable Computing Language
 `-- Device #0: cpu-11th Gen Intel(R) Core(TM) i7-1185G7 @ 3.00GHz
Platform #2: Clover
$ RUSTICL_ENABLE=iris clinfo
output a lot of details
$ RUSTICL_ENABLE=iris clpeak
output a lot of details

This gets up the basic pieces up and running for OpenCL, there’s of course more to do and not all use cases are complete. Eventually we won’t need to have the environment variables to enable rusticl. The details of drivers and other features enabled by environment variables are documented here. I plan to do some more posts as follow ups to build on this basis.

Using iwd for WiFi in Fedora

Fedora uses NetworkManager as the default for managing all the various different types of network. Underneath NetworkManager uses wpa_supplicant to connect to 802.11 based, AKA WiFi, wireless networks. There is an alternative called iwd which in a number of use cases works better, it also has the advantage that it offloads a bunch of things like crypto to the kernel interfaces which makes it smaller, and it’s under active development. iwd has a nice straight forward interface as well as being supported as an alternative NetworkManager so it just works in Fedora whether via nmcli or your chosen desktop environment.

So how do you make use of it in Fedora? Well it’s been packaged and supported for some time so it’s quite straight forward and there’s two ways to use it with NetworkManager. You can either swap it out for wpa_supplicant, or they can be installed side by side and you can change the NetworkManager default in a config to enable easy testing/swapping.

Option 1 (side by side):

sudo dnf install -y iwd
sudo cat >> /etc/NetworkManager/conf.d/iwd.conf << EOF
[device]
wifi.backend=iwd
EOF
sudo systemctl restart NetworkManager

Option 2 (swap):

sudo dnf swap -y wpa_supplicant iwd
sudo systemctl restart NetworkManager

You can now connect to WiFi networks a before with NetworkManager. Note it loses exciting configured WiFi networks.

Using fwupdmgr to update NVME firmware

The fabulous fwupdmgr provides the ability to easily update firmware that is published to Linux Vendor Firmware Service (LVFS) but it can also be used to apply updates that aren’t necessarily in LVFS. One type of firmware that it supports updating is NVME firmware, that’s basically any NMVE, because the standard specifies a standardised mechanism for updating the firmware on all NVME devices.

I had a need to update a NVME firmware in an aarch64 device to see if it fixed an issue I was seeing. The Crucial P2 supported options were of course x86 only. The ISO download actually contained a little LinuxOS in an initrd on the .iso. The advice from Richard the fwupd technical lead was to “Look for a ~4mb high entropy blob” so mounting it up, I mounted the iso, extracted the initrd, and then used fwupdmmgr to apply the new firmware.

Find the NVME and check the firmware version:

$ cat /sys/class/nvme/nvme0/firmware_rev 
P2CR010 

So once I’d downloaded the update file I did the following to extract and update the firmware. Note I did this all as root, you can do most of it as non root.

# unzip iso_p2cr012.zip
# mount -o loop iso_p2cr012.iso /mnt/
# mkdir ~/tmp
# cp /mnt/boot/corepure64.gz tmp/
# cd tmp
# gunzip corepure64.gz
# cpio -iv < corepure64
# fwupdtool install-blob opt/firmware/P2CR012/1.bin
Loading…                 [-                                      ]
Loading…                 [-                                      ]
Choose a device:
0.	Cancel
1.	71b677ca0f1bc2c5b804fa1d59e52064ce589293 (CT250P2SSD8)
2.	2270d251f7c1dc37a29a2aa720a566aa0fa0ecde (spi1.0)
1
Waiting…                 [************************************** ] Less than one minute remaining…
An update requires a reboot to complete. Restart now? [y|N]: y

And away it goes, a reboot later and did it work?

$ cat /sys/class/nvme/nvme0/firmware_rev 
P2CR012

YES!!

HW video offload on Fedora Arm

There’s been the starting pieces of hardware video offload with the stateless engine support for some time and it now supports at least H264/HEVC/VP8/VP9/mepg2 decode offload depending on the hardware capabilities. The problem has been support for software/userspace has taken longer then the initial drivers but now that’s catching up now with gstreamer support landing in 2020 and with apps like clapper now using it. I’ve been meaning to play with this and work out how to make it work in Fedora as it’s useful for devices based on the AllWinner/Rockchip/NXP i.MX8 devices like the Pine64 laptops/phones plus a bunch of other devices, even NVIDIA Jetson devices should work before long.

You’ll need to ensure you have gstreamer1-plugins-bad-free installed and the video application I was testing with is clapper:

$ sudo dnf install -y gstreamer1-plugins-bad-free clapper

Seeing what hardware offload is supported:

$ gst-inspect-1.0 v4l2codecs
Plugin Details:
  Name                     v4l2codecs
  Description              V4L2 CODEC Accelerators plugin
  Filename                 /usr/lib64/gstreamer-1.0/libgstv4l2codecs.so
  Version                  1.20.0
  License                  LGPL
  Source module            gst-plugins-bad
  Source release date      2022-02-03
  Binary package           Fedora GStreamer-plugins-bad package
  Origin URL               http://download.fedoraproject.org

  v4l2slh264dec: V4L2 Stateless H.264 Video Decoder
  v4l2slmpeg2dec: V4L2 Stateless Mpeg2 Video Decoder
  v4l2slvp8alphadecodebin: VP8 Alpha Decoder
  v4l2slvp8dec: V4L2 Stateless VP8 Video Decoder
  v4l2slvp9alphadecodebin: VP9 Alpha Decoder
  v4l2slvp9dec: V4L2 Stateless VP9 Video Decoder

  6 features:
  +-- 6 elements

Finally in Clapper you need to enable playbin3 option, I also enabled Pipewire audio support:
Clapper Preferences

We will also be enabling decode support in Chromium and Chromium freeworld before long, there’s a little more work to do here, but as usual once it lands it’ll all just start to work in Chromium too!

Fedora on NVIDIA Jetson Xavier

The last two years or so I’ve been working with NVIDIA on general distro support including UEFI and ACPI for their Jetson Xavier platforms. Their Xavier platform, except a few quirks, are mostly SystemReady-ES compliant, so having a SBBR compliant firmware goes quite some way to having a widely available, relatively affordable, platform that “just works” for the arm ecosystem. I was very excited to finally have NVIDIA finally release the first version in March this year. This firmware is a standard UEFI firmware based on the open source TianoCore/EDK2 reference firmware, it allows booting in either ACPI or Device-Tree mode and supports all the basic things needed. The ACPI mode is not as fully featured as the Device-Tree mode as yet. In ACPI you get compute (cpu/memory/virt etc), PCIe, USB, network, which is just fine if you’re just looking for standard server or for testing a SystemReady system but there’s no display or accelerator support as yet. The Device-Tree mode is more feature full but both work pretty well with upstream kernels and NVIDIA are improving and upstreaming more things regularly.

For flashing with the latest Fedora releases you’ll want the Linux for Tegra (L4T) R32.7.6 release and the latest UEFI firmware (1.1.2 ATM). The R32.7.6 release fixes issues with python3.9 and later so you’ll need that for Fedora. The following will extract everything into a directory called Linux_for_Tegra. Note the release for Xavier is different to the L4T for the TX1/TX2 series of devices such as the nano.


$ sudo dnf install -y libxml2-utils lz4 usbutils xxd
$ tar xvf Jetson_Linux_R32.7.6_aarch64.tbz2
$ tar xvf nvidia-l4t-jetson-uefi-R32.6.1-20211119125725.tbz2
$ cd Linux_for_Tegra

To flash either the Xavier AGX or NX you need to put them into recovery mode and connect a USB cable, USB-C for AGX or micro-USB for NX. Once you’re in recovery mode you can flash them.

For the Xavier AGX:

$ lsusb | grep -i NV
Bus 001 Device 086: ID 0955:7019 NVIDIA Corp. APX
$ sudo ./flash.sh jetson-xavier-uefi-min external

For the Xavier NX:

$ lsusb | grep -i NV
Bus 001 Device 089: ID 0955:7e19 NVIDIA Corp. APX
$ sudo ./flash.sh jetson-xavier-nx-uefi-acpi internal

There will be a bunch of output and it will eventually return to the prompt and reset the device. You can now install Fedora on the device. You can use any of the pre-canned aarch64 image or traditional installer available from the fedora website. When running in ACPI mode you don’t get display output so you’ll need to use a serial console, in both ACPI and Device-Tree mode there’s not currently support for accelerated GPU graphics/AI/ML support. If you want to be able to easily switch between ACPI/Device-Tree modes you’ll want to install the dracut-config-generic package to have a generic initrd to make it easy to reboot between both modes.

SystemReady ES support for MacchiatoBin

I’ve had a MacchiatoBin Double Shot board for some time. It runs various services for my local network and generally just works. I run a TianoCore EDK2 firmware on it using ACPI. It’s purely a network device so I don’t bother with any form of graphics and in the very few occasions I need to access it locally I do so via the built in USB serial TTL.

Recently Solid Run announced the MacciatoBin is now SystemReady ES certified. Excellent news! I’ve worked with Arm for some time on both the SytemReady ES (Embedded Server) and SystemReady IR (IoT Ready) standards and recently the certification program has been finalised so it’s nice to start to see the fruits from all the hard work myself, and may others, have done over a number of years appear.

The EDK2 firmware I was running was coming up to two years old and there’s been a number of enhancements to the various components of the firmwares that make up a complete update so I decided to download the latest firmware and update it. Eventually I am sure Solid Run will have these published to LVFS to make the process even easier but I know that to get to this stage has been a LOT of effort so it’s still a great step forward.

The first step of updating a EDK2 firmware is to download it and put it on the EFI partition:

peter@macbin:~ $ wget https://github.com/Semihalf/edk2-platforms/wiki/releases/flash-image-a8k-mcbin.bin_r20210630
peter@macbin:~ $ sudo mv flash-image-a8k-mcbin.bin_r20210630 /boot/efi
peter@macbin:~ $ sudo reboot

On reboot you’re given a prompt to interrupt the boot process. From the menu select the option for the shell:

Shell> fs0:
FS0:\> ls
Directory of: FS0:\
04/06/2021  19:08          4,096  EFI
07/25/2021  17:01           2,855,040  flash-image-a8k-mcbin.bin_r20210630
          1 File(s)   2,855,040 bytes
          1 Dir(s)

FS0:\> fupdate flash-image-a8k-mcbin.bin_r20210630
Detected w25q32bv SPI NOR flash with page size 256 B, erase size 4 KB, total 4 MB
Updating, 99%
fupdate: Update 2855040 bytes at offset 0x0 succeeded!
FS0:\> reset

It then reboots and we’re done, you see a very similar output to previously with some updated versions of various firmware and before long you’re back through grub and running Fedora again. Painless!

I’m really happy to see this is such a straightforward process, and I’m looking forward to seeing more features, enhancements and fixes to the firmware including capsule updates and the associated LVFS/fwupdmgr support, and improvements around firmware security (fwupdmgr –force security). Top marks to the Solid Run team!

Fedora on the Pinebook Pro

Note: Some updates for Fedora 40+ with reduced steps.

First thing to note here is that this is not limited to the Pinebook Pro, I’m just using it as the example for 64 bit Rockchip devices with SPI flash on Fedora. This post is focused on devices with SPI but I’ll do a separate follow-up post for other devices including details for writing to eMMC over USB.

The story of Fedora on the Pinebook Pro, and other Rockchip devices, has been a sordid story of a lack of time, bugs, rabbit holes, more bugs and various other things. Not at all sordid at all really, mostly just a lack of time on my behalf, and nobody else stepping up to assist in a way to benefit all Fedora users, mostly they do one time hacks to sort themselves. Overall the support in Fedora for Rockchip devices has been quite solid for a number of releases. The problem has been with the early boot firmware, notable because without SPI flash it wants to splat itself across the first 8Mb of the disk, and if there was SPI flash it generally wasn’t overly stable/straight forward.

Anyway we’re now in a place where devices with SPI flash should mostly work just fine, those devices without it will work with a little manual intervention, and while the support isn’t complete, and will need more polish, they’re all details we can polish with little interruption to users by standard package updates. By default users will have accelerated graphics and from my testing on GNOME 40 it’s by all accounts a pretty decent experience!

Setting up the firmware

First step is to get the firmware written to SPI flash. This is a two step process, the first is to write out a micro SD card from another device, the second is to boot that mSD card on the Pinebook Pro, or another device like the Rockpro64, and write the firmware to the SPI flash.

There’s some nuances to this process, and the way the early boot firmware works, if another version of U-Boot takes precedence that is likely OK as it should still be able to work, the fall back is to use the internal switch to turn off the eMMC temporarily. I also have no idea if the Pine64 shipped U-Boot has any display output, the Fedora build does, if not you’ll need to use the option to disable eMMC or use a serial console cable. Anyway on to the steps:

Set up the mSD card
Use a mSD card that has no data you wish to keep, this process will wipe it out. You want at least U-Boot build 2021.04-3.fc34, you can adjust the umount to be more specific, and you need to substitute XXX for the media, otherwise it’s a relatively quick and straightforward process:

sudo dnf install --enablerepo=updates-testing -y arm-image-installer uboot-images-armv8
sudo umount /run/media/USERNAME/*
sudo spi-flashing-disk --target=pinebook-pro-rk3399 --media=/dev/XXX

Write the firmware to flash
Now remove the mSD card from your host and put it into the Pinebook Pro. Press the power button, from experience you likely need to press and momentarily hold and in a second or two the display will light up with text output. Interrupt the boot by pressing space. Next up we write out the flash:

Hit any key to stop autoboot:  0 
=> ls mmc 1:1
   182272   idbloader.img
   364544   idbloader-spi.img
  1079808   u-boot.itb
  1997312   u-boot-rockchip.bin
  1997312   u-boot-rockchip-spi.bin

5 file(s), 0 dir(s)

=> sf probe
SF: Detected gd25q128 with page size 256 Bytes, erase size 4 KiB, total 16 MiB

=> load mmc 1:1 ${fdt_addr_r} u-boot-rockchip-spi.bin
1997312 bytes read in 39 ms (8.2 MiB/s)

=> sf update ${fdt_addr_r} 0 ${filesize}
device 0 offset 0x0, size 0x52000
61440 bytes written, 274432 bytes skipped in 0.803s, speed 427777 B/s

Once the last command above has completed eject the mSD card and type reset at the => prompt and the device should reboot and you should see output similar to before but running from the SPI flash!

If you had to turn off the eMMC you can now turn it back on.

Installing Fedora

The nice thing with the firmware on SPI flash it should now work mostly like any other laptop and you can use either the pre canned desktop images (Workstation, KDE, XFCE, Sugar), the Workstation LiveCD iso or the standard everything network installer.

To run the arm Workstation image off a micro SD card or USB stick you can do the following:

arm-image-installer --media=/dev/XXX --resizefs --target=none --image=Fedora-Workstation-40-1.13.aarch64.raw.xz

Note ATM you’ll need to use the USB port on the right hand side, I need to investigate the USB/USB-C port on the left as it appears not to currently work in firmware, but works fine once Fedora is running.

Next steps and improvements

The two biggest issues remaining for the Pinebook Pro is enabling PCIe support (supported from June 2021) and the lack of the brcmfmac firmware, both WiFi and bluetooth, being upstream. For the later issue if there’s anyone from Synaptics that can assist in resolving that problem please reach out to me! A interim WiFi firmware to use is here.

Some things at the Fedora level I’ve not really tested and will do so more, and likely polish with OS updates, in the coming weeks include sound, USB-C port (charging and display output). On the firmware level there’s still some more improvements to be done, tweaks to improve the USB support, turning on the power LED as early as possible to give an indicator, improvements to the EFI framebuffer to ensure consistent early boot output, support for UEFI BGRT to enable smooth boot etc.

For support please email the Fedora Arm mailing list or reach out on IRC via #fedora-arm on Libera.Chat.