Any Linux distro on NVIDIA Jetson Orin with JetPack 6

NVIDIA has just released the Jetpack 6 Developer Preview for the NVIDIA Jetson Orin hardware. The thing that is most exciting about this release is they finally support the ability to use upstream kernels and other Linux distributions. This means you can start to use both RHEL (9.3 and later) and Fedora on the Jetson Orin hardware! This has been a LONG time coming, something I’ve been involved with for 5 years!

So while this is a developer preview, AKA public Beta, it’s still very usable and for people that are interested in using other Linux distributions now is the time to get stuck in. Like all things it’s not perfect and there’s still work to be done, but many hands do make light work!

You start by downloading the BSP from there you can follow the following instructions and you should end up with a device you can easily install Fedora 39 or RHEL 9.3 or other distros with the appropriate bits enabled.

To flash the firmware you need to follow the Orin AGX guide for recovery and cabling, for Orin NX/nano you need to use the HW pins near the mSD card, to put the device in recovery mode and cabling then do the following for Orin AGX:

$ tar zvf Jetson_Linux_R36.2.0_aarch64.tbz2
$ cd Linux_for_Tegra/
$ lsusb|grep -i nv
Bus 003 Device 044: ID 0955:7045 NVIDIA Corp. [unknown]
Bus 003 Device 045: ID 0955:7023 NVIDIA Corp. [unknown]
$ sudo ./flash.sh p3737-0000-p3701-0000-qspi external
removed a lot of output
*** The target generic has been flashed successfully. ***
Make the target filesystem available to the device and reset the board to boot from external external.
$

The command for other Orin devices such as NX and Nano will be similar, you’ll just have to swap the p3737 variable, eg for Orin Nano use: sudo ./flash.sh p3737-0000-p3701-0000-qspi external.

Once the flash completes the device will reboot and you will be able to use the usual mechanisms to install your OS, whether the RHEL or Fedora installers or a Fedora Arm image. I’ve tested running OSes off both the microSD and a NVME card, plus installing off USB, the DisplayPort output should work in EFI console mode. The firmware is based upon the widely known TianoCore/EDK2 so the firmware interface should be straight forward. For those that may need a serial console if it’s not automatically detected you can use console=ttyAMA0,115200, This runs off the microUSB port on /dev/ttyACM2 on the host device.

For hardware vendors that have hardware based on the NVIDIA Orin hardware they will be able to adopt and make this available to their customers that may wish to run distributions other than L4T. If they are unsure feel free to reach out to me in the usual locations.

Getting started with OpenCL using mesa/rusticl

Mesa, the open source low level graphics stack, has featured support for Open Compute Language (OpenCL) for some time via a front end called Clover. The problem was that the GPUs that it supported were limited, it didn’t have Image support, and wasn’t really under active development. Around a year ago Karol Herbst filed a merge request adding rusticl to Mesa 22.3 release, and soon after that I enabled it optionally in Fedora.. What is rusticl? It’s a OpenCL API implementation written in rust for Mesa, it will eventually replace clover. One advantage it has is image support from the outset and also works on much wider range of OpenCL capable GPUs, including some ARM GPUs, and the support is actively improving all the time.

In the year since it landed in a stable mesa release rusticl keeps evolving, faster, more HW support, more features, less crashes. I’ve tinkered with it as I’ve had spare time on the weekends and evenings as well as trying to work out details of how you’d use it to run higher level stacks.

As of writing it works with following gallium drivers: iris (Intel), nouveau (Nvidia), radeonsi/r600 (AMD ATI), and panfrost (Arm MALI). There’s other drivers that are various stages of development but are not yet upstream.

So let’s get the basics up and running on Fedora if you have supported hardware. First install the core software stack:

$ sudo dnf install -y mesa-libOpenCL mesa-dri-drivers spirv-llvm-translator spirv-tools-libs clinfo clpeak

Next we run clinfo and clpeak with required parameters, from the driver names above (in this case an Intel laptop), to enable rusticl. The two commands output a lot of information so I’m not going to post them here but the output shows OpenCL running and some details about what’s supported:

$ RUSTICL_ENABLE=iris clinfo --list
WARNING: OpenCL support via iris+clover is incomplete.
For a complete and conformant OpenCL implementation, use
https://github.com/intel/compute-runtime instead
Platform #0: rusticl
 `-- Device #0: Mesa Intel(R) Xe Graphics (TGL GT2)
Platform #1: Portable Computing Language
 `-- Device #0: cpu-11th Gen Intel(R) Core(TM) i7-1185G7 @ 3.00GHz
Platform #2: Clover
$ RUSTICL_ENABLE=iris clinfo
output a lot of details
$ RUSTICL_ENABLE=iris clpeak
output a lot of details

This gets up the basic pieces up and running for OpenCL, there’s of course more to do and not all use cases are complete. Eventually we won’t need to have the environment variables to enable rusticl. The details of drivers and other features enabled by environment variables are documented here. I plan to do some more posts as follow ups to build on this basis.

Using iwd for WiFi in Fedora

Fedora uses NetworkManager as the default for managing all the various different types of network. Underneath NetworkManager uses wpa_supplicant to connect to 802.11 based, AKA WiFi, wireless networks. There is an alternative called iwd which in a number of use cases works better, it also has the advantage that it offloads a bunch of things like crypto to the kernel interfaces which makes it smaller, and it’s under active development. iwd has a nice straight forward interface as well as being supported as an alternative NetworkManager so it just works in Fedora whether via nmcli or your chosen desktop environment.

So how do you make use of it in Fedora? Well it’s been packaged and supported for some time so it’s quite straight forward and there’s two ways to use it with NetworkManager. You can either swap it out for wpa_supplicant, or they can be installed side by side and you can change the NetworkManager default in a config to enable easy testing/swapping.

Option 1 (side by side):

sudo dnf install -y iwd
sudo cat >> /etc/NetworkManager/conf.d/iwd.conf << EOF
[device]
wifi.backend=iwd
EOF
sudo systemctl restart NetworkManager

Option 2 (swap):

sudo dnf swap -y wpa_supplicant iwd
sudo systemctl restart NetworkManager

You can now connect to WiFi networks a before with NetworkManager. Note it loses exciting configured WiFi networks.

Getting started with CircuitPython on Fedora

One of my little maker projects has a need for a battery powered micro controller, some DIY to clean up some old equipment and some basic HW design. Ultimately I’d like to be able to integrate it as a fun thing into home automation possibly even using Matter. More on the project for another post.

To move things forward and actually be able to play with LEDs and buttons rather than doing a board port I decided to use the standard firmware. I looked through my collection of micro controllers and found a FeatherS2, it’s based on the ESP-32-S2 MCU with WiFi in a Adafruit Feather form factor with builtin LiPo charger circuit, a USB-C interface, an onboard RGB LED and runs CircuitPython.

Next up I needed to get a development environment up and running on Fedora, batteries and other HW bits can come later. First step was to work out how to getting it running the latest CircuitPython. On the CircuitPython page for the FeatherS2 there’s two firmware, one for boot and USB and then CircuitPython itself.

Step one: Ensure you’re in the dialout group
A lot of micro contollers use serial ports to interface with so you need to be able to talk to a variety of them such as ttyACM0, ttyUSB0 or ttyS0. Simplest way to do this is to add yourself to the dialout group as follows:

sudo usermod -aG dialout ${USER}

Step two: Upgrade UF2 Bootloader
This step was probably unneeded but I upgraded to the 0.11.0 release to ensure I had all the latest fixes. For this bit you need to put the board into recovery. There’s two buttons, you plug it in, press and hold BOOT, press and release RST (reset) and then a second later release BOOT. You can then run the following commands to upgrade UF2:

sudo dnf install -y esptool unzip wget
mkdir ~/CircuitPython; cd ~/CircuitPython
wget https://github.com/adafruit/tinyuf2/releases/download/0.11.0/tinyuf2-unexpectedmaker_feathers2-0.11.0.zip
unzip tinyuf2-unexpectedmaker_feathers2-0.11.0.zip
esptool.py -p /dev/ttyACM0 write_flash 0x0 combined.bin

Once the flash completes you press the RST button and the board will got into a UF2 download mode with the RGB shining a bright green. You’ll also find you have a new USB drive appear with the name of UFTHRS2BOOT. Next step is CircuitPython!

Step three: Install CircuitPython
The latest stable release when I started playing with this was 7.3.3. To install is a simple copy of the uf2 file. While it’s copying the RGB will flash orange, once complete the board will reset and reboot into CircuitPython:

cd ~/CircuitPython
wget https://downloads.circuitpython.org/bin/unexpectedmaker_feathers2/en_GB/adafruit-circuitpython-unexpectedmaker_feathers2-en_GB-7.3.3.uf2
cp adafruit-circuitpython-unexpectedmaker_feathers2-en_GB-7.3.3.uf2 /run/media/${USER}/UFTHRS2BOOT/

Step three: Actually make it do something
The standard development environment for MicroPython and/or CircuitPython is an IDE called MU Editor. It’s nicely packaged up in Fedora and seems to just work. The code in CircuitPython appears in a /run/media/${USER}/CIRCUITPY/ drive which mu automatically detects and you can get access to the python console directly within MU (correct permissions via the dialout group in step one). By default the FeatherS2 has a little app that cycles through colours on the RGB.

sudo dnf install -y mu

Next steps
The CircuitPython firmware has lots of built in functionality that just works, there’s also a large ecosystem of libraries and drivers available for Micropython and CircuitPython. Adafruit have a lot of extremely good tutorials to get you going. Next up I plan to play with getting it on my IoT network using WiFi and communicating via MQTT.

Using fwupdmgr to update NVME firmware

The fabulous fwupdmgr provides the ability to easily update firmware that is published to Linux Vendor Firmware Service (LVFS) but it can also be used to apply updates that aren’t necessarily in LVFS. One type of firmware that it supports updating is NVME firmware, that’s basically any NMVE, because the standard specifies a standardised mechanism for updating the firmware on all NVME devices.

I had a need to update a NVME firmware in an aarch64 device to see if it fixed an issue I was seeing. The Crucial P2 supported options were of course x86 only. The ISO download actually contained a little LinuxOS in an initrd on the .iso. The advice from Richard the fwupd technical lead was to “Look for a ~4mb high entropy blob” so mounting it up, I mounted the iso, extracted the initrd, and then used fwupdmmgr to apply the new firmware.

Find the NVME and check the firmware version:

$ cat /sys/class/nvme/nvme0/firmware_rev 
P2CR010 

So once I’d downloaded the update file I did the following to extract and update the firmware. Note I did this all as root, you can do most of it as non root.

# unzip iso_p2cr012.zip
# mount -o loop iso_p2cr012.iso /mnt/
# mkdir ~/tmp
# cp /mnt/boot/corepure64.gz tmp/
# cd tmp
# gunzip corepure64.gz
# cpio -iv < corepure64
# fwupdtool install-blob opt/firmware/P2CR012/1.bin
Loading…                 [-                                      ]
Loading…                 [-                                      ]
Choose a device:
0.	Cancel
1.	71b677ca0f1bc2c5b804fa1d59e52064ce589293 (CT250P2SSD8)
2.	2270d251f7c1dc37a29a2aa720a566aa0fa0ecde (spi1.0)
1
Waiting…                 [************************************** ] Less than one minute remaining…
An update requires a reboot to complete. Restart now? [y|N]: y

And away it goes, a reboot later and did it work?

$ cat /sys/class/nvme/nvme0/firmware_rev 
P2CR012

YES!!

HW video offload on Fedora Arm

There’s been the starting pieces of hardware video offload with the stateless engine support for some time and it now supports at least H264/HEVC/VP8/VP9/mepg2 decode offload depending on the hardware capabilities. The problem has been support for software/userspace has taken longer then the initial drivers but now that’s catching up now with gstreamer support landing in 2020 and with apps like clapper now using it. I’ve been meaning to play with this and work out how to make it work in Fedora as it’s useful for devices based on the AllWinner/Rockchip/NXP i.MX8 devices like the Pine64 laptops/phones plus a bunch of other devices, even NVIDIA Jetson devices should work before long.

You’ll need to ensure you have gstreamer1-plugins-bad-free installed and the video application I was testing with is clapper:

$ sudo dnf install -y gstreamer1-plugins-bad-free clapper

Seeing what hardware offload is supported:

$ gst-inspect-1.0 v4l2codecs
Plugin Details:
  Name                     v4l2codecs
  Description              V4L2 CODEC Accelerators plugin
  Filename                 /usr/lib64/gstreamer-1.0/libgstv4l2codecs.so
  Version                  1.20.0
  License                  LGPL
  Source module            gst-plugins-bad
  Source release date      2022-02-03
  Binary package           Fedora GStreamer-plugins-bad package
  Origin URL               http://download.fedoraproject.org

  v4l2slh264dec: V4L2 Stateless H.264 Video Decoder
  v4l2slmpeg2dec: V4L2 Stateless Mpeg2 Video Decoder
  v4l2slvp8alphadecodebin: VP8 Alpha Decoder
  v4l2slvp8dec: V4L2 Stateless VP8 Video Decoder
  v4l2slvp9alphadecodebin: VP9 Alpha Decoder
  v4l2slvp9dec: V4L2 Stateless VP9 Video Decoder

  6 features:
  +-- 6 elements

Finally in Clapper you need to enable playbin3 option, I also enabled Pipewire audio support:
Clapper Preferences

We will also be enabling decode support in Chromium and Chromium freeworld before long, there’s a little more work to do here, but as usual once it lands it’ll all just start to work in Chromium too!

Helpful git tips

So chatting with a colleague about some git tricks this week I discovered that not everyone was aware you could change the bash prompt to give certain git status, such as branch, and things like if you’re in merge/am/bisect modes etc. I’ve had the pieces in my .bashrc for so long I had literally got to the point it was assumed functionality that every one has enabled.

The following snippet is what I have in my ~/.bashrc:

# git branch display
source /usr/share/git-core/contrib/completion/git-prompt.sh
export GIT_PS1_SHOWDIRTYSTATE=true
export GIT_PS1_SHOWUNTRACKEDFILES=true
export PS1='[\[\e[0;32m\]\u\[\e[0m\]@\[\e[0;35m\]\h\[\e[0m\] \W\[\e[0;33m\]$(__git_ps1 " (%s)")\[\e[0m\]]\[\e[0;32m\]\$ \[\e[0m\]'

And with that you get a more useful prompt that looks like the prompt below, in this case merging bits, for all git repos with added colours too!:

[peter@localhost linux (master *+|MERGING)]$

Using nmcli to configure a static dual stack wired network interface

I recently managed to break the network on my VM that hosts this blog. Basically I removed the NetworkManager-initscripts-ifcfg-rh package because I don’t use the old style ifcfg configuration anywhere else and I had forgotten how long I’d had this VM. So I went into the web console, manually bought up the network with ip commands and reinstalled the package but it made no difference. Oh well! Time to just move it to the new config so I just worked out the nmcli options for all the bits in the old ifcfg. This VM network is nothing special, it’s basically dual IPv4/IPv6 interface with associated DNS.

Step 1: Show existing connections:

$ sudo nmcli c
NAME  UUID                                  TYPE      DEVICE 
eth0  a603bba7-fad8-3c71-9d4c-2cd5dc50e114  ethernet  eth0   

Step 2: Delete existing connection:

$ sudo nmcli c del a603bba7-fad8-3c71-9d4c-2cd5dc50e114

Step 3: Create a new connection (Note the IP addresses are random, the DNS servers are the Google public ones):

$ sudo nmcli c add type ethernet ifname eth0 con-name eth0 mac 80:00:00:ab:cd:ef ip4 192.168.10.6/24 gw4 192.168.10.1 ip6 fe80::b257:377c:e7b3:29ed/64 gw6 2A03:B0C0:0003:00D0:0000:0000:0000:0001 ipv4.dns "8.8.8.8 8.8.4.4" ipv6.dns "2001:4860:4860::8888 2001:4860:4860::8844"

Now the blog is back! The new connection is stored in /etc/NetworkManager/system-connections/eth0.nmconnection

Short history of ARMv7/armhfp/arm32 in Fedora

Back in mid November I proposed a change for Fedora 37 to retire ARMv7 as an architecture, FESCo accepted the proposal. Per the Fedora 36 schedule we branched Fedora 36 this week. Last night I enacted the last of the process to disable it in rawhide so to quote “It’s dead Jim”. The last release of Fedora to support ARMv7 AKA armhfp AKA arm32 will be Fedora 36 which will go end of life around June 2023.

I thought I’d cover a few of the things we achieved with Fedora ARM and some of the impact it’s had on the wider Linux on ARM ecosystem which people may not have realised.

First a little bit of ARM history in the Fedora ecosystem. The beginnings of ARM support actually precedes Fedora all the way back to 1998 with a fork of Red Hat Linux 4.2 and more officially with Red Hat Linux 5.1 on the Corel Netwinder (I always wanted one of those but they weren’t available in Aus).

In Fedora itself the earliest details I remember was that Marvell bootstrapped ARMv5 in Fedora 7 and continued to build and support it through to Fedora 12. This “software architecture” was known as softfp. It was optimised for the ARMv5 architecture which didn’t have a hard requirement on a floating point unit so emulated it when it was needed hence “software floating point”. In Fedora 13 Seneca College took over the ARMv5 infrastructure and building from Marvell. I officially got involved in the Fedora 14 build process and soon after was also contracted by OLPC to drive Fedora on OLPC for their ARM based XO laptops as well as work on their i686 devices to have a single OS for all of them.

In mid 2011, the Fedora 15 timeframe, a small Red Hat team started to do a ARMv7 hard floating point, AKA hardfp or armhfp, bootstrap as ARM’s new v7 mandated a floating point unit. The bootstrap included the core toolchain (binutils/gcc/glibc/elfutils and friends) and ultimately the entire distribution, I drove this effort from a community, build and packaging perspective. This required 100s of patches to upstream projects that made many assumptions about ARM only being softfp, but it also allowed us at the time to fix many general architecture assumptions in these projects. The hard floating point bootstrapping was useful for the wider community too, it was used by Nokia as the base of it’s hardfp efforts for Maemo, plus other distros used it as as it’s much easier/quicker if you already have a full distro running the architecture you wish to boostrap. What wasn’t generally known at the time was also the first new architecture that has been bootstrapped in the Fedora/RHEL ecosystem since x86_64 a long time before and it allowed Red Hat to refresh it’s memory on how to do this in preparation of the then unannounced aarch64 architecture and the POWER Little Endian intentions, basically it provided a cover story. We also worked to get other languages such as Fortran, golang, rust and others building and working on armhfp and those other architectures. The final piece of this was ARMv7 being promoted to a primary Fedora architecture in Fedora 20. This then later went on to my proposal to redefine secondary architectures in Fedora.

In the wider community of Linux Fedora ARM was the first distribution to adopt the kernel “multi platform” work enabling us to go from building 5 different kernels to support a handful of arm devices to a single kernel supporting 100s of devices in a very short period of time. I worked with closely Arnd Bergmann from Linaro on issues with the early pieces of the multiplatform work. In upstream U-Boot we posted the first distro_boot patches to support booting Linux in the same way across all the devices we actively supported so we didn’t need specially wrapped kernels and know exact offsets for every SoC or device. The distro_boot support evolved, working with SUSE, into UEFI support in U-Boot further standardising the ARM boot process by abstracting the pieces that were different and letting the firmware deal with them. This work ultimately evolved into EBBR and the ARM System Ready IR spec. In Fedora 34 we moved to soley supporting UEFI on both ARM architectures. A lot of Linux distros still have specific kernels for each device and use non standard boot methods for devices and hence have an image for each device/use-case they wish to use. This was something Fedora identified very early on as something that would not scale!

Fedora also leads a lot of things in the gcc toolchain stack across all our supported architectures, we’ve actively enabled a lot of security features and other things like LTO early on. As the Fedora gcc maintainers, employed by Red Hat, are also key upstream GCC maintainers we’re almost always the first distribution to rebase onto a new release before it’s a stable release, for example Fedora 36 had just had a mass rebuild against a gcc-12 pre release snapshot. This builds all of the 50k or more source packages with the pre-release of the new toolchain making for a much better release for the wider GCC community because this picks up a number of bugs/regressions in both the general support but also in the architectures Fedora supports which means the ARMv7 hardfp support in GCC has benefited from 100s of bugs we’ve detected in gcc/binutils/glibc etc before they land in a stable gcc release. With the retirement of ARMv7 in Fedora this is going to be something the wider ARMv7 community is going to have to pick up post the GCC-12 release.

Over the subsequent 11 years of ARMv7 support in Fedora, and much longer if you include the early ARMv5 the distribution has also enabled a number of other innovative features like support for containers, support for devices like the Raspberry Pi 2 and 3 in Fedora 25. as mentioned various toolchains, and fun things like robots. Of course we also lose some things too. Devices like the BeagleBone don’t yet have a 64 bit sibling, but there’s less and less 32 bit devices coming out and the use of armhfp is waning quickly and the maintenance cost is rising as the industry moves more generally to 64 bit even in embedded use cases and the fact is with devices like the $15 Raspberry Pi Zero 2W it makes less and less sense even if I do still actively run BeagleBones, a Panda-ES and 3 different i.MX6 devices.

So I engaged with the wider Arm ecosystem and it made sense to finally sunset our ARMv7 32 bit support. We’re of course leaving it in good shape with things like gcc-12, the latest rust and golang toolchains and 5.17 kernels, much newer by the time F-36 goes EOL in June 2023, it will be in good shape if people wish to use it as the basis of some form of continuing ARMv7 supported Linux distribution.

Sail off into the sunset friend, it’s been a fun 12 years of hacking on those projects!

Fedora on NVIDIA Jetson Xavier

The last two years or so I’ve been working with NVIDIA on general distro support including UEFI and ACPI for their Jetson Xavier platforms. Their Xavier platform, except a few quirks, are mostly SystemReady-ES compliant, so having a SBBR compliant firmware goes quite some way to having a widely available, relatively affordable, platform that “just works” for the arm ecosystem. I was very excited to finally have NVIDIA finally release the first version in March this year. This firmware is a standard UEFI firmware based on the open source TianoCore/EDK2 reference firmware, it allows booting in either ACPI or Device-Tree mode and supports all the basic things needed. The ACPI mode is not as fully featured as the Device-Tree mode as yet. In ACPI you get compute (cpu/memory/virt etc), PCIe, USB, network, which is just fine if you’re just looking for standard server or for testing a SystemReady system but there’s no display or accelerator support as yet. The Device-Tree mode is more feature full but both work pretty well with upstream kernels and NVIDIA are improving and upstreaming more things regularly.

For flashing with the latest Fedora releases you’ll want the Linux for Tegra (L4T) R32.6.1 release and the latest UEFI firmware (1.1.2 ATM). The R32.6.1 release fixes issues with python3.9 and later so you’ll need that for Fedora. The following will extract everything into a directory called Linux_for_Tegra. Note the release for Xavier is different to the L4T for the TX1/TX2 series of devices such as the nano.

$ tar xvf Jetson_Linux_R32.6.1_aarch64.tbz2
$ tar xvf nvidia-l4t-jetson-uefi-R32.6.1-20211119125725.tbz2
$ cd Linux_for_Tegra

To flash either the Xavier AGX or NX you need to put them into recovery mode and connect a USB cable, USB-C for AGX or micro-USB for NX. Once you’re in recovery mode you can flash them.

For the Xavier AGX:

$ lsusb | grep -i NV
Bus 001 Device 086: ID 0955:7019 NVIDIA Corp. APX
$ sudo ./flash.sh jetson-xavier-uefi-min external

For the Xavier NX:

$ lsusb | grep -i NV
Bus 001 Device 089: ID 0955:7e19 NVIDIA Corp. APX
$ sudo ./flash.sh jetson-xavier-nx-uefi-acpi internal

There will be a bunch of output and it will eventually return to the prompt and reset the device. You can now install Fedora on the device. You can use any of the pre-canned aarch64 image or traditional installer available from the fedora website. When running in ACPI mode you don’t get display output so you’ll need to use a serial console, in both ACPI and Device-Tree mode there’s not currently support for accelerated GPU graphics/AI/ML support. If you want to be able to easily switch between ACPI/Device-Tree modes you’ll want to install the dracut-config-generic package to have a generic initrd to make it easy to reboot between both modes.