Any Linux distro on NVIDIA Jetson Orin with JetPack 6

NVIDIA has just released the Jetpack 6 Developer Preview for the NVIDIA Jetson Orin hardware. The thing that is most exciting about this release is they finally support the ability to use upstream kernels and other Linux distributions. This means you can start to use both RHEL (9.3 and later) and Fedora on the Jetson Orin hardware! This has been a LONG time coming, something I’ve been involved with for 5 years!

So while this is a developer preview, AKA public Beta, it’s still very usable and for people that are interested in using other Linux distributions now is the time to get stuck in. Like all things it’s not perfect and there’s still work to be done, but many hands do make light work!

You start by downloading the BSP from there you can follow the following instructions and you should end up with a device you can easily install Fedora 39 or RHEL 9.3 or other distros with the appropriate bits enabled.

To flash the firmware you need to follow the Orin AGX guide for recovery and cabling, for Orin NX/nano you need to use the HW pins near the mSD card, to put the device in recovery mode and cabling then do the following for Orin AGX:

$ tar zvf Jetson_Linux_R36.2.0_aarch64.tbz2
$ cd Linux_for_Tegra/
$ lsusb|grep -i nv
Bus 003 Device 044: ID 0955:7045 NVIDIA Corp. [unknown]
Bus 003 Device 045: ID 0955:7023 NVIDIA Corp. [unknown]
$ sudo ./flash.sh p3737-0000-p3701-0000-qspi external
removed a lot of output
*** The target generic has been flashed successfully. ***
Make the target filesystem available to the device and reset the board to boot from external external.
$

The command for other Orin devices such as NX and Nano will be similar, you’ll just have to swap the p3737 variable, eg for Orin Nano use: sudo ./flash.sh p3737-0000-p3701-0000-qspi external.

Once the flash completes the device will reboot and you will be able to use the usual mechanisms to install your OS, whether the RHEL or Fedora installers or a Fedora Arm image. I’ve tested running OSes off both the microSD and a NVME card, plus installing off USB, the DisplayPort output should work in EFI console mode. The firmware is based upon the widely known TianoCore/EDK2 so the firmware interface should be straight forward. For those that may need a serial console if it’s not automatically detected you can use console=ttyAMA0,115200, This runs off the microUSB port on /dev/ttyACM2 on the host device.

For hardware vendors that have hardware based on the NVIDIA Orin hardware they will be able to adopt and make this available to their customers that may wish to run distributions other than L4T. If they are unsure feel free to reach out to me in the usual locations.

HW video offload on Fedora Arm

There’s been the starting pieces of hardware video offload with the stateless engine support for some time and it now supports at least H264/HEVC/VP8/VP9/mepg2 decode offload depending on the hardware capabilities. The problem has been support for software/userspace has taken longer then the initial drivers but now that’s catching up now with gstreamer support landing in 2020 and with apps like clapper now using it. I’ve been meaning to play with this and work out how to make it work in Fedora as it’s useful for devices based on the AllWinner/Rockchip/NXP i.MX8 devices like the Pine64 laptops/phones plus a bunch of other devices, even NVIDIA Jetson devices should work before long.

You’ll need to ensure you have gstreamer1-plugins-bad-free installed and the video application I was testing with is clapper:

$ sudo dnf install -y gstreamer1-plugins-bad-free clapper

Seeing what hardware offload is supported:

$ gst-inspect-1.0 v4l2codecs
Plugin Details:
  Name                     v4l2codecs
  Description              V4L2 CODEC Accelerators plugin
  Filename                 /usr/lib64/gstreamer-1.0/libgstv4l2codecs.so
  Version                  1.20.0
  License                  LGPL
  Source module            gst-plugins-bad
  Source release date      2022-02-03
  Binary package           Fedora GStreamer-plugins-bad package
  Origin URL               http://download.fedoraproject.org

  v4l2slh264dec: V4L2 Stateless H.264 Video Decoder
  v4l2slmpeg2dec: V4L2 Stateless Mpeg2 Video Decoder
  v4l2slvp8alphadecodebin: VP8 Alpha Decoder
  v4l2slvp8dec: V4L2 Stateless VP8 Video Decoder
  v4l2slvp9alphadecodebin: VP9 Alpha Decoder
  v4l2slvp9dec: V4L2 Stateless VP9 Video Decoder

  6 features:
  +-- 6 elements

Finally in Clapper you need to enable playbin3 option, I also enabled Pipewire audio support:
Clapper Preferences

We will also be enabling decode support in Chromium and Chromium freeworld before long, there’s a little more work to do here, but as usual once it lands it’ll all just start to work in Chromium too!

Fedora on NVIDIA Jetson Xavier

The last two years or so I’ve been working with NVIDIA on general distro support including UEFI and ACPI for their Jetson Xavier platforms. Their Xavier platform, except a few quirks, are mostly SystemReady-ES compliant, so having a SBBR compliant firmware goes quite some way to having a widely available, relatively affordable, platform that “just works” for the arm ecosystem. I was very excited to finally have NVIDIA finally release the first version in March this year. This firmware is a standard UEFI firmware based on the open source TianoCore/EDK2 reference firmware, it allows booting in either ACPI or Device-Tree mode and supports all the basic things needed. The ACPI mode is not as fully featured as the Device-Tree mode as yet. In ACPI you get compute (cpu/memory/virt etc), PCIe, USB, network, which is just fine if you’re just looking for standard server or for testing a SystemReady system but there’s no display or accelerator support as yet. The Device-Tree mode is more feature full but both work pretty well with upstream kernels and NVIDIA are improving and upstreaming more things regularly.

For flashing with the latest Fedora releases you’ll want the Linux for Tegra (L4T) R32.6.1 release and the latest UEFI firmware (1.1.2 ATM). The R32.6.1 release fixes issues with python3.9 and later so you’ll need that for Fedora. The following will extract everything into a directory called Linux_for_Tegra. Note the release for Xavier is different to the L4T for the TX1/TX2 series of devices such as the nano.

$ tar xvf Jetson_Linux_R32.6.1_aarch64.tbz2
$ tar xvf nvidia-l4t-jetson-uefi-R32.6.1-20211119125725.tbz2
$ cd Linux_for_Tegra

To flash either the Xavier AGX or NX you need to put them into recovery mode and connect a USB cable, USB-C for AGX or micro-USB for NX. Once you’re in recovery mode you can flash them.

For the Xavier AGX:

$ lsusb | grep -i NV
Bus 001 Device 086: ID 0955:7019 NVIDIA Corp. APX
$ sudo ./flash.sh jetson-xavier-uefi-min external

For the Xavier NX:

$ lsusb | grep -i NV
Bus 001 Device 089: ID 0955:7e19 NVIDIA Corp. APX
$ sudo ./flash.sh jetson-xavier-nx-uefi-acpi internal

There will be a bunch of output and it will eventually return to the prompt and reset the device. You can now install Fedora on the device. You can use any of the pre-canned aarch64 image or traditional installer available from the fedora website. When running in ACPI mode you don’t get display output so you’ll need to use a serial console, in both ACPI and Device-Tree mode there’s not currently support for accelerated GPU graphics/AI/ML support. If you want to be able to easily switch between ACPI/Device-Tree modes you’ll want to install the dracut-config-generic package to have a generic initrd to make it easy to reboot between both modes.

nullr0ute

Updated – Apr 2024
You now used the latest R32.7.4 release and it now works with the latest Python releases. Some minor edits below.

Overview
Nvidia launched the Jetson Nano Developer Kit in March 2019, since there there’s been a few minor refreshes including a just announced cheaper 2Gb model. I received the original 4Gb rev A device shortly after they were launched.

Over the last year or so as part of my role at Red Hat I started working with some of the NVidia Tegra team to improve support for the Jetson devices. This work has been wide ranging and though it’s taken awhile, with Fedora 33 we’re starting to see the fruits of that collaboration. The first is improved support for the Jetson Nano. The official L4T (Linux 4 Tegra) Jetson Nano images look a lot like an Android phone with numerous partitions across the mSD card. This makes it harder to support a generic Linux distribution like Fedora as there are assumptions by distributions of control they can have over the storage, so while it was certainly possible to get Fedora to run on these devices it generally wasn’t for the faint of heart. As of the recent L4T releases, you definitely want at least R32.4.4, it’s now a supported option to flash all the firmware to the onboard SPI flash enabling the use of the entire mSD card for the OS of your choice, which as we all know will be Fedora 😉 but the instructions here should be adaptable to work for any distribution.

Before we begin
We do it in two stages, first is to flash the new firmware to the SPI over the micro USB port, second we’ll prepare the Fedora OS for the mSD card. For the first stage you’ll need the latest L4T Release R32.7.4 and the Fedora U-Boot builds installed locally.

Before we get started you’ll need the following:

  • A USB-A to micro USB cable for flashing
  • A HDMI monitor and a USB keyboard
  • A jumper, a jumper wire or something to close the connection on the FRC pins for recovery mode
  • A 3.3v USB Serial TTY (optional)
  • An appropriate 5v barrel PSU (optional)

If you wish to use a serial TTY there’s a good guide here for connecting it to the RevA nano, the RevB has two camera connectors so they’ve moved the serial console headers to near the mSD card slot. The command to see serial output is:

screen /dev/ttyUSB0 115200

Flashing the Jetson Nano
So let’s get started with flashing the firmware. This step with the firmware on the SPI doesn’t have to be done often. First we’ll extract the L4T release and get all the bits installed that we need to flash the firmware:

sudo dnf install -y usbutils uboot-images-armv8 arm-image-installer
tar xvf ~/Downloads/Jetson-210_Linux_R32.7.4_aarch64.tbz2
cd Linux_for_Tegra
cp /usr/share/uboot/p3450-0000/u-boot.bin bootloader/t210ref/p3450-0000/

Next, based on instructions from the NVidia Jetson Nano Quick Start Guide, we need to put the Jetson Nano into Force Recovery Mode (FRC) to prepare for flashing the firmware:

  1. Ensure that your Jetson Nano Developer Kit is powered off. There’s no need for a mSD card ATM, we’re just writing to the SPI flash.
  2. Connect the Micro-USB OTG cable to the Micro USB port on the Nano. Don’t plug it into the host computer just yet.
  3. Enable Force Recovery mode by placing a jumper across the FRC pins of the Button Header on the carrier board.
    a. For carrier board revision A02, these are pins 3 and 4 of Button Header (J40) which is located near the camera header.
    b. For carrier board revision B01, these are pins 9 and 10 of Button Header (J50), which is located on the edge of the carrier board under the Jetson module.
  4. Only if you wish to use a separate PSU place a jumper across J48 to enable use of a DC power adapter.
  5. Connect a DC power adapter to J25. The developer kit powers on automatically and enters Force Recovery mode. Note it may be possible to do this with USB power but I’ve not tested it.
  6. Remove the jumper from the FRC pins of the Button Header.
  7. See if you can see the Jetson Nano is in recovery mode by running:
    lsusb | grep -i nvidia

Now we can actually flash the firmware (make sure you’re still in the Linux_for_Tegra directory):

sudo ./flash.sh p3448-0000-max-spi external

You will see a lot of output as the command runs, and if you have a serial TTY you’ll see some output there but eventually you’ll be returned to the command prompt and the system will reset. If you have a HDMI monitor attached you’ll see the NVidia logo pop up, if you have a serial console you’ll see a bunch of output and eventually the output of U-Boot and the associated U-Boot prompt.

Jetson TX1 and TX2
You can basically follow the same instructions above for the older TX1/TX2 devices except for two things. For the TX1 you can use the same L4T release, for the TX2 you need to download a different L4T release.

For the U-Boot copy there’s a different U-Boot for each device which needs to be copied to a different location. For the firmware copy I treat the eMMC as if it was the SPI flash, and run the OS off a SD card, it’s not the most efficient but it keeps things more straight forward:

TX1:

cp /usr/share/uboot/p2371-2180/u-boot* bootloader/t210ref/p2371-2180/
sudo ./flash.sh jetson-tx1 mmcblk0p1

TX2:

cp /usr/share/uboot/p2771-0000-500/* bootloader/t186ref/p2771-0000/500/
sudo ./flash.sh jetson-tx2 mmcblk0p1

Getting Fedora running
Now we have the firmware flashed we can prepare Fedora for the mSD card. Download the Fedora Workstation for aarch64 raw image. You can of course also use XFCE, Minimal or Server images. Put the mSD card in reader and after unmounting any filesystem run the following command (look at the help for other options around users/ssh-keys):

sudo arm-image-installer --media=/dev/XXX --resizefs --target=none --image=~/Downloads/Fedora-Workstation-33-1.3.aarch64.raw.xz

Note you need to replace XXX with the right device, and you don’t need a target option as we’re not writing the firmware to the mSD card.

Once that completes you should be able to pop the mSD card into your Jetson Nano and reset the device and see it boot. You will see all the output if you have a serial console attached. If you’re using HDMI it may take a little while once the NVidia logo disappears for the GNOME first user setup to appear.

Also note that while a lot of things work on this device, like the nouveau driver for display, it’s not perfect yet and we’re actively working to fix and improve the support for the Jetson Nano, most of these will come via the standard Fedora update mechanism. If you have queries please engage in the usual ways via the mailing list or #fedora-arm on Libera.Chat or on arm channel on matrix.