Any Linux distro on NVIDIA Jetson Orin with JetPack 6

NVIDIA has just released the Jetpack 6 Developer Preview for the NVIDIA Jetson Orin hardware. The thing that is most exciting about this release is they finally support the ability to use upstream kernels and other Linux distributions. This means you can start to use both RHEL (9.3 and later) and Fedora on the Jetson Orin hardware! This has been a LONG time coming, something I’ve been involved with for 5 years!

So while this is a developer preview, AKA public Beta, it’s still very usable and for people that are interested in using other Linux distributions now is the time to get stuck in. Like all things it’s not perfect and there’s still work to be done, but many hands do make light work!

You start by downloading the BSP from there you can follow the following instructions and you should end up with a device you can easily install Fedora 39 or RHEL 9.3 or other distros with the appropriate bits enabled.

To flash the firmware you need to follow the Orin AGX guide for recovery and cabling, for Orin NX/nano you need to use the HW pins near the mSD card, to put the device in recovery mode and cabling then do the following for Orin AGX:

$ tar zvf Jetson_Linux_R36.2.0_aarch64.tbz2
$ cd Linux_for_Tegra/
$ lsusb|grep -i nv
Bus 003 Device 044: ID 0955:7045 NVIDIA Corp. [unknown]
Bus 003 Device 045: ID 0955:7023 NVIDIA Corp. [unknown]
$ sudo ./flash.sh p3737-0000-p3701-0000-qspi external
removed a lot of output
*** The target generic has been flashed successfully. ***
Make the target filesystem available to the device and reset the board to boot from external external.
$

The command for other Orin devices such as NX and Nano will be similar, you’ll just have to swap the p3737 variable, eg for Orin Nano use: sudo ./flash.sh p3737-0000-p3701-0000-qspi external.

Once the flash completes the device will reboot and you will be able to use the usual mechanisms to install your OS, whether the RHEL or Fedora installers or a Fedora Arm image. I’ve tested running OSes off both the microSD and a NVME card, plus installing off USB, the DisplayPort output should work in EFI console mode. The firmware is based upon the widely known TianoCore/EDK2 so the firmware interface should be straight forward. For those that may need a serial console if it’s not automatically detected you can use console=ttyAMA0,115200, This runs off the microUSB port on /dev/ttyACM2 on the host device.

For hardware vendors that have hardware based on the NVIDIA Orin hardware they will be able to adopt and make this available to their customers that may wish to run distributions other than L4T. If they are unsure feel free to reach out to me in the usual locations.

HW video offload on Fedora Arm

There’s been the starting pieces of hardware video offload with the stateless engine support for some time and it now supports at least H264/HEVC/VP8/VP9/mepg2 decode offload depending on the hardware capabilities. The problem has been support for software/userspace has taken longer then the initial drivers but now that’s catching up now with gstreamer support landing in 2020 and with apps like clapper now using it. I’ve been meaning to play with this and work out how to make it work in Fedora as it’s useful for devices based on the AllWinner/Rockchip/NXP i.MX8 devices like the Pine64 laptops/phones plus a bunch of other devices, even NVIDIA Jetson devices should work before long.

You’ll need to ensure you have gstreamer1-plugins-bad-free installed and the video application I was testing with is clapper:

$ sudo dnf install -y gstreamer1-plugins-bad-free clapper

Seeing what hardware offload is supported:

$ gst-inspect-1.0 v4l2codecs
Plugin Details:
  Name                     v4l2codecs
  Description              V4L2 CODEC Accelerators plugin
  Filename                 /usr/lib64/gstreamer-1.0/libgstv4l2codecs.so
  Version                  1.20.0
  License                  LGPL
  Source module            gst-plugins-bad
  Source release date      2022-02-03
  Binary package           Fedora GStreamer-plugins-bad package
  Origin URL               http://download.fedoraproject.org

  v4l2slh264dec: V4L2 Stateless H.264 Video Decoder
  v4l2slmpeg2dec: V4L2 Stateless Mpeg2 Video Decoder
  v4l2slvp8alphadecodebin: VP8 Alpha Decoder
  v4l2slvp8dec: V4L2 Stateless VP8 Video Decoder
  v4l2slvp9alphadecodebin: VP9 Alpha Decoder
  v4l2slvp9dec: V4L2 Stateless VP9 Video Decoder

  6 features:
  +-- 6 elements

Finally in Clapper you need to enable playbin3 option, I also enabled Pipewire audio support:
Clapper Preferences

We will also be enabling decode support in Chromium and Chromium freeworld before long, there’s a little more work to do here, but as usual once it lands it’ll all just start to work in Chromium too!

Fedora on NVIDIA Jetson Xavier

The last two years or so I’ve been working with NVIDIA on general distro support including UEFI and ACPI for their Jetson Xavier platforms. Their Xavier platform, except a few quirks, are mostly SystemReady-ES compliant, so having a SBBR compliant firmware goes quite some way to having a widely available, relatively affordable, platform that “just works” for the arm ecosystem. I was very excited to finally have NVIDIA finally release the first version in March this year. This firmware is a standard UEFI firmware based on the open source TianoCore/EDK2 reference firmware, it allows booting in either ACPI or Device-Tree mode and supports all the basic things needed. The ACPI mode is not as fully featured as the Device-Tree mode as yet. In ACPI you get compute (cpu/memory/virt etc), PCIe, USB, network, which is just fine if you’re just looking for standard server or for testing a SystemReady system but there’s no display or accelerator support as yet. The Device-Tree mode is more feature full but both work pretty well with upstream kernels and NVIDIA are improving and upstreaming more things regularly.

For flashing with the latest Fedora releases you’ll want the Linux for Tegra (L4T) R32.6.1 release and the latest UEFI firmware (1.1.2 ATM). The R32.6.1 release fixes issues with python3.9 and later so you’ll need that for Fedora. The following will extract everything into a directory called Linux_for_Tegra. Note the release for Xavier is different to the L4T for the TX1/TX2 series of devices such as the nano.

$ tar xvf Jetson_Linux_R32.6.1_aarch64.tbz2
$ tar xvf nvidia-l4t-jetson-uefi-R32.6.1-20211119125725.tbz2
$ cd Linux_for_Tegra

To flash either the Xavier AGX or NX you need to put them into recovery mode and connect a USB cable, USB-C for AGX or micro-USB for NX. Once you’re in recovery mode you can flash them.

For the Xavier AGX:

$ lsusb | grep -i NV
Bus 001 Device 086: ID 0955:7019 NVIDIA Corp. APX
$ sudo ./flash.sh jetson-xavier-uefi-min external

For the Xavier NX:

$ lsusb | grep -i NV
Bus 001 Device 089: ID 0955:7e19 NVIDIA Corp. APX
$ sudo ./flash.sh jetson-xavier-nx-uefi-acpi internal

There will be a bunch of output and it will eventually return to the prompt and reset the device. You can now install Fedora on the device. You can use any of the pre-canned aarch64 image or traditional installer available from the fedora website. When running in ACPI mode you don’t get display output so you’ll need to use a serial console, in both ACPI and Device-Tree mode there’s not currently support for accelerated GPU graphics/AI/ML support. If you want to be able to easily switch between ACPI/Device-Tree modes you’ll want to install the dracut-config-generic package to have a generic initrd to make it easy to reboot between both modes.

Installing Fedora on the NVIDIA Jetson nano

Updated – Aug 2021
You now used the latest R32.6.1 release and it now works with the latest Python releases. Some minor edits below.

Overview
Nvidia launched the Jetson Nano Developer Kit in March 2019, since there there’s been a few minor refreshes including a just announced cheaper 2Gb model. I received the original 4Gb rev A device shortly after they were launched.

Over the last year or so as part of my role at Red Hat I started working with some of the NVidia Tegra team to improve support for the Jetson devices. This work has been wide ranging and though it’s taken awhile, with Fedora 33 we’re starting to see the fruits of that collaboration. The first is improved support for the Jetson Nano. The official L4T (Linux 4 Tegra) Jetson Nano images look a lot like an Android phone with numerous partitions across the mSD card. This makes it harder to support a generic Linux distribution like Fedora as there are assumptions by distributions of control they can have over the storage, so while it was certainly possible to get Fedora to run on these devices it generally wasn’t for the faint of heart. As of the recent L4T releases, you definitely want R32.4.4, it’s now a supported option to flash all the firmware to the onboard SPI flash enabling the use of the entire mSD card for the OS of your choice, which as we all know will be Fedora 😉 but the instructions here should be adaptable to work for any distribution.

Before we begin
We do it in two stages, first is to flash the new firmware to the SPI over the micro USB port, second we’ll prepare the Fedora OS for the mSD card. For the first stage you’ll need the latest L4T Release R32.6.1 and the Fedora U-Boot builds installed locally.

Before we get started you’ll need the following:

  • A USB-A to micro USB cable for flashing
  • A HDMI monitor and a USB keyboard
  • A jumper, a jumper wire or something to close the connection on the FRC pins for recovery mode
  • A 3.3v USB Serial TTY (optional)
  • An appropriate 5v barrel PSU (optional)

If you wish to use a serial TTY there’s a good guide here for connecting it to the RevA nano, the RevB has two camera connectors so they’ve moved the serial console headers to near the mSD card slot. The command to see serial output is:

screen /dev/ttyUSB0 115200

Flashing the Jetson Nano
So let’s get started with flashing the firmware. This step with the firmware on the SPI doesn’t have to be done often. First we’ll extract the L4T release and get all the bits installed that we need to flash the firmware:

sudo dnf install -y usbutils uboot-images-armv8 arm-image-installer
tar xvf ~/Downloads/Jetson-210_Linux_R32.6.1_aarch64.tbz2
cd Linux_for_Tegra
cp /usr/share/uboot/p3450-0000/u-boot.bin bootloader/t210ref/p3450-0000/

Next, based on instructions from the NVidia Jetson Nano Quick Start Guide, we need to put the Jetson Nano into Force Recovery Mode (FRC) to prepare for flashing the firmware:

  1. Ensure that your Jetson Nano Developer Kit is powered off. There’s no need for a mSD card ATM, we’re just writing to the SPI flash.
  2. Connect the Micro-USB OTG cable to the Micro USB port on the Nano. Don’t plug it into the host computer just yet.
  3. Enable Force Recovery mode by placing a jumper across the FRC pins of the Button Header on the carrier board.
    a. For carrier board revision A02, these are pins 3 and 4 of Button Header (J40) which is located near the camera header.
    b. For carrier board revision B01, these are pins 9 and 10 of Button Header (J50), which is located on the edge of the carrier board under the Jetson module.
  4. Only if you wish to use a separate PSU place a jumper across J48 to enable use of a DC power adapter.
  5. Connect a DC power adapter to J25. The developer kit powers on automatically and enters Force Recovery mode. Note it may be possible to do this with USB power but I’ve not tested it.
  6. Remove the jumper from the FRC pins of the Button Header.
  7. See if you can see the Jetson Nano is in recovery mode by running:
    lsusb | grep -i nvidia

Now we can actually flash the firmware (make sure you’re still in the Linux_for_Tegra directory):

sudo ./flash.sh p3448-0000-max-spi external

You will see a lot of output as the command runs, and if you have a serial TTY you’ll see some output there but eventually you’ll be returned to the command prompt and the system will reset. If you have a HDMI monitor attached you’ll see the NVidia logo pop up, if you have a serial console you’ll see a bunch of output and eventually the output of U-Boot and the associated U-Boot prompt.

Jetson TX1 and TX2
You can basically follow the same instructions above for the older TX1/TX2 devices except for two things. For the TX1 you can use the same L4T release, for the TX2 you need to download a different L4T release.

For the U-Boot copy there’s a different U-Boot for each device which needs to be copied to a different location. For the firmware copy I treat the eMMC as if it was the SPI flash, and run the OS off a SD card, it’s not the most efficient but it keeps things more straight forward:

TX1:

cp /usr/share/uboot/p2371-2180/u-boot* bootloader/t210ref/p2371-2180/
sudo ./flash.sh jetson-tx1 mmcblk0p1

TX2:

cp /usr/share/uboot/p2771-0000-500/* bootloader/t186ref/p2771-0000/500/
sudo ./flash.sh jetson-tx2 mmcblk0p1

Getting Fedora running
Now we have the firmware flashed we can prepare Fedora for the mSD card. Download the Fedora Workstation for aarch64 raw image. You can of course also use XFCE, Minimal or Server images. Put the mSD card in reader and after unmounting any filesystem run the following command (look at the help for other options around users/ssh-keys):

sudo arm-image-installer --media=/dev/XXX --resizefs --target=none --image=~/Downloads/Fedora-Workstation-33-1.3.aarch64.raw.xz

Note you need to replace XXX with the right device, and you don’t need a target option as we’re not writing the firmware to the mSD card.

Once that completes you should be able to pop the mSD card into your Jetson Nano and reset the device and see it boot. You will see all the output if you have a serial console attached. If you’re using HDMI it may take a little while once the NVidia logo disappears for the GNOME first user setup to appear.

Also note that while a lot of things work on this device, like the nouveau driver for display, it’s not perfect yet and we’re actively working to fix and improve the support for the Jetson Nano, most of these will come via the standard Fedora update mechanism. If you have queries please engage in the usual ways via the mailing list or #fedora-arm on Libera.Chat.

Three ways to speed up dnf on arm devices

I have a large bunch of Arm Single Board Computers I use for testing a lot. Most of the testing ends up being pretty basic stuff like firmware, kernels, and the various bits of hardware peripherals that people use like storage, network, display and sound output, plus things like sensors and HAT support.

The problem is that these devices often aren’t the fastest in the world for various reasons so I want to be able to apply updates to the basic system as quickly as possible to find out the results. Over time I’ve worked out that these three things speed up dnf quite a bit for the sort of testing I wish to do are as follows:

  1. Disable modularity:
    sed -i 's/enabled=1/enabled=0/' /etc/yum.repos.d/fe*mod*
  2. Don’t install weak dependencies:
    echo "install_weak_deps=False" >> /etc/dnf/dnf.conf
  3. Disable dnf makecache. It never seems to be up to date when you need it anyway:
    systemctl disable dnf-makecache; systemctl mask dnf-makecache

You may need to re-do some of these each major update as they seem to want to force you to have them every time.

Fedora 21 and ARM device support

As we slowly meander our way towards the pointy end of the Fedora 21 release, with Alpha speeding up in the rear view mirror, the Fedora ARM team are starting to discuss the best way to deal with the blossoming amount of ARMv7 devices that can and do run out of the box on Fedora.

With our 3.16 kernel containing device tree blobs for 200+ devices, the Fedora 3.17 rawhide kernel already containing 230+, it’s truly impossible to actively test and support all of those devices. So much like previous releases we’ll be focusing on testing a group of “primary devices” with the remainder being considered as secondary. This doesn’t mean they won’t work, it just means they’re not necessarily a testing focus of the regular contributors or they might not be readily available to purchase.

So what makes a device primary? Well there’s a number of considerations we’ve put into the list. Firstly the device has to be widely available and well supported upstream. Some will notice that some of the devices are no longer widely available (yes Panda, Trimslice and Calxeda I’m looking at you!) and I did consider their removal from the list but given a lot of contributors have them I think it’s worthwhile keeping them around for the moment. The primary devices list won’t be release blocking, we don’t block x86 releases for specific single devices, so I don’t believe we should for ARMv7 either.

Astute readers will notice the proposed primary list of around two dozen devices is much larger than the core devices we supported in Fedora 20! YAY! is all I have to say about that 🙂

The list is not final, at the moment it’s a suggested list and one open for discussion to some degree and what we’ll be heading from Alpha to Beta with. I fully expect it to be tweaked as we go along, there might be cool new shiny Chromebooks 😉 that arrive on the scene and end up working nicely and are hence worth actively supporting (no EXYNOS Chromebooks I’m not looking at you!) and some devices on the list below that end up not making the grade. One thing is for sure the grade includes that they support Cute Embedded Nonsense Hacks ie DeviceTree… there’s no board support here.

Primary:

  • Wandboard (all models/revisions)
  • Utilite (all models)
  • Cubox-i (all models)
  • Hummingboard (all models)
  • RIoTBoard
  • BeagleBone Black
  • Tegra K1 Jetson
  • CubieBoard (all models)
  • Banana Pi
  • Trimslice
  • PandaBoard (all models)
  • Calxeda Highbank/Midway
  • VExpress (qemu)

Secondary:

  • BeagleBone White
  • Beagle xM
  • Novena
  • UDOO
  • AC100
  • Qualcomm (IFC6410, DragonBoard)
  • Various Marvell devices (Mirabox, AX3, CuBox)
  • Various Exynos devices
  • Other AllWinner devices (as per available u-boot/DT support)
  • Gumstix Overo series of devices
  • OMAP5 EVM
  • STE SnowBall

ARM64:

  • AMD Seattle
  • APM Mustang
  • VExpress (qemu)

So what can a user expect from the primary devices above? Will all the functionality of a device work? Well it depends on the specific device and the associated SoC. For example the AllWinner SoC GPU support is far from upstream so unfortunately there will be no graphical UX for those devices, the Tegra K1 support for the GPU isn’t quite there yet but we’re hoping by GA it will be. Some will be better than others in terms of certain features but for example the AllWinner devices would make good storage devices with their SoC attached SATA and Network, no ugly usb storage/network here, so they are useful to support as a primary device and can easily have feature enablement in the F-21 cycle with a “yum upgrade” to a newer kernel.

We’ll delve deeper into the specifics of each device and the final list closer to beta.

3.16 Fedora ARM kernel status

So 3.16 is has quite a few new features in terms of newly supported devices, also some what surprisingly this blog post will be out before 3.16! In terms of new device support all the SoCs listed here are exciting for a number of reasons for Fedora ARM. Aarch64 (ARM64) makes it’s first debut with support of real hardware although we’ve actually had kernel support enable for it for some time in Fedora even if only usable on the glacial Foundation emulator.

The 3.16 release is also very likely to be the kernel that ships with Fedora 21 GA and with the Alpha due in about a month we’re starting to polish and test all the platforms and devices we want to support for GA.

Anyway without any further a do let’s get into the gritty details:

  • NVIDIA Jetson TK1 support: While we’ve had the basics of this for a while all of the bits are there now.
  • EXYNOS support: This SoC is probably the most asked about platform and finally after a long wait the multiplatform support has landed upstream. We currently ship around 20 dtb files for exynos4 and 5 (Chromebook support anyone?). Testing is sought and feedback and greatly appreciated.
  • Qualcomm MSM 8×60, 8960 and 8974 support: While the multiplatform support for these devices landed upstream a few releases a go they’re now to the point they should be relatively usable so it’s time to get wider testing. This should be the beginning of supporting the venerable ifc6410 and dragonboard devices.
  • APM X-GENE support: One of our first aarch64 supported pieces of hardware. Similar to the QCom SoC the initial support has been upstream for a while but with 3.16 it becomes usable with the vast majority of basic support upstream so minimal patches are needed. More on aarch64 soon.
  • AMD Seattle support: The other of our aarch64 supported pieces of hardware if you’re lucky enough to get your mits on a device.

The other feature we’re starting to see mature is GPU and Graphics support. I’m not exactly sure yet as to what the final state of this functionality will be for Fedora 21 GA but we potentially will have suppport for:

  • nouveau/mesa support on the NVIDIA Tegra K1
  • freedreno/mesa support on the Qualcomm boards
  • etnaviv/mesa support on i.MX6 devices
  • improved modesetting support for a number of other devices. Some of this has already landed and is usable in rawhide now.

What covered above is just a high level overview of what’s new in the upcoming release. There’s been numerous other improvements in existing supported SoCs and devices all over the place that would take too long to cover off here but in short with all the shiny that’s landed in 3.16 what Fedora ARM will look like as part of the Fedora 21 GA release is quickly starting to take shape.

3.14 Fedora ARM kernel status

With 3.14.1 out Josh and Justin are preparing to land the 3.14 kernel into Fedora 20. So what does it mean in terms of ARM on Fedora. Well it’s an evolution. There’s of course the usual raft of new devices and some new SoCs, and best of all lots of improvements in support for existing devices. Even the return of some old favourites! Generally from the stash of devices Paul, I and others have that get regularly tested things are looking pretty good with 3.14.

New SoCs and device support:

  • Tegra 4/K1 support has been enabled. For 3.14 this doesn’t mean much as there’s not a wide level of devices out there that we ship device tree blobs for but it’s a good preparation for 3.15 as we should have pretty reasonable support for the nice new Tegra K1 dev board!
  • TI devices: The 3.14 finally brings working support for the OMAP5 EVM board, this will improve further in 3.15 but it boots and generally works. It also adds support for the original BeagleBone and the USB is now finally all back for the Beagle-XM devices so they go back into the fully working list! Also the DT bits for the Overo devices has started to land so if is interested in those I’d love feedback from people with those devices.
  • Freescale i.MX: To this lovely growing list we add initial support for the various Cubox-i devices and the hummingboard. Still no HDMI support but here’s hoping for 3.15
  • Xilinx Zynq 7000: The SD controller has finally landed for this which means they should be bootable to login but from there I’m not sure the status of the rest of the support. We ship DT for the zynq-zc702, zynq-zc706 and zynq-zed devices so if you’ve got one of those and have time to test feedback would be welcome.

Interesting bugs fixed:
Nothing particularly exciting comes to mind here. The fix for the Beagle-XM usb hub power up is nice, as was the final config option for the BeagleBone White. There’s obviously a deluge of other ARM driver fixes and improvements (PTP high precision time support for modular CPSW ethernet on the BeagleBone’s anyone?).

Outstanding bugs and issues:
So this is really the list of items I have outstanding for 3.14 so that I can spin some new images. Feedback on any of these and anything I might not be aware of would be very useful.

  • OMAP DRM display. In 3.11 a new display framework landed for the OMAP DRM driver and in 3.12 the old one was dropped. This broke X on devices like the Pandaboard for 3.12+ kernels. I know roughly the problematic area but I just need to get the cycles to debug this. Any help is welcome.
  • Serial over USB. While this isn’t a kernel bug but rather just needs me to hack together some scripts it’s a blocker for easy OOTB support for devices like the BeagleBone Black
  • Tegra DRM display. After a re-write it’s back and modular again in 3.14. Just need to ensure it’s working and ready to go
  • Testing… testing… testing 🙂

That’s mostly all I have on my list. If there’s anything I’m not aware of please do let me know and I’ll endevour to help out where possible. In particular I’m very interested in boot issues for devices that would would be supportable with new ARM images based on 3.14. From the 3.11 kernel that F-20 GA shipped with there’s been a lot of change and improvements, and while non boot enhancements are easy to do with a “yum update” issues with boot aren’t quite so easy to deal with!