Any Linux distro on NVIDIA Jetson Orin with JetPack 6

NVIDIA has just released the Jetpack 6 Developer Preview for the NVIDIA Jetson Orin hardware. The thing that is most exciting about this release is they finally support the ability to use upstream kernels and other Linux distributions. This means you can start to use both RHEL (9.3 and later) and Fedora on the Jetson Orin hardware! This has been a LONG time coming, something I’ve been involved with for 5 years!

So while this is a developer preview, AKA public Beta, it’s still very usable and for people that are interested in using other Linux distributions now is the time to get stuck in. Like all things it’s not perfect and there’s still work to be done, but many hands do make light work!

You start by downloading the BSP from there you can follow the following instructions and you should end up with a device you can easily install Fedora 39 or RHEL 9.3 or other distros with the appropriate bits enabled.

To flash the firmware you need to follow the Orin AGX guide for recovery and cabling, for Orin NX/nano you need to use the HW pins near the mSD card, to put the device in recovery mode and cabling then do the following for Orin AGX:

$ tar zvf Jetson_Linux_R36.2.0_aarch64.tbz2
$ cd Linux_for_Tegra/
$ lsusb|grep -i nv
Bus 003 Device 044: ID 0955:7045 NVIDIA Corp. [unknown]
Bus 003 Device 045: ID 0955:7023 NVIDIA Corp. [unknown]
$ sudo ./flash.sh p3737-0000-p3701-0000-qspi external
removed a lot of output
*** The target generic has been flashed successfully. ***
Make the target filesystem available to the device and reset the board to boot from external external.
$

The command for other Orin devices such as NX and Nano will be similar, you’ll just have to swap the p3737 variable, eg for Orin Nano use: sudo ./flash.sh p3737-0000-p3701-0000-qspi external.

Once the flash completes the device will reboot and you will be able to use the usual mechanisms to install your OS, whether the RHEL or Fedora installers or a Fedora Arm image. I’ve tested running OSes off both the microSD and a NVME card, plus installing off USB, the DisplayPort output should work in EFI console mode. The firmware is based upon the widely known TianoCore/EDK2 so the firmware interface should be straight forward. For those that may need a serial console if it’s not automatically detected you can use console=ttyAMA0,115200, This runs off the microUSB port on /dev/ttyACM2 on the host device.

For hardware vendors that have hardware based on the NVIDIA Orin hardware they will be able to adopt and make this available to their customers that may wish to run distributions other than L4T. If they are unsure feel free to reach out to me in the usual locations.

Using fwupdmgr to update NVME firmware

The fabulous fwupdmgr provides the ability to easily update firmware that is published to Linux Vendor Firmware Service (LVFS) but it can also be used to apply updates that aren’t necessarily in LVFS. One type of firmware that it supports updating is NVME firmware, that’s basically any NMVE, because the standard specifies a standardised mechanism for updating the firmware on all NVME devices.

I had a need to update a NVME firmware in an aarch64 device to see if it fixed an issue I was seeing. The Crucial P2 supported options were of course x86 only. The ISO download actually contained a little LinuxOS in an initrd on the .iso. The advice from Richard the fwupd technical lead was to “Look for a ~4mb high entropy blob” so mounting it up, I mounted the iso, extracted the initrd, and then used fwupdmmgr to apply the new firmware.

Find the NVME and check the firmware version:

$ cat /sys/class/nvme/nvme0/firmware_rev 
P2CR010 

So once I’d downloaded the update file I did the following to extract and update the firmware. Note I did this all as root, you can do most of it as non root.

# unzip iso_p2cr012.zip
# mount -o loop iso_p2cr012.iso /mnt/
# mkdir ~/tmp
# cp /mnt/boot/corepure64.gz tmp/
# cd tmp
# gunzip corepure64.gz
# cpio -iv < corepure64
# fwupdtool install-blob opt/firmware/P2CR012/1.bin
Loading…                 [-                                      ]
Loading…                 [-                                      ]
Choose a device:
0.	Cancel
1.	71b677ca0f1bc2c5b804fa1d59e52064ce589293 (CT250P2SSD8)
2.	2270d251f7c1dc37a29a2aa720a566aa0fa0ecde (spi1.0)
1
Waiting…                 [************************************** ] Less than one minute remaining…
An update requires a reboot to complete. Restart now? [y|N]: y

And away it goes, a reboot later and did it work?

$ cat /sys/class/nvme/nvme0/firmware_rev 
P2CR012

YES!!

Using nmcli to configure a static dual stack wired network interface

I recently managed to break the network on my VM that hosts this blog. Basically I removed the NetworkManager-initscripts-ifcfg-rh package because I don’t use the old style ifcfg configuration anywhere else and I had forgotten how long I’d had this VM. So I went into the web console, manually bought up the network with ip commands and reinstalled the package but it made no difference. Oh well! Time to just move it to the new config so I just worked out the nmcli options for all the bits in the old ifcfg. This VM network is nothing special, it’s basically dual IPv4/IPv6 interface with associated DNS.

Step 1: Show existing connections:

$ sudo nmcli c
NAME  UUID                                  TYPE      DEVICE 
eth0  a603bba7-fad8-3c71-9d4c-2cd5dc50e114  ethernet  eth0   

Step 2: Delete existing connection:

$ sudo nmcli c del a603bba7-fad8-3c71-9d4c-2cd5dc50e114

Step 3: Create a new connection (Note the IP addresses are random, the DNS servers are the Google public ones):

$ sudo nmcli c add type ethernet ifname eth0 con-name eth0 mac 80:00:00:ab:cd:ef ip4 192.168.10.6/24 gw4 192.168.10.1 ip6 fe80::b257:377c:e7b3:29ed/64 gw6 2A03:B0C0:0003:00D0:0000:0000:0000:0001 ipv4.dns "8.8.8.8 8.8.4.4" ipv6.dns "2001:4860:4860::8888 2001:4860:4860::8844"

Now the blog is back! The new connection is stored in /etc/NetworkManager/system-connections/eth0.nmconnection

Changing ssh ports on Fedora or RHEL

I always forget the exact commands to change the port ssh (or any default service in the case of the selinux bits) runs on. It’s nicely simple though!

Edit /etc/ssh/sshd_config to change the port number:

Port 2022

You can add a second line if you wish to initially leave it running on Port 22 too in case something goes wrong, obviously don’t forget to remove it once the new port is working!

To add port 2022 to port contexts, enter:
# semanage port -a -t ssh_port_t -p tcp 2022

You can verify new settings, enter:

# semanage port -l | grep ssh
Sample outputs:
ssh_port_t tcp 2022,22

Reload the sshd service to pick up the new config:
#systemctl reload sshd.service

And of course don’t forget to update your firewall to allow the new port through.

network bridge for libvirt host with a single live IP

The default network config for libvirt is simple and works for most basic use cases but there’s a number of use cases where you need more complex config like Adam outlined for a local bridged config.

I run a number of VMs on a hosted server on the internet and I’ve had on my ToDo list for some time to add a IPSEC site to site VPN but the default network doesn’t make that easy because libvirtd deals with the iptables networking including the NAT automagically.

The network config looks like this:

                    *-----*
192.168.100.0/24  --| Hyp |-(eth0)- internet
 (br0) VM net       *-----*

Create non routed network bridge
Initially create a basic network bridge and disable STP (spanning tree protocol). Note we don’t bind it with eth0 which is a public internet facing interface.

nmcli con add type bridge ifname br0
nmcli con modify bridge-br0 bridge.stp no

I then edited the /etc/sysconfig/network-scripts/ifcfg-bridge-br0 file and to add IP network config and adjust a few bits to get the following:

DEVICE=br0
STP=no
TYPE=Bridge
BOOTPROTO=static
DEFROUTE=yes
IPV4_FAILURE_FATAL=no
IPADDR=192.168.100.254
NETMASK=255.255.255.0
IPV6INIT=no
IPV6_FAILURE_FATAL=no
NAME=bridge-br0
UUID=
ONBOOT=yes

Once we’ve done that we can bring the bridge online and check the config looks OK:

ifup br0
nmcli c show
nmcli -f bridge con show bridge-br0
ip addr

Now we have a network bridge with an IP address you can now edit any VM configuration and reassign the virtual NICs to the new bridge and adjust the VM network config to the new subnet and assign static IPs to each VM, or configure dhcpd to assign IPs on the br0 interface. Once that’s done you should be able to ping the gateway (192.168.101.254) and have local network connectivity.

Once you’ve moved everything over you can delete the original libvirtd network config.

Outbound NATed networking
Using the traditional iptables.service (firewalld investigation is on my todo list) you can add a basic outbound NAT configuration which restores the last of the missing functionality with the following basic rule set which will NAT by masquerading the br0 network out through the public IP on eth0:

iptables -t nat -A POSTROUTING -o eth0 -j MASQUERADE
iptables -A INPUT -m state --state RELATED,ESTABLISHED -j ACCEPT
iptables -A INPUT -p icmp -j ACCEPT
iptables -A INPUT -i lo -j ACCEPT
iptables -A INPUT -m state --state NEW -p tcp -m tcp --dport 22 -j ACCEPT
iptables -A INPUT -j REJECT --reject-with icmp-host-prohibited
iptables -A FORWARD -i eth0 -o br0 -m state --state RELATED,ESTABLISHED -j ACCEPT
iptables -A FORWARD -i br0 -o eth0 -j ACCEPT
iptables -A FORWARD -j REJECT --reject-with icmp-host-prohibited
iptables-save > /etc/sysconfig/iptables
systemctl enable iptables.service
systemctl start iptables.service

With the network now under complete control of NetworkManager and a IP firewall/NAT configuration under control from a single point it now makes it easier to add things like IPSEC connections and IPv6 configuration both of which are next on the list.

using ssh keys with screen

It always annoyed me I couldn’t use my ssh key in a screen session. Every now and again I would try and work it out with google and some trial and error. Eventually with the help of a couple of good bits off the net I worked out what I thought to be the easiest way to achieve it consistently.

Firstly the ssh config bits:

Add the following to your ~/.ssh/config file, creating it if you don’t already have one:

host *
  ControlMaster auto
  ControlPath ~/.ssh/master-%r@%h:p

And create the ~/.ssh/rc file:

#!/bin/bash
if test "$SSH_AUTH_SOCK" ; then
    ln -sfv $SSH_AUTH_SOCK ~/.ssh/ssh_auth_sock
fi

And make sure they have the correct permissions for ssh:

chmod 600 ~/.ssh/config ~/.ssh/rc

Finally add the following to your ~/.screenrc file:

setenv SSH_AUTH_SOCK $HOME/.ssh/ssh_auth_sock

I’m not sure it’s the best and most effective way but it’s nice and simple and to date it’s been working well for me, I’ve not had issues with it. Any suggestions for improvement feel free to comment.

Disable SSLv3 in Dovecot

Disabling SSLv3 in Dovecot is nice and straight forward.

In the /etc/dovecot/conf.d/10-ssl.conf file edit the ssl_cipher_list line to look as below (or adjust to suit your specific requirements):

ssl_cipher_list = ALL:!ADH:!LOW:!SSLv2:!SSLv3:!EXP:!aNULL:+HIGH:+MEDIUM

To test the option to ensure it’ll work you can run the following command before you restart dovecot and the output should look something like below:

$ openssl ciphers -v 'ALL:!ADH:!LOW:!SSLv2:!SSLv3:!EXP:!aNULL:+HIGH:+MEDIUM'
ECDHE-RSA-AES256-GCM-SHA384 TLSv1.2 Kx=ECDH     Au=RSA  Enc=AESGCM(256) Mac=AEAD
ECDHE-ECDSA-AES256-GCM-SHA384 TLSv1.2 Kx=ECDH     Au=ECDSA Enc=AESGCM(256) Mac=AEAD
ECDHE-RSA-AES256-SHA384 TLSv1.2 Kx=ECDH     Au=RSA  Enc=AES(256)  Mac=SHA384
ECDHE-ECDSA-AES256-SHA384 TLSv1.2 Kx=ECDH     Au=ECDSA Enc=AES(256)  Mac=SHA384
DHE-DSS-AES256-GCM-SHA384 TLSv1.2 Kx=DH       Au=DSS  Enc=AESGCM(256) Mac=AEAD
DHE-RSA-AES256-GCM-SHA384 TLSv1.2 Kx=DH       Au=RSA  Enc=AESGCM(256) Mac=AEAD
DHE-RSA-AES256-SHA256   TLSv1.2 Kx=DH       Au=RSA  Enc=AES(256)  Mac=SHA256
DHE-DSS-AES256-SHA256   TLSv1.2 Kx=DH       Au=DSS  Enc=AES(256)  Mac=SHA256
ECDH-RSA-AES256-GCM-SHA384 TLSv1.2 Kx=ECDH/RSA Au=ECDH Enc=AESGCM(256) Mac=AEAD
ECDH-ECDSA-AES256-GCM-SHA384 TLSv1.2 Kx=ECDH/ECDSA Au=ECDH Enc=AESGCM(256) Mac=AEAD
ECDH-RSA-AES256-SHA384  TLSv1.2 Kx=ECDH/RSA Au=ECDH Enc=AES(256)  Mac=SHA384
ECDH-ECDSA-AES256-SHA384 TLSv1.2 Kx=ECDH/ECDSA Au=ECDH Enc=AES(256)  Mac=SHA384
AES256-GCM-SHA384       TLSv1.2 Kx=RSA      Au=RSA  Enc=AESGCM(256) Mac=AEAD
AES256-SHA256           TLSv1.2 Kx=RSA      Au=RSA  Enc=AES(256)  Mac=SHA256
ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH     Au=RSA  Enc=AESGCM(128) Mac=AEAD
ECDHE-ECDSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH     Au=ECDSA Enc=AESGCM(128) Mac=AEAD
ECDHE-RSA-AES128-SHA256 TLSv1.2 Kx=ECDH     Au=RSA  Enc=AES(128)  Mac=SHA256
ECDHE-ECDSA-AES128-SHA256 TLSv1.2 Kx=ECDH     Au=ECDSA Enc=AES(128)  Mac=SHA256
DHE-DSS-AES128-GCM-SHA256 TLSv1.2 Kx=DH       Au=DSS  Enc=AESGCM(128) Mac=AEAD
DHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=DH       Au=RSA  Enc=AESGCM(128) Mac=AEAD
DHE-RSA-AES128-SHA256   TLSv1.2 Kx=DH       Au=RSA  Enc=AES(128)  Mac=SHA256
DHE-DSS-AES128-SHA256   TLSv1.2 Kx=DH       Au=DSS  Enc=AES(128)  Mac=SHA256
ECDH-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH/RSA Au=ECDH Enc=AESGCM(128) Mac=AEAD
ECDH-ECDSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH/ECDSA Au=ECDH Enc=AESGCM(128) Mac=AEAD
ECDH-RSA-AES128-SHA256  TLSv1.2 Kx=ECDH/RSA Au=ECDH Enc=AES(128)  Mac=SHA256
ECDH-ECDSA-AES128-SHA256 TLSv1.2 Kx=ECDH/ECDSA Au=ECDH Enc=AES(128)  Mac=SHA256
AES128-GCM-SHA256       TLSv1.2 Kx=RSA      Au=RSA  Enc=AESGCM(128) Mac=AEAD
AES128-SHA256           TLSv1.2 Kx=RSA      Au=RSA  Enc=AES(128)  Mac=SHA256

Finally to test it from client side you can run the following command to ensure it’s not enabled. If it’s configured as expected the negotiation should fail and it’ll return you straight to the command prompt:

openssl s_client -connect mail.example.com:993 -ssl3

Configure Firefox for Kerberos SSO on Fedora or RHEL

Single Sign On (SSO) is one of those nirvana things that rarely happens and a lot of people don’t understand (a central single username/password is not SSO people if you have to keep entering the password!!) but it’s very useful when it does work. With Firefox on Fedora or RHEL it works but it does take a little bit of config to do so efficiently.

  1. In the Firefox address bar type about:config to display the list of current configuration options.
  2. Filter the options with ‘negotiate’ to reduce the list of options.
  3. Double-click the network.negotiate-auth.trusted-uris entry and add the kerberos auth domain, eg: “.example.com” and click OK.
  4. Repeat the process for the network.negotiate-auth.delegation-uris entry, using the same domain.

Now you need to ensure you have a kerberos ticket. If you’ve configured an “Enterprise Login” using the GNOME Online Accounts check you have a ticket with the ‘klist’ command from a terminal window. If you don’t have a valid ticket run ‘kinit’ and from there you should be able to visit SSO enable sites and be auto logged in. Magic!

Simple WOL with RHEL and Fedora

I have a number of systems I use at home, they run a combination of RHEL and Fedora and are a variety of architectures (well x86 and ARM). Travelling a lot for $dayjob I don’t always want them on needlessly chewing power but on the flip side if I’m away for two weeks and I’d like to remotely use one and it’s off it’s a little tough to power it up if you’re 12 hours travel away!

So the obvious solution to this problem to get the best of both worlds is to use Wake On LAN (WOL) for supported hardware which means I can leave devices off or suspended and then just bring them up to full power when I actually want to use them from where ever I happen to be at the time, whether it be on the couch across the room or the other side of the world.

There’s a few places where you need to configure the device you wish to wake up. If you want to wake from power off you need to configure the BIOS/Firmware to have WOL enabled, this differs from device to device so this bit is an exercise for the reader. The quickest way to determine if your hardware supports WOL and its enabled in the BIOS/Firmware is to check to see if when the device is turned off and cabled up to a switch is if there’s a link light on the NIC is still lit.

Once you’ve got the HW and BIOS/Firmware sorted the next bit is pretty straight forward. First enable WOL on boot for the NIC in /etc/sysconfig/network-scripts/ifcfg- by adding the following line to the end of the file:

ETHTOOL_OPTIONS="wol g"

Then run

# ethtool -s  wol g

to enable it now. Finally record the ether address from the output from the “ip link” command so you know what MAC address you need to use to wake it up.

On your gateway make sure you have the appropriate tool to initiate the remote wakeup. On Fedora and RHEL this is ether-wake and it’s in the net-tools package. It’s then as simple as running

# ether-wake -i local-interface mac-address

puppetmaster on apache with passenger in 5 mins

Config management, DevOPs and all those sorts of time saving things are all well and good for readily repeatable tasks that happen on the same systems on a regular basis. But as a consultant my tasks are rarely the same and change from day to day depending on the customer, their mood or the phase of the moon. So while I’ve used puppet a lot I’ve only deployed a puppet platform once or twice and every time I do I end up digging around trying to work out the latest best practice.

So primarily to remind myself so I have a quick guide for the next time here’s my quick puppetmaster in five minutes using just rpm packages (so they’re easily upgraded and checked for security issues etc) on RHEL-6 or Fedora.

First ensure you have proper forward and reverse DNS records for the puppet master server and all puppet clients.

  1. Set SELinux to Permissive. This is a FIXME as there appears to be an issue running puppetmaster on rails on apache. A quick check it seems to be related to comms between httpd and rails. RHBZ 730837 has more info but I need to investigate but it should be straight forward .
    sed 's/SELINUX=enforcing/SELINUX=permissive/' /etc/sysconfig/selinux
    setenforce permissve
    
  2. On Fedora you already have all the repos required out of the box. On RHEL I’m using the upstream PuppetLabs repos so I used the following:
    rhn-channel -a -c rhel-x86_64-server-optional-6
    rpm -ivh http://yum.puppetlabs.com/el/6/products/x86_64/puppetlabs-release-6-10.noarch.rpm
    rpm -ivh http://dl.fedoraproject.org/pub/epel/6/x86_64/epel-release-6-8.noarch.rpm
    yum-config-manager --enable epel puppetlabs-products puppetlabs-deps
    
  3. Install the needed packages
    yum install puppet-server httpd mod_ssl rubygem-rack mod_passenger
    
  4. Start the puppetmaster and set it to run on startup. While we won’t be using WEBrick we start it initially so it will initialise the default puppet config and setup things like certificates.
    service puppetmaster start
    puppet resource service puppetmaster ensure=running enable=true
    
  5. Configure the puppetmaster Rails app
    mkdir -p /usr/share/puppet/rack/puppetmasterd
    mkdir /usr/share/puppet/rack/puppetmasterd/public /usr/share/puppet/rack/puppetmasterd/tmp
    cp /usr/share/puppet/ext/rack/config.ru /usr/share/puppet/rack/puppetmasterd/
    chown -R puppet.puppet /usr/share/puppet/rack
    restorecon -Rv /usr/share/puppet
    mkdir /var/run/passenger
    restorecon -Rv /var/run/passenger
    
  6. Configure apache by adding the config below and updating FQDN.pem to your local certificate name.
    vim /etc/httpd/conf.d/puppetmaster.conf

    # RHEL/CentOS:
    # And the passenger performance tuning settings:
    PassengerHighPerformance On
    PassengerUseGlobalQueue On
    # Set this to about 1.5 times the number of CPU cores in your master:
    PassengerMaxPoolSize 6
    # Recycle master processes after they service 1000 requests
    PassengerMaxRequests 1000
    # Stop processes if they sit idle for 10 minutes
    PassengerPoolIdleTime 600
    PassengerTempDir /var/run/passenger
    
    Listen 8140
    <VirtualHost *:8140>
        SSLEngine On
    
        # Only allow high security cryptography. Alter if needed for compatibility.
        SSLProtocol             All -SSLv2
        SSLCipherSuite          HIGH:!ADH:RC4+RSA:-MEDIUM:-LOW:-EXP
        SSLCertificateFile      /var/lib/puppet/ssl/certs/FQDN.pem
        SSLCertificateKeyFile   /var/lib/puppet/ssl/private_keys/FQDN.pem
        SSLCertificateChainFile /var/lib/puppet/ssl/ca/ca_crt.pem
        SSLCACertificateFile    /var/lib/puppet/ssl/ca/ca_crt.pem
        SSLCARevocationFile     /var/lib/puppet/ssl/ca/ca_crl.pem
        SSLVerifyClient         optional
        SSLVerifyDepth          1
        SSLOptions              +StdEnvVars +ExportCertData
    
        # These request headers are used to pass the client certificate
        # authentication information on to the puppet master process
        RequestHeader set X-SSL-Subject %{SSL_CLIENT_S_DN}e
        RequestHeader set X-Client-DN %{SSL_CLIENT_S_DN}e
        RequestHeader set X-Client-Verify %{SSL_CLIENT_VERIFY}e
    
        RackAutoDetect On
        DocumentRoot /usr/share/puppet/rack/puppetmasterd/public/
        <Directory /usr/share/puppet/rack/puppetmasterd/>
            Options None
            AllowOverride None
            Order Allow,Deny
            Allow from All
        </Directory>
    </VirtualHost>
    

    restorecon -Rv /etc/httpd/conf.d

  7. Disable the puppetmaster so it runs via Rails on Apache
    chkconfig puppetmaster off; service puppetmaster stop
    chkconfig httpd on; service httpd restart
    
  8. Setup a basic puppet config files:
    • autosign.conf

      vim /etc/puppet/autosign.conf

      *.infra.example.com
      
    • site.pp
      vim /etc/puppet/manifests/site.pp

      # /etc/puppet/manifests/site.pp
      
      import "nodes"
      
      # The filebucket option allows for file backups to the server
      filebucket { main: server => 'puppetmaster.infra.example.com' }
      
      # Set global defaults - including backing up all files to the main filebucket and adds a global path
      File { backup => main }
      Exec { path => "/usr/bin:/usr/sbin:/bin:/sbin" }
      
    • modules.pp
      vim /etc/puppet/manifests/modules.pp

      # /etc/puppet/manifests/modules.pp
      
      import "dns-client"
      import "ntp-client"
      
    • nodes.pp
      vim /etc/puppet/manifests/nodes.pp

      # /etc/puppet/manifests/nodes.pp
      
      node 'common' {
              include dns-client
              include ntp-client
      }
      
      node default inherits common {
      }
      
      node 'puppetmaster.infra.example.com' inherits common {
      }
      

    With the above you obviously need to have the dns-client and ntp-client modules in place in /etc/puppet/modules/ or update the config files to reflect the modules you have installed!

  9. Relabel all the various puppet directories to ensure we’re good with selinux:
    restorecon -Rv /etc/puppet
  10. Reboot to ensure it all works when it comes back online 😉

P.S. I’ll update this post once I’ve had time to work out the SELinux issues, pointers welcome in the comments below 🙂