Short history of ARMv7/armhfp/arm32 in Fedora

Back in mid November I proposed a change for Fedora 37 to retire ARMv7 as an architecture, FESCo accepted the proposal. Per the Fedora 36 schedule we branched Fedora 36 this week. Last night I enacted the last of the process to disable it in rawhide so to quote “It’s dead Jim”. The last release of Fedora to support ARMv7 AKA armhfp AKA arm32 will be Fedora 36 which will go end of life around June 2023.

I thought I’d cover a few of the things we achieved with Fedora ARM and some of the impact it’s had on the wider Linux on ARM ecosystem which people may not have realised.

First a little bit of ARM history in the Fedora ecosystem. The beginnings of ARM support actually precedes Fedora all the way back to 1998 with a fork of Red Hat Linux 4.2 and more officially with Red Hat Linux 5.1 on the Corel Netwinder (I always wanted one of those but they weren’t available in Aus).

In Fedora itself the earliest details I remember was that Marvell bootstrapped ARMv5 in Fedora 7 and continued to build and support it through to Fedora 12. This “software architecture” was known as softfp. It was optimised for the ARMv5 architecture which didn’t have a hard requirement on a floating point unit so emulated it when it was needed hence “software floating point”. In Fedora 13 Seneca College took over the ARMv5 infrastructure and building from Marvell. I officially got involved in the Fedora 14 build process and soon after was also contracted by OLPC to drive Fedora on OLPC for their ARM based XO laptops as well as work on their i686 devices to have a single OS for all of them.

In mid 2011, the Fedora 15 timeframe, a small Red Hat team started to do a ARMv7 hard floating point, AKA hardfp or armhfp, bootstrap as ARM’s new v7 mandated a floating point unit. The bootstrap included the core toolchain (binutils/gcc/glibc/elfutils and friends) and ultimately the entire distribution, I drove this effort from a community, build and packaging perspective. This required 100s of patches to upstream projects that made many assumptions about ARM only being softfp, but it also allowed us at the time to fix many general architecture assumptions in these projects. The hard floating point bootstrapping was useful for the wider community too, it was used by Nokia as the base of it’s hardfp efforts for Maemo, plus other distros used it as as it’s much easier/quicker if you already have a full distro running the architecture you wish to boostrap. What wasn’t generally known at the time was also the first new architecture that has been bootstrapped in the Fedora/RHEL ecosystem since x86_64 a long time before and it allowed Red Hat to refresh it’s memory on how to do this in preparation of the then unannounced aarch64 architecture and the POWER Little Endian intentions, basically it provided a cover story. We also worked to get other languages such as Fortran, golang, rust and others building and working on armhfp and those other architectures. The final piece of this was ARMv7 being promoted to a primary Fedora architecture in Fedora 20. This then later went on to my proposal to redefine secondary architectures in Fedora.

In the wider community of Linux Fedora ARM was the first distribution to adopt the kernel “multi platform” work enabling us to go from building 5 different kernels to support a handful of arm devices to a single kernel supporting 100s of devices in a very short period of time. I worked with closely Arnd Bergmann from Linaro on issues with the early pieces of the multiplatform work. In upstream U-Boot we posted the first distro_boot patches to support booting Linux in the same way across all the devices we actively supported so we didn’t need specially wrapped kernels and know exact offsets for every SoC or device. The distro_boot support evolved, working with SUSE, into UEFI support in U-Boot further standardising the ARM boot process by abstracting the pieces that were different and letting the firmware deal with them. This work ultimately evolved into EBBR and the ARM System Ready IR spec. In Fedora 34 we moved to soley supporting UEFI on both ARM architectures. A lot of Linux distros still have specific kernels for each device and use non standard boot methods for devices and hence have an image for each device/use-case they wish to use. This was something Fedora identified very early on as something that would not scale!

Fedora also leads a lot of things in the gcc toolchain stack across all our supported architectures, we’ve actively enabled a lot of security features and other things like LTO early on. As the Fedora gcc maintainers, employed by Red Hat, are also key upstream GCC maintainers we’re almost always the first distribution to rebase onto a new release before it’s a stable release, for example Fedora 36 had just had a mass rebuild against a gcc-12 pre release snapshot. This builds all of the 50k or more source packages with the pre-release of the new toolchain making for a much better release for the wider GCC community because this picks up a number of bugs/regressions in both the general support but also in the architectures Fedora supports which means the ARMv7 hardfp support in GCC has benefited from 100s of bugs we’ve detected in gcc/binutils/glibc etc before they land in a stable gcc release. With the retirement of ARMv7 in Fedora this is going to be something the wider ARMv7 community is going to have to pick up post the GCC-12 release.

Over the subsequent 11 years of ARMv7 support in Fedora, and much longer if you include the early ARMv5 the distribution has also enabled a number of other innovative features like support for containers, support for devices like the Raspberry Pi 2 and 3 in Fedora 25. as mentioned various toolchains, and fun things like robots. Of course we also lose some things too. Devices like the BeagleBone don’t yet have a 64 bit sibling, but there’s less and less 32 bit devices coming out and the use of armhfp is waning quickly and the maintenance cost is rising as the industry moves more generally to 64 bit even in embedded use cases and the fact is with devices like the $15 Raspberry Pi Zero 2W it makes less and less sense even if I do still actively run BeagleBones, a Panda-ES and 3 different i.MX6 devices.

So I engaged with the wider Arm ecosystem and it made sense to finally sunset our ARMv7 32 bit support. We’re of course leaving it in good shape with things like gcc-12, the latest rust and golang toolchains and 5.17 kernels, much newer by the time F-36 goes EOL in June 2023, it will be in good shape if people wish to use it as the basis of some form of continuing ARMv7 supported Linux distribution.

Sail off into the sunset friend, it’s been a fun 12 years of hacking on those projects!

Fedora on the Pinebook Pro

First thing to note here is that this is not limited to the Pinebook Pro, I’m just using it as the example for 64 bit Rockchip devices with SPI flash on Fedora. This post is focused on devices with SPI but I’ll do a separate follow-up post for other devices including details for writing to eMMC over USB.

The story of Fedora on the Pinebook Pro, and other Rockchip devices, has been a sordid story of a lack of time, bugs, rabbit holes, more bugs and various other things. Not at all sordid at all really, mostly just a lack of time on my behalf, and nobody else stepping up to assist in a way to benefit all Fedora users, mostly they do one time hacks to sort themselves. Overall the support in Fedora for Rockchip devices has been quite solid for a number of releases. The problem has been with the early boot firmware, notable because without SPI flash it wants to splat itself across the first 8Mb of the disk, and if there was SPI flash it generally wasn’t overly stable/straight forward.

Anyway we’re now in a place where devices with SPI flash should mostly work just fine, those devices without it will work with a little manual intervention, and while the support isn’t complete, and will need more polish, they’re all details we can polish with little interruption to users by standard package updates. By default users will have accelerated graphics and from my testing on GNOME 40 it’s by all accounts a pretty decent experience!

Setting up the firmware

First step is to get the firmware written to SPI flash. This is a two step process, the first is to write out a micro SD card from another device, the second is to boot that mSD card on the Pinebook Pro, or another device like the Rockpro64, and write the firmware to the SPI flash.

There’s some nuances to this process, and the way the early boot firmware works, if another version of U-Boot takes precedence that is likely OK as it should still be able to work, the fall back is to use the internal switch to turn off the eMMC temporarily. I also have no idea if the Pine64 shipped U-Boot has any display output, the Fedora build does, if not you’ll need to use the option to disable eMMC or use a serial console cable. Anyway on to the steps:

Set up the mSD card
Use a mSD card that has no data you wish to keep, this process will wipe it out. You want at least U-Boot build 2021.04-3.fc34, you can adjust the umount to be more specific, and you need to substitute XXX for the media, otherwise it’s a relatively quick and straightforward process:

sudo dnf install --enablerepo=updates-testing -y arm-image-installer uboot-images-armv8
sudo umount /run/media//*
sudo spi-flashing-disk --target=pinebook-pro-rk3399 --media=/dev/XXX

Write the firmware to flash
Now remove the mSD card from your host and put it into the Pinebook Pro. Press the power button, from experience you likely need to press and momentarily hold and in a second or two the display will light up with text output. Interrupt the boot by pressing space. Next up we write out the flash:

Hit any key to stop autoboot:  0 
=> ls mmc 1:1
   167509   idbloader.img
   335872   idbloader.spi
   975872   u-boot.itb
  9331712   u-boot-rockchip.bin

4 file(s), 0 dir(s)

=> sf probe
SF: Detected gd25q128 with page size 256 Bytes, erase size 4 KiB, total 16 MiB

=> load mmc 1:1 ${fdt_addr_r} idbloader.spi
335872 bytes read in 39 ms (8.2 MiB/s)

=> sf update ${fdt_addr_r} 0 ${filesize}
device 0 offset 0x0, size 0x52000
61440 bytes written, 274432 bytes skipped in 0.803s, speed 427777 B/s

=> load mmc 1:1 ${fdt_addr_r} u-boot.itb
975872 bytes read in 107 ms (8.7 MiB/s)

=> sf update ${fdt_addr_r} 60000 ${filesize}
device 0 offset 0x60000, size 0xee400
914432 bytes written, 61440 bytes skipped in 9.415s, speed 106127 B/s

Once the last command above has completed eject the mSD card and type reset at the => prompt and the device should reboot and you should see output similar to before but running from the SPI flash!

If you had to turn off the eMMC you can now turn it back on.

Installing Fedora

The nice thing with the firmware on SPI flash it should now work mostly like any other laptop and you can use either the pre canned desktop images (Workstation, KDE, XFCE, Sugar), the Workstation LiveCD iso or the standard everything network installer.

To run the arm Workstation image off a micro SD card or USB stick you can do the following:

arm-image-installer --media=/dev/XXX --resizefs --target=none --image=Fedora-Workstation-34-1.2.aarch64.raw.xz

Note ATM you’ll need to use the USB port on the right hand side, I need to investigate the USB/USB-C port on the left as it appears not to currently work in firmware, but works fine once Fedora is running.

Next steps and improvements

The two biggest issues remaining for the Pinebook Pro is enabling PCIe support (supported from June 2021) and the lack of the brcmfmac firmware, both WiFi and bluetooth, being upstream. For the later issue if there’s anyone from Synaptics that can assist in resolving that problem please reach out to me! A interim WiFi firmware to use is here.

Some things at the Fedora level I’ve not really tested and will do so more, and likely polish with OS updates, in the coming weeks include sound, USB-C port (charging and display output). On the firmware level there’s still some more improvements to be done, tweaks to improve the USB support, turning on the power LED as early as possible to give an indicator, improvements to the EFI framebuffer to ensure consistent early boot output, support for UEFI BGRT to enable smooth boot etc.

For support please email the Fedora Arm mailing list or reach out on IRC via #fedora-arm on Libera.Chat.

Installing Fedora on the NVIDIA Jetson nano

Updated – Aug 2021
You now used the latest R32.6.1 release and it now works with the latest Python releases. Some minor edits below.

Overview
Nvidia launched the Jetson Nano Developer Kit in March 2019, since there there’s been a few minor refreshes including a just announced cheaper 2Gb model. I received the original 4Gb rev A device shortly after they were launched.

Over the last year or so as part of my role at Red Hat I started working with some of the NVidia Tegra team to improve support for the Jetson devices. This work has been wide ranging and though it’s taken awhile, with Fedora 33 we’re starting to see the fruits of that collaboration. The first is improved support for the Jetson Nano. The official L4T (Linux 4 Tegra) Jetson Nano images look a lot like an Android phone with numerous partitions across the mSD card. This makes it harder to support a generic Linux distribution like Fedora as there are assumptions by distributions of control they can have over the storage, so while it was certainly possible to get Fedora to run on these devices it generally wasn’t for the faint of heart. As of the recent L4T releases, you definitely want R32.4.4, it’s now a supported option to flash all the firmware to the onboard SPI flash enabling the use of the entire mSD card for the OS of your choice, which as we all know will be Fedora 😉 but the instructions here should be adaptable to work for any distribution.

Before we begin
We do it in two stages, first is to flash the new firmware to the SPI over the micro USB port, second we’ll prepare the Fedora OS for the mSD card. For the first stage you’ll need the latest L4T Release R32.6.1 and the Fedora U-Boot builds installed locally.

Before we get started you’ll need the following:

  • A USB-A to micro USB cable for flashing
  • A HDMI monitor and a USB keyboard
  • A jumper, a jumper wire or something to close the connection on the FRC pins for recovery mode
  • A 3.3v USB Serial TTY (optional)
  • An appropriate 5v barrel PSU (optional)

If you wish to use a serial TTY there’s a good guide here for connecting it to the RevA nano, the RevB has two camera connectors so they’ve moved the serial console headers to near the mSD card slot. The command to see serial output is:

screen /dev/ttyUSB0 115200

Flashing the Jetson Nano
So let’s get started with flashing the firmware. This step with the firmware on the SPI doesn’t have to be done often. First we’ll extract the L4T release and get all the bits installed that we need to flash the firmware:

sudo dnf install -y usbutils uboot-images-armv8 arm-image-installer
tar xvf ~/Downloads/Jetson-210_Linux_R32.6.1_aarch64.tbz2
cd Linux_for_Tegra
cp /usr/share/uboot/p3450-0000/u-boot.bin bootloader/t210ref/p3450-0000/

Next, based on instructions from the NVidia Jetson Nano Quick Start Guide, we need to put the Jetson Nano into Force Recovery Mode (FRC) to prepare for flashing the firmware:

  1. Ensure that your Jetson Nano Developer Kit is powered off. There’s no need for a mSD card ATM, we’re just writing to the SPI flash.
  2. Connect the Micro-USB OTG cable to the Micro USB port on the Nano. Don’t plug it into the host computer just yet.
  3. Enable Force Recovery mode by placing a jumper across the FRC pins of the Button Header on the carrier board.
    a. For carrier board revision A02, these are pins 3 and 4 of Button Header (J40) which is located near the camera header.
    b. For carrier board revision B01, these are pins 9 and 10 of Button Header (J50), which is located on the edge of the carrier board under the Jetson module.
  4. Only if you wish to use a separate PSU place a jumper across J48 to enable use of a DC power adapter.
  5. Connect a DC power adapter to J25. The developer kit powers on automatically and enters Force Recovery mode. Note it may be possible to do this with USB power but I’ve not tested it.
  6. Remove the jumper from the FRC pins of the Button Header.
  7. See if you can see the Jetson Nano is in recovery mode by running:
    lsusb | grep -i nvidia

Now we can actually flash the firmware (make sure you’re still in the Linux_for_Tegra directory):

sudo ./flash.sh p3448-0000-max-spi external

You will see a lot of output as the command runs, and if you have a serial TTY you’ll see some output there but eventually you’ll be returned to the command prompt and the system will reset. If you have a HDMI monitor attached you’ll see the NVidia logo pop up, if you have a serial console you’ll see a bunch of output and eventually the output of U-Boot and the associated U-Boot prompt.

Jetson TX1 and TX2
You can basically follow the same instructions above for the older TX1/TX2 devices except for two things. For the TX1 you can use the same L4T release, for the TX2 you need to download a different L4T release.

For the U-Boot copy there’s a different U-Boot for each device which needs to be copied to a different location. For the firmware copy I treat the eMMC as if it was the SPI flash, and run the OS off a SD card, it’s not the most efficient but it keeps things more straight forward:

TX1:

cp /usr/share/uboot/p2371-2180/u-boot* bootloader/t210ref/p2371-2180/
sudo ./flash.sh jetson-tx1 mmcblk0p1

TX2:

cp /usr/share/uboot/p2771-0000-500/* bootloader/t186ref/p2771-0000/500/
sudo ./flash.sh jetson-tx2 mmcblk0p1

Getting Fedora running
Now we have the firmware flashed we can prepare Fedora for the mSD card. Download the Fedora Workstation for aarch64 raw image. You can of course also use XFCE, Minimal or Server images. Put the mSD card in reader and after unmounting any filesystem run the following command (look at the help for other options around users/ssh-keys):

sudo arm-image-installer --media=/dev/XXX --resizefs --target=none --image=~/Downloads/Fedora-Workstation-33-1.3.aarch64.raw.xz

Note you need to replace XXX with the right device, and you don’t need a target option as we’re not writing the firmware to the mSD card.

Once that completes you should be able to pop the mSD card into your Jetson Nano and reset the device and see it boot. You will see all the output if you have a serial console attached. If you’re using HDMI it may take a little while once the NVidia logo disappears for the GNOME first user setup to appear.

Also note that while a lot of things work on this device, like the nouveau driver for display, it’s not perfect yet and we’re actively working to fix and improve the support for the Jetson Nano, most of these will come via the standard Fedora update mechanism. If you have queries please engage in the usual ways via the mailing list or #fedora-arm on Libera.Chat.

Three ways to speed up dnf on arm devices

I have a large bunch of Arm Single Board Computers I use for testing a lot. Most of the testing ends up being pretty basic stuff like firmware, kernels, and the various bits of hardware peripherals that people use like storage, network, display and sound output, plus things like sensors and HAT support.

The problem is that these devices often aren’t the fastest in the world for various reasons so I want to be able to apply updates to the basic system as quickly as possible to find out the results. Over time I’ve worked out that these three things speed up dnf quite a bit for the sort of testing I wish to do are as follows:

  1. Disable modularity:
    sed -i 's/enabled=1/enabled=0/' /etc/yum.repos.d/fe*mod*
  2. Don’t install weak dependencies:
    echo "install_weak_deps=False" >> /etc/dnf/dnf.conf
  3. Disable dnf makecache. It never seems to be up to date when you need it anyway:
    systemctl disable dnf-makecache; systemctl mask dnf-makecache

You may need to re-do some of these each major update as they seem to want to force you to have them every time.

Raspberry Pi improvements in Fedora 29

So Fedora 29 is probably going to account for the largest single improvement to support on the Raspberry Pi support in Fedora since we added initial support in Fedora 25. It certainly wasn’t without issue, but after quite a bit of debug we’ve got the post release issues with the WiFi back to being stable!

WiFi improvements
The support for upstream NVRAM files and the ability to add those files to linux-firmware means we get WiFi support for the Raspberry Pi 3 Series of devices out of the box! No need to grab anything, it just works! Well mostly, we had some issues with WiFi being very intermittent, as well as a missed bug around aarch64 but now with the 4.19.10 kernel everything appears to be working and stable. This makes me very happy and it took longer than I had hoped but we’re there. This device specific NVRAM driver support will also help another bunch of cheap Arm and x86 based devices that ship with Broadcom/Cyprus based WiFi support moving forward.

ZRAM enabled by default
By supporting and enabling ZRAM swap by default we get a more responsive device and less wear on the MicroSD storage. Over all we’ve seen reasonable performance improvements and to no date no major issues.

GNOME performance improvements
In May 2018 the Raspberry Pi Foundation kindly hosted a GNOME Performance Hackfest in the lovely Cambridge. Over a couple of days we managed to fix a number of issues seen, review and document a number of issues and work on a number of ways of reducing the memory usage of GNOME. Of course this improvement is primarily seen constrained devices like the Raspberry Pi but ultimately less memory utilisation by GNOME even helps devices with decent amounts of RAM and CPUs too. The fixes didn’t arrive in time for Fedora 28, but a bunch have landed in Fedora 29 providing noticeable improvements, and the GNOME team is by no means done and there will be more coming in Fedora 30 and beyond! It was an excellent start and I expect there will be ongoing enhancements here into the future especially with devices like the Purism Phone which will have similar constraints.

Initial CPU frequency support
Another of the largest issues around the Raspberry Pi is complaints it was slow, part of the issue here is that there’s no upstream CPU Frequency driver which means all models of the Raspberry Pi run at a glacial, but safe, 600Mhz out of the box compared to the highest speed, which on the 3B+ is 1400Mhz. With Fedora 29 we’ve landed an experimental cpufreq driver which allows us to run the Raspberry Pi 3-Series at much closer to optimal speeds. While this is experimental it might not stay around if we find out it causes issues or ends up being a maintenance burden but to date it hasn’t yet appeared to have caused any issues.

HWmon Voltage Sensor
There’s a new driver that reports when the voltage supplied by the PSU drops below the required voltage. It can be a bit noisy in dmesg but one of the biggest support problems we have with the Raspberry Pis is people using a power supply that’s not powerful enough, this issue is more of a problem with Fedora 29 because with the support for running at faster frequencies due to the cpufreq driver it means we also draw more power and some PSUs that were previously fine now cause issues because they can’t supply enough current.

Enhanced support for config.txt
A lot of the hardware addons are supported in Raspbian are done by enabling things in the config.txt file, this in turn does things like loading DT overlays and merging them with the base DT to enable extra hardware like HAT support. We have enhanced the way Fedora works with this which enables us to be much closer to the way Raspbian handles these things. The advantage this has is that the documentation that’s written for Raspbian is then usable by Fedora in the wider Raspberry Pi ecosystem which in turn makes it easier for end users to get HW up and running due to less differences in process. There’s further enhancements to make here but every step closer is easier for everyone to enable and use their favourite HATs.

Improved bcm283x firmware support
In preparation for grub2 support we enhanced how we deal with the firmware that the Raspberry Pi uses for booting. This deals with the early startup. We never use to upgrade it by default to ensure things didn’t break, but it also meant most users also didn’t by default get the fixes and enhancements. Now we do. The config.txt is also handled directly which means if you never edit the file you now automatically get any changes we make, because rpm handles the file as a config file, if we change it you get a .rpmnew file so you won’t lose your changes.

Camera support
This wasn’t available in the Fedora 29 4.18 kernels, but with the rebase to the 4.19 kernel the support for the camera on the Raspberry Pi CSI Camera interface improved enough we could enable this in Fedora. The early 4.19 kernels don’t automatically detect and load support if the camera module is attached. There’s some patches in 4.20 in rawhide for this, and we’ll bring some of this to 4.19 soon, and we’re working with upstream to further improve the camera support. You’ll also want the latest bcm283x firmware which tweaks some of the config.txt and updates to a firmware with ISP fixes.

Another improvements
There was also a number of general Arm improvements which sped up crypto on the Raspberry Pi, improved the USB, fixed up some issues with the wired ethernet on the 3B+, power and a number of other fixes. As always there’s more coming. The 4.20 kernel rebase should also bring with it analog sound support early in the new year.

Conclusion
Overall I was pleased with the work that landed in Fedora over 2018 for the Raspberry Pi. The WiFi regression was disappointing, but now with that fixed in 4.19.10 we have WiFi support out of the box without users needing to download anything which moving forward will make things a lot more straight forward. The initial support for the camera makes it much more useful in numerous use cases and we’ll really polish up the HAT support in Fedora 30 which for me is the last remaining big ticket item for Raspberry Pi support. There’s still some annoying bits around the EDID detection in the display, but there’s work to improve that upstream, and also there’s work to land the media decode offloading upstream too which will also one of the few remaining bits.

Fedora on the UDOO Neo

Some time ago I backed the UDOO Neo Kickstarter as it looked like a nifty, well featured, IoT device. I got the full option which came with 1Gb RAM and both wired and wireless Ethernet and some add-on sensors. It was a well run kickstarter campaign and the device was well packaged with a fab box. It has both a Cortex-A9 processor to run Fedora and a Cortex-M4 embedded processor to enable you to do Arduino style functionality which should be interesting to experiment with.

For various reasons it has sat around gathering dust, it’s been a bit of a long drawn out process with me randomly poking it as time allowed.. Primarily this was because there was no decent upstream U-Boot and kernel support, and I’d not had the time to hack that up myself from various downstream git repositories, but even without Fedora support their forked Ubuntu distro in the form of UDOObuntu has an experience that is truly terrible!

Late 2016 the problem of a lack of upstream support for U-Boot and kernel changed with initial basic support landing upstream for all three (Basic, Extended and Full) models so with a few cycles over a weekend it was time to dust it off to see if I could get Fedora 26 (did I mention this has been long running?) running on it and to see what worked.

The first thing for me to do was to setup a serial console for easy debugging. The UDOO Neo documentation is generally outstanding and the pins for the UART1 TTL are documented. Two things to note here is that the headers are female rather than the usual SBC male pins so I had to bodge my usual usb to serial TTL with some male-male jumper wires and you’ll need a ground for the TTL which is undocumented on their page, I used one of the GNDs as documented on connector J7 and all was good.

So after an initial set of fixes to the U-Boot support it saw my Fedora install and started to boot! Success! Well sort of, as mentioned above the initial support is rudimentary, it started to boot the kernel and very quickly managed to corrupt and destroy the filesystem not making it much beyond switch root. That wasn’t good. In the last week or two I’ve had a little time to look again, similar issues, it was better than it was a year or so ago but it still ended up with corruption. I reached out to one of the maintainers from NXP that deals with a bunch of the i.MX platforms and I got directed to a handful of patches, a test kernel and image later and a test boot… all the way to initial-setup! SUCCESS!

The core support for the i.MX6SX SoC and the UDOO Neo is pretty reasonable, with the MMC fixes it’s been very stable, all the core bits are working as expected, included wired and wireless network, thermal, cpufreq, crypto and it looks like the display should work fine. There’s a few quirks that I need to investigate further which should provide for a fun evening or weekend hacking. There has also been recently merged support for the i.MX6SX Cortex-M4 land upstream in Zephyr upstream for the 1.13 release, so getting that running and communication using Open-AMP between Fedora and Zephyr should also be an interesting addition. I think this will be a welcome addition to Fedora 29, and not a moment too soon!!

The Raspberry Pi 3 B+ in Fedora

So I’m sure none of you are surprised to hear that I’ve been asked a bunch about support for the Raspberry Pi 3 B+ in Fedora. Well the good news is that it’ll be supported in Fedora 28. Most of the bits were there for the official Fedora 28 beta, it just needed a minor work around, but nightly images since Beta have had all the bits integrated so the upcoming Fedora 28 GA release will support the Raspberry Pi 3 B+ to the same levels as the original 3 B on both ARMv7 and aarch64. The Fedora Raspberry Pi FAQ has now been updated with all the details of both the RPi3+ and Fedora 28.

WiFi

As with the original 3 there’s files with the firmware we can’t redistribute. The details are documented in the Fedora Raspberry Pi FAQ.

You can grab the files like for the 3 although there’s now an extra one, which you don’t really need, but it gives you all the 802.11a frequencies:

$ sudo curl https://fedora.roving-it.com/brcmfmac43455-sdio.txt -o /lib/firmware/brcm/brcmfmac43455-sdio.txt
$ sudo curl https://fedora.roving-it.com/brcmfmac43455-sdio.clm_blob -o /lib/firmware/brcm/brcmfmac43455-sdio.clm_blob

I’ve also done a rpm of the files for both editions of the RPi3 (plus a few other brcm adapters used on ARM boards). You can either just grab the rpm or setup the repo if you want to get the latest if I update it:

$ sudo curl https://fedora.roving-it.com/wireless.repo -o /etc/yum.repos.d//wireless.repo
$ sudo dnf install brcm-firmware

Issues

Like all new things there’s often a few teething problems. Over all the support for the new RPi 3 B+ is relatively solid but like all new devices there’s some bits still bedding down, and the combination of a new device and new OS there’s been a few minor issues that have been seen in some circumstances. We pulled in the latest firmware just before GA to fix some issues but no doubt as it gets wider testing both in Fedora and in the wider Raspberry Pi community more issues may well come up.

The ones we’re aware of are:

  • The new USB hub and gigbit ethernet interface have seen a few issues in some cases. We’ve pulled in quite a few patches to help stabilise the NIC in Linux and it now mostly works in the vast majority of use cases.
  • The USB hub in u-boot, uEFI and grub on aarch64 can cause some issues. If there’s too many USB devices connected it sometimes won’t boot or will do so really slowly. The work around for the moment is to disconnect all the USB devices until Fedora has started to boot and then plug them in.

In Fedora we’ll deliver updates and fixes by the usual updates, in particular as fixes land upstream we’ll review and land them where useful into Fedora, more than likely via a kernel update. If you see a uboot-tools or a bcm283x-firmware update you’ll want to run the rpi-firmware-update command to update the firmware and then reboot for it to take effect.

Older releases

We won’t be supported the new device in the older releases. Why I hear you say? Well it’s possible but it needs update to the firmware, U-Boot and kernel to work. The Raspberry Pi foundation respun Raspbian to support it and basically it’s not straight forward. Much better to have a new shiny Fedora for a new shiny device!

Getting started with Fedora on the 96boards Dragonboard

Support for this board has been a long time coming, it was originally announced in March 2015 and shipped later that summer. Two years on we can finally add support for it to Fedora. The enablement here will also assist us with supporting the newly announced 600c and 820c boards more quickly. We’re not all the way there yet, there’s still some firmwares that needs to go upstream into linux-firmware, but the improvement is fantastic and it’s been a pleasure working with the 96boards and Qualcomm teams getting to where we are today.

At the moment we support running Fedora off either an micro SD card or a USB stick. We don’t currently support running off the eMMC and currently basically treat that as the location of the firmware. Anyway lets get started!

Updating the firmware

You’ll want to update to the latest firmwares, my board originally had an old firmware without support for PSCI and so it didn’t bring up all four cores or support reboot. OOPS! You’ll need the latest linux rescue images from the 96boards download site. As I write this the latest is the 17.09 release (version 88). Create a directory for this file before you unzip it because it’ll expand all into the current directory. While there we also need a u-boot build that’s prepared for flashing, the upstream support isn’t quite complete, we add a few patches to the Fedora build to get everything working nicely. You can grab a pre-built version here and also get LK firmware build which enables display output.

You’ll need a host with the fastboot utility, in Fedora this is found in the android-tools package, and a micro USB cable. This process is very similar to flashing a phone with a new image, not surprising given the chipset really. If you have a serial console on the board you can follow along on the console but it’s not required for this board.

To put the board into fastboot mode we hold down the volume down button, labeled as ‘(-)’ near the middle USB port and then power it on. Wait around 30 seconds to ensure it’s booted to fastboot. You can test this with the fastboot devices command. You’ll likely want to run the next commands as root, or use sudo, and be in the directory you created with the extracted firmware and u-boot build:

sudo ./flashall
sudo fastboot flash aboot emmc_appsboot.mbn
sudo fastboot flash boot u-boot.img
sudo fastboot oem select-display-panel adv7533_1080p

The flashall command runs a series of fastboot command to write out various early boot firmware to the eMMC, then we write u-boot out to the boot partition, and finally ensure that output is configured to appear on the HDMI port. Assuming you don’t get any errors from fastboot that should be all the firmware done and in place.

Fedora image and further setup

Next up is the Fedora image. I chose the Workstation image, but we also have a Minimal Image and a traditional Server image. GNOME not the fastest in the world as 1Gb of RAM isn’t really enough for GNOME-3 anymore, but it works well enough. On a USB stick or Micro-SD card (I’ve tried both). We need to write out the image, then expand the rootfs (Note: update XXX for the device you’re writing to):

xzcat Fedora-Workstation-27-1.6.aarch64.raw.xz | sudo dd status=progress bs=4M of=/dev/XXX
sudo gparted /dev/XXX (expand the last partition)
partprobe

Next up we need to adjust the kernel command line slightly, mount up the first partition and edit /EFI/fedora/grub.cfg and search for the string cma=256MB and delete it, then add in it’s place the following console=tty1 console=ttyMSM0,115200n8. Next mount the boot partition (partition 2) and create a sym link

ln -s dtb-4.13.9-300.fc27.aarch64 dtb

. Unmount the partitions and we should be good to go on the Dragonboard.

Plug in a keyboard, mouse (and/or a usb cable for the serial console if you’re going that route) and a HDMI cable, plug in the USB stick or SD card and power it up. If you’re following along on the serial console you should see output straight away, screen might take a little longer.

Once you’ve booted you should be able to complete initial-setup (text or the one from Workstation) and login. To get the WiFI and Bluetooth working you need to install a Radio (WiFi and friends) firmware package which I’ve made into a rpm you can grab from here until it lands into linux-firmware.

What next?

The DragonBoard 410c is pretty functional. I’ve not widely tested sound, the Venus media offload components (we have all the firmware and kernel bits for this), the GPS or some of the other more advanced components but I’ll have more details about those soon. I’ll be documenting the above plus other bits on the Fedora ARM wiki so keep an eye on that or get involved and help out 😛

Overview of aarch64 SBC support in Fedora 27

Support for ARM 64 bit (aarch64) Single Board Computers (SBCs) has been one of the most highly requested features along side the Raspberry Pi. It’s something I’ve been working towards almost as long too. Finally with Fedora 27 I felt we would have enough of the bits in place for a Minimum Viable Product (MVP).

You’ll note from the Fedora change linked above I was very cautious with what we planned to achieve. The change has a very focused list of images: Server, Workstation and Minimal and a limited list of devices: basically the Raspberry Pi 3, the 96boards Dragonboard 410c and HiKey, and a handful of AllWinner devices with a focus on the Pine64 series of boards. The reason for this was I knew there was going to be a lot of low level boot and kernel bits that needed focus and polish and the Fedora 27 cycle was severely limited time and resource wise so the plan was to focus on getting all the core bits into place for Fedora 27 and have a couple of well polished devices and then expand that rapidly for Fedora 28.

The key functionality we were aiming for was a well polished uEFI implementation in u-boot to enable a single install/boot path in Fedora on aarch64 using uEFI/shim/grub2 to boot Fedora on both SBCs and SBSA compliant aarch64 platforms. We now have that platform in place, primarily due to Herculean efforts of Rob Clark and Peter Jones, as well as many others who have provided insight into the deep dark details of the uEFI specification. Fedora 27 will ship with a quite heavily patched, well by Fedora’s standards anyway, u-boot 2017.09 which provides us the core of this functionality enabling us to use a vanilla upstream shim and grub2 to boot a standard Fedora. All this work is already upstream, or making it’s way there in 2017.11. In Fedora 28 there will be even more improvements that will enable us to do a bunch of other cool stuff (that’ll be a post for later!) and also enable much quicker upstream board enablement now all the core bits are in place.

So what do we actually support? Well all the usual bits that you would expect on a standard Fedora install, whether it be x86_64, ARMv7 or aarch64, like SELinux, containers, desktops and all the other bits. There’s a few bits and pieces that are a little rough around the edges but overall the feature is pretty robust. On a board by board feature set lets break the this down across the boards:

Raspberry Pi 3

The support for the Raspberry Pi3 is the equivalent to the ARMv7 support but with boot via uEFI/grub2. The memory isn’t quite as good as on 32-bit but that’s to be expected, overall it’s pretty reasonable for a device of the specs and cost. Like on 32 bit support we’re seeing regular improvements each release and throughout the releases. The aarch64 support for the RPi3 is just an evolution to this.

DragonBoard 410c

The support for the DragonBoard 410c is looking pretty decent. Qualcomm has been doing a pretty decent effort to get stuff upstream, we have firmwares for the GPU and for video decode/encode upstream as well, along with kernel drivers and the open freedreno 3D drivers, HDMI audio should work as well. The WiFi firmware isn’t yet upstream but I’ll document how/where to get that and hopefully that should be in linux-firmware soon as well. Overall I’m quite happy with the status of this device, although like all devices with 1Gb RAM it’s a little constrained, but that should make the newly announced 820c with 3Gb of RAM a decent device ;-). All the details for getting it running will soon be in the Fedora 96boards wiki page.

HiKey

Most features and functionality of the HiKey are supported, note this isn’t the HiKey960 (look to F-28 for support for that), except accelerated graphics due to the use of a MALI GPU. Other than that the functionality is pretty decent. You’ll likely want the latest tianocore firmware and the details for that can be found on the Fedora 96boards wiki page.

Pine64 (AllWinner A64 SoC)

We actually should have a number of devices based on the AllWinner A64 SoC working here but we’ve only tested the 3, 2Gb/1Gb/512Mb, Pine64 device sizes. The support for these devices is headless and you will need a serial console else you’re on your own as none of the display bits in the kernel have made it upstream, and of course the GPU is a MALI 400 series so when it does it won’t be fast. The support for the rest of the device is basic, it’s usable for a headless server style device, we support network, USB, KVM, RTC and a few other bits. Other than display we don’t yet support the SDIO attached wireless, sound, crypto offload or any of the other media interfaces. A lot of this is under review upstream so I think Fedora 28 should look much better for this series of devices and 4.15 might even bring very basic console output. Speaking of series of devices which ones should actually work other than the three Pine64 devices? Well the following A64 SoC devices have a Fedora built u-boot and kernel DT support so should work as well as the Pine64: BananaPi-m64, OrangePi Win, SoPine baseboard (PineBook boots if you’re happy with serial console), NanoPi-A64 and the A64-OLinuXino. We had some troubles with the AllWinner H5 SoC devices earlier in the cycle but I’ve had a couple of reports that it seems to be resolved so they should work too and that adds the Orange Pi PC2, Prime and Zero+ 2 as well as the NanoPi NEO2. So that’s around a dozen or so devices! 🙂

Other ARM64 SBCs

I’ve had reports that other aarch64 SBCs boot on Fedora just fine. I’ve not listed those where I can’t verify whether they boot with our uEFI enabled u-boot. Looking around on my desk I do have a number of devices that I expect us to be supporting in Fedora 28, or maybe even just enabling u-boot bits in a F-27 update.

Overall I’m pretty happy with the state of Aarch64 SBCs for Fedora 27 and what we’ve managed to achieve is such a short cycle!

Why I’m not backing the Purism Librem 5 phone

NOTE: This is a post about my opinion on the device, hence the title of “Why I’m not backing….”, people have been explicitly asking me why I’ve not backed it, this documents it so I don’t need to keep repeating myself!

Numerous people have come up to me and asked “So will you get Fedora to run on your Librem phone?” and when my response is “No, I’ve not backed it” I get weird looks with a question of “Why?” I had thought it was time to do document my concerns with this laudable venture. This was certainly further qualified when I had to inform someone from the EFF that they’re delirious about the “it doesn’t need closed source firmware” on the i.MX chips that are being proposed. While I applaud the general principals and ideas of a fully open rights protecting phone I can’t help but feel that the group doing it either are being false at worst or naive at best with some of their statements.

Firmware

Their site claims “The i.MX 6/8 CPU will be completely free software without any binaries whatsoever!” while this could be sort of true if you want a hobbled device it’s not really the case at all. There’s a number of firmwares that are needed to make a number of pieces of functionality of a mobile device useful. Firstly the SDMA driver needs a firmware to run at any level of reasonable speed. Secondly the accelerated media decode, which will be required if you actually want to consume meda on your phone and have more than moments of battery life, also needs firmware. You’ll note on the media decode I don’t reference the actual firmware! Why? Because it’s not actually distributed in the linux-firmware repository so you have to request it from NXP or somewhere with appropriate signups (while I’m not 100% sure I believe having a driver in the linux kernel without required firmware in linux-firmware is a breach of the requirements of said driver in the upstream kernel).

i.MX6 or i.MX8 SoC

The i.MX 6 or i.MX 8 option concerns me. Basically make a choice! The core issue I have is the i.MX6 SoC is ancient, being announced back in Jan 2011, and based on a Cortex-A9 SoC. It doesn’t support USB-C so to ship as promised they’ll need to add a PCI-e attached USB-3 controller, charging circuitry, and maybe even a LVDS to Display Port option plus a chip to MUX the three through the actual USB-C port if they want to be able to ‘dock’ which will make power consumption on the phone even worse! If I’d spent $600 and received a phone in January 2019 based on an eight year old 32-bit chip design I’d be seriously pissed off. To be quite honest the i.MX8 is no spring chicken either, being announced in September 2013, it’s based on the already quite long in the teeth Cortex-A53 SoC design which was the original aarch64 “little” low end design but it does at least support USB-C on the SoC. The i.MX SoCs are generally quite nice, and the i.MX6 is quite well supported upstream, but the whole line of SoCs are more targeted towards embedded applications like cars, so they do have a long support cycle. They’re not a mobile phone focused SoC though, they also tend to be quite slow to get moving in the market/availability and the i.MX8 has little upstream support in the kernel as yet. Sure the etnaviv driver enables an open 3D accelerated driver, one that isn’t supported by the vendor so doesn’t enable all the features upstream, but you can run this today on Fedora 26/27 on the i.MX6 SoCs now but a free GPU driver is not the only reason to choose a platform.

General concerns

I can’t help but feel that there’s going to be a lot of disappointed people that will end up receiving an expensive sub standard device, probably late, that ends up not taking us much further along the road, to a fully open rights protecting phone, than the days of the Nokia n9xx series phones running Maemo. The HW, if based on an i.MX6, will certainly not be much further along that route and still stuck in the past in terms of HW. I personally believe the project would be better off engaging with the Qualcomm community team to use the Qualcomm 820c/600c SoCs because there’s an open driver that Qualcomm are working to improve, and while there’s a need for firmware for GPU and media offload, in reality it’s no better/worse than the i.MX devices with the bonus that their devices that are more current and aimed at mobile workloads with the vendor actively working to upsteam the enhance the support support of their SoCs.

Ultimately I think a device that closely resembles the specs of a phone of recent history, than that of ancient smart phone history, is likely to get a better following and hence a better software ecosystem than one that’s the same era as the Motorola DROID 4.

Update: So just to clarify a few things that were bought up in a couple of threads:

First this reddit thread:

  • I don’t question the value of hardware isolation of the various wireless interfaces that they claim, or the ability of them being able to deliver that bit on what ever SoC they choose, that is why it wasn’t addressed above.
  • I bought up USB-C because it is explicitly claimed as a feature.
  • I bought up display port because given USB-C above and their “It can be a desktop computer and phone all-in-one” claim, including a nice picture, then DP over USB-C is the only real option for that functionality.
  • Qualcomm chips were an example of other SoCs I believe might be a better fit, yes they do provide options in their APQ line without built in radios, they were just an example of another option, there are other possibilities, it wasn’t meant to provide a guarantee, it was an alternative example.
  • Yes, all ARM processors have onboard boot firmware, it’s generically referred to as “PBL” (Primary Boot Loader), so do most other processors.

Second this purism thread:

  • Sure you can disable the media engine with an e-FUSE but media is kind of useful for a lot of use cases like video config, audio calls and music, not just watching videos. If they choose this route, I would hope they leave the fuses unblown and document how a end user can do it else the $600 is a whole lot less useful for a lot of people

Ultimately the main points is I have to make about all of the above is two fold, firstly it’s my opinions and why I didn’t back the device, and secondly there appears to be some large discrepancies in the statements they make about the device and SoCs which I would not have expected and that is the key problem for me because it causes concerns over the ability to deliver a working and useful device to me. $600 is not a small amount of money, for me at least, to hedge on what *might* be a i.MX8 or might be an old 32 bit i.MX6.