Short history of ARMv7/armhfp/arm32 in Fedora

Back in mid November I proposed a change for Fedora 37 to retire ARMv7 as an architecture, FESCo accepted the proposal. Per the Fedora 36 schedule we branched Fedora 36 this week. Last night I enacted the last of the process to disable it in rawhide so to quote “It’s dead Jim”. The last release of Fedora to support ARMv7 AKA armhfp AKA arm32 will be Fedora 36 which will go end of life around June 2023.

I thought I’d cover a few of the things we achieved with Fedora ARM and some of the impact it’s had on the wider Linux on ARM ecosystem which people may not have realised.

First a little bit of ARM history in the Fedora ecosystem. The beginnings of ARM support actually precedes Fedora all the way back to 1998 with a fork of Red Hat Linux 4.2 and more officially with Red Hat Linux 5.1 on the Corel Netwinder (I always wanted one of those but they weren’t available in Aus).

In Fedora itself the earliest details I remember was that Marvell bootstrapped ARMv5 in Fedora 7 and continued to build and support it through to Fedora 12. This “software architecture” was known as softfp. It was optimised for the ARMv5 architecture which didn’t have a hard requirement on a floating point unit so emulated it when it was needed hence “software floating point”. In Fedora 13 Seneca College took over the ARMv5 infrastructure and building from Marvell. I officially got involved in the Fedora 14 build process and soon after was also contracted by OLPC to drive Fedora on OLPC for their ARM based XO laptops as well as work on their i686 devices to have a single OS for all of them.

In mid 2011, the Fedora 15 timeframe, a small Red Hat team started to do a ARMv7 hard floating point, AKA hardfp or armhfp, bootstrap as ARM’s new v7 mandated a floating point unit. The bootstrap included the core toolchain (binutils/gcc/glibc/elfutils and friends) and ultimately the entire distribution, I drove this effort from a community, build and packaging perspective. This required 100s of patches to upstream projects that made many assumptions about ARM only being softfp, but it also allowed us at the time to fix many general architecture assumptions in these projects. The hard floating point bootstrapping was useful for the wider community too, it was used by Nokia as the base of it’s hardfp efforts for Maemo, plus other distros used it as as it’s much easier/quicker if you already have a full distro running the architecture you wish to boostrap. What wasn’t generally known at the time was also the first new architecture that has been bootstrapped in the Fedora/RHEL ecosystem since x86_64 a long time before and it allowed Red Hat to refresh it’s memory on how to do this in preparation of the then unannounced aarch64 architecture and the POWER Little Endian intentions, basically it provided a cover story. We also worked to get other languages such as Fortran, golang, rust and others building and working on armhfp and those other architectures. The final piece of this was ARMv7 being promoted to a primary Fedora architecture in Fedora 20. This then later went on to my proposal to redefine secondary architectures in Fedora.

In the wider community of Linux Fedora ARM was the first distribution to adopt the kernel “multi platform” work enabling us to go from building 5 different kernels to support a handful of arm devices to a single kernel supporting 100s of devices in a very short period of time. I worked with closely Arnd Bergmann from Linaro on issues with the early pieces of the multiplatform work. In upstream U-Boot we posted the first distro_boot patches to support booting Linux in the same way across all the devices we actively supported so we didn’t need specially wrapped kernels and know exact offsets for every SoC or device. The distro_boot support evolved, working with SUSE, into UEFI support in U-Boot further standardising the ARM boot process by abstracting the pieces that were different and letting the firmware deal with them. This work ultimately evolved into EBBR and the ARM System Ready IR spec. In Fedora 34 we moved to soley supporting UEFI on both ARM architectures. A lot of Linux distros still have specific kernels for each device and use non standard boot methods for devices and hence have an image for each device/use-case they wish to use. This was something Fedora identified very early on as something that would not scale!

Fedora also leads a lot of things in the gcc toolchain stack across all our supported architectures, we’ve actively enabled a lot of security features and other things like LTO early on. As the Fedora gcc maintainers, employed by Red Hat, are also key upstream GCC maintainers we’re almost always the first distribution to rebase onto a new release before it’s a stable release, for example Fedora 36 had just had a mass rebuild against a gcc-12 pre release snapshot. This builds all of the 50k or more source packages with the pre-release of the new toolchain making for a much better release for the wider GCC community because this picks up a number of bugs/regressions in both the general support but also in the architectures Fedora supports which means the ARMv7 hardfp support in GCC has benefited from 100s of bugs we’ve detected in gcc/binutils/glibc etc before they land in a stable gcc release. With the retirement of ARMv7 in Fedora this is going to be something the wider ARMv7 community is going to have to pick up post the GCC-12 release.

Over the subsequent 11 years of ARMv7 support in Fedora, and much longer if you include the early ARMv5 the distribution has also enabled a number of other innovative features like support for containers, support for devices like the Raspberry Pi 2 and 3 in Fedora 25. as mentioned various toolchains, and fun things like robots. Of course we also lose some things too. Devices like the BeagleBone don’t yet have a 64 bit sibling, but there’s less and less 32 bit devices coming out and the use of armhfp is waning quickly and the maintenance cost is rising as the industry moves more generally to 64 bit even in embedded use cases and the fact is with devices like the $15 Raspberry Pi Zero 2W it makes less and less sense even if I do still actively run BeagleBones, a Panda-ES and 3 different i.MX6 devices.

So I engaged with the wider Arm ecosystem and it made sense to finally sunset our ARMv7 32 bit support. We’re of course leaving it in good shape with things like gcc-12, the latest rust and golang toolchains and 5.17 kernels, much newer by the time F-36 goes EOL in June 2023, it will be in good shape if people wish to use it as the basis of some form of continuing ARMv7 supported Linux distribution.

Sail off into the sunset friend, it’s been a fun 12 years of hacking on those projects!

When creating updates remember to build for rawhide and Fedora 25 (devel)

When ever we branch for a new release of Fedora I, and others, end up spending a non trivial amount of time ensuring that there’s a clean upgrade path for packages. From the moment we branch you need to build new versions and bug fixes of packages for rawhide (currently what will become Fedora 26), for the current stabilising release (what will become Fedora 25) as well as what ever stable releases you need to push the fix for. For rawhide you don’t need to submit it as an update but for the current release that’s stabilising you do need to submit it as an update as it won’t just automagically get tagged into the release.

As a packager you should know this, it’s been like it for a VERY LONG TIME! Yet each cycle from the moment of branching right through to when a new release goes GA I still end up having to fix packages that “get downgraded” when people upgrade between releases!!

So far this cycle I’ve fixed about 20 odd with the latest being bash-completion (built but not submitted as an update for F-25) and certmonger (numerous fixes missing from F-25 and master branch).

The other silly packaging bug I end up having to fix quite a bit is NVR downgrades where even though it’s a newer package the way the NVR is handled makes rpm/dnf/yum think the newer package is a lesser version than the current version and hence you’re new shiny fix won’t actually make it to end users. I see this a lot where people push a beta/RC package to a devel (F-25/rawhide) release. Just something to be aware of, there’s lots of good docs around the way rpm/dnf/yum handles eNVR upgrades.

Fedora 24 released on all architectures simultaneously!!

So for the first time ever we’ve released Fedora 24 across both primary and alternate architectures pretty much simultaneously! That’s the three primary architectures, x86_64, ARMv7 and i686, plus the alternate architectures of aarch64, ppc64, ppc64le and s390x. This is the first time we’ve ever released SEVEN architectures on the same day!

Fedora 24 from a Release Engineering perspective has been fairly momentous, we’ve made the single biggest change to our tooling that composes the release since Fedora Core and Fedora Extras was merged in Fedora 7! The plans for this change had been long discussed and first started to move back in the Fedora 21 cycle with the hope it could be implemented in Fedora 22…. boy were we wrong! But on the flip slide, while it’s still not perfect, it has massively improved the process for the releases. We now FINALLY have a full compose every single night! No more Test Composes! It now means QA can automate tests against any bit of the compose output, it means the installer team can see the results of their changes the next day. For the average user this has no material visible impact, for those much closer into the process it has a hugely positive impact!

From the alternate “secondary” architecture perspective I’ve ticked of a massive amount of “goal check boxes” that I made for myself when I joined this role almost two years ago.

My big ticket item was “make the release process like primary”. Our release process was a lot more manual than “primary” but we had also a lot less “features”. Well with this release neither are now true. We’re not 100% of the primary process, but the difference is tiny. We have a full nightly compose now, like primary, where previously we basically had a “newRepo” process we now have a full installer stack with network and iso installs produced. This in itself is a massive improvement! To this we add docker to aarch64 and Power64, cloud to aarch64 (and more automation of cloud to Power64), and initial “tech preview” disk images to aarch64 for Single Board Computers (I’m sorry I really did want to nail down this feature but even I have to sleep!) like the Pine64.

Some of the “smaller” tick box improvements to non primary arches is that we’re now fully 100% ansible run for the aarch64 and Power64 infrastructure, and only have a few minor bits to work out for s390x. This means from the back end everything is 100% like primary and orchestrated as such, the hubs all looks the same and the experience is consistent. Changes are deployed simultaneously and hence consistently. YAY automation! We’ve also simplified the way the secondary content gets to the mirrors so it’s more consistent and faster!

On the not x86 architecture front we’ve got a whole bunch more exciting features planned, and improvements to make, for the Fedora 25 cycle and beyond, some of which will begin happening very soon. Watch this space, the second half of the year looks to me to be just as frantic as the first!!