Short history of ARMv7/armhfp/arm32 in Fedora

Back in mid November I proposed a change for Fedora 37 to retire ARMv7 as an architecture, FESCo accepted the proposal. Per the Fedora 36 schedule we branched Fedora 36 this week. Last night I enacted the last of the process to disable it in rawhide so to quote “It’s dead Jim”. The last release of Fedora to support ARMv7 AKA armhfp AKA arm32 will be Fedora 36 which will go end of life around June 2023.

I thought I’d cover a few of the things we achieved with Fedora ARM and some of the impact it’s had on the wider Linux on ARM ecosystem which people may not have realised.

First a little bit of ARM history in the Fedora ecosystem. The beginnings of ARM support actually precedes Fedora all the way back to 1998 with a fork of Red Hat Linux 4.2 and more officially with Red Hat Linux 5.1 on the Corel Netwinder (I always wanted one of those but they weren’t available in Aus).

In Fedora itself the earliest details I remember was that Marvell bootstrapped ARMv5 in Fedora 7 and continued to build and support it through to Fedora 12. This “software architecture” was known as softfp. It was optimised for the ARMv5 architecture which didn’t have a hard requirement on a floating point unit so emulated it when it was needed hence “software floating point”. In Fedora 13 Seneca College took over the ARMv5 infrastructure and building from Marvell. I officially got involved in the Fedora 14 build process and soon after was also contracted by OLPC to drive Fedora on OLPC for their ARM based XO laptops as well as work on their i686 devices to have a single OS for all of them.

In mid 2011, the Fedora 15 timeframe, a small Red Hat team started to do a ARMv7 hard floating point, AKA hardfp or armhfp, bootstrap as ARM’s new v7 mandated a floating point unit. The bootstrap included the core toolchain (binutils/gcc/glibc/elfutils and friends) and ultimately the entire distribution, I drove this effort from a community, build and packaging perspective. This required 100s of patches to upstream projects that made many assumptions about ARM only being softfp, but it also allowed us at the time to fix many general architecture assumptions in these projects. The hard floating point bootstrapping was useful for the wider community too, it was used by Nokia as the base of it’s hardfp efforts for Maemo, plus other distros used it as as it’s much easier/quicker if you already have a full distro running the architecture you wish to boostrap. What wasn’t generally known at the time was also the first new architecture that has been bootstrapped in the Fedora/RHEL ecosystem since x86_64 a long time before and it allowed Red Hat to refresh it’s memory on how to do this in preparation of the then unannounced aarch64 architecture and the POWER Little Endian intentions, basically it provided a cover story. We also worked to get other languages such as Fortran, golang, rust and others building and working on armhfp and those other architectures. The final piece of this was ARMv7 being promoted to a primary Fedora architecture in Fedora 20. This then later went on to my proposal to redefine secondary architectures in Fedora.

In the wider community of Linux Fedora ARM was the first distribution to adopt the kernel “multi platform” work enabling us to go from building 5 different kernels to support a handful of arm devices to a single kernel supporting 100s of devices in a very short period of time. I worked with closely Arnd Bergmann from Linaro on issues with the early pieces of the multiplatform work. In upstream U-Boot we posted the first distro_boot patches to support booting Linux in the same way across all the devices we actively supported so we didn’t need specially wrapped kernels and know exact offsets for every SoC or device. The distro_boot support evolved, working with SUSE, into UEFI support in U-Boot further standardising the ARM boot process by abstracting the pieces that were different and letting the firmware deal with them. This work ultimately evolved into EBBR and the ARM System Ready IR spec. In Fedora 34 we moved to soley supporting UEFI on both ARM architectures. A lot of Linux distros still have specific kernels for each device and use non standard boot methods for devices and hence have an image for each device/use-case they wish to use. This was something Fedora identified very early on as something that would not scale!

Fedora also leads a lot of things in the gcc toolchain stack across all our supported architectures, we’ve actively enabled a lot of security features and other things like LTO early on. As the Fedora gcc maintainers, employed by Red Hat, are also key upstream GCC maintainers we’re almost always the first distribution to rebase onto a new release before it’s a stable release, for example Fedora 36 had just had a mass rebuild against a gcc-12 pre release snapshot. This builds all of the 50k or more source packages with the pre-release of the new toolchain making for a much better release for the wider GCC community because this picks up a number of bugs/regressions in both the general support but also in the architectures Fedora supports which means the ARMv7 hardfp support in GCC has benefited from 100s of bugs we’ve detected in gcc/binutils/glibc etc before they land in a stable gcc release. With the retirement of ARMv7 in Fedora this is going to be something the wider ARMv7 community is going to have to pick up post the GCC-12 release.

Over the subsequent 11 years of ARMv7 support in Fedora, and much longer if you include the early ARMv5 the distribution has also enabled a number of other innovative features like support for containers, support for devices like the Raspberry Pi 2 and 3 in Fedora 25. as mentioned various toolchains, and fun things like robots. Of course we also lose some things too. Devices like the BeagleBone don’t yet have a 64 bit sibling, but there’s less and less 32 bit devices coming out and the use of armhfp is waning quickly and the maintenance cost is rising as the industry moves more generally to 64 bit even in embedded use cases and the fact is with devices like the $15 Raspberry Pi Zero 2W it makes less and less sense even if I do still actively run BeagleBones, a Panda-ES and 3 different i.MX6 devices.

So I engaged with the wider Arm ecosystem and it made sense to finally sunset our ARMv7 32 bit support. We’re of course leaving it in good shape with things like gcc-12, the latest rust and golang toolchains and 5.17 kernels, much newer by the time F-36 goes EOL in June 2023, it will be in good shape if people wish to use it as the basis of some form of continuing ARMv7 supported Linux distribution.

Sail off into the sunset friend, it’s been a fun 12 years of hacking on those projects!

Three ways to speed up dnf on arm devices

I have a large bunch of Arm Single Board Computers I use for testing a lot. Most of the testing ends up being pretty basic stuff like firmware, kernels, and the various bits of hardware peripherals that people use like storage, network, display and sound output, plus things like sensors and HAT support.

The problem is that these devices often aren’t the fastest in the world for various reasons so I want to be able to apply updates to the basic system as quickly as possible to find out the results. Over time I’ve worked out that these three things speed up dnf quite a bit for the sort of testing I wish to do are as follows:

  1. Disable modularity:
    sed -i 's/enabled=1/enabled=0/' /etc/yum.repos.d/fe*mod*
  2. Don’t install weak dependencies:
    echo "install_weak_deps=False" >> /etc/dnf/dnf.conf
  3. Disable dnf makecache. It never seems to be up to date when you need it anyway:
    systemctl disable dnf-makecache; systemctl mask dnf-makecache

You may need to re-do some of these each major update as they seem to want to force you to have them every time.

My ARM grab bag device list

They say the first step of coming to terms with addiction is admitting you have a problem… I have a problem with collecting ARM devices… there I said it! How big is this problem you ask? How about I list them out and let you decide!

I’ll break the list down into categories as I believe it’s big enough to do so :-/

The aarch64 set of devices currently stands at:

  • 2x Applied Mustang (different x-gene SoC revs)
  • AMD Seattle
  • 96boards HiKey (hi6220)

The ARMv7 boards list is currently:

  • Compulabs Trimslice (tegra-2)
  • Toshiba AC100 (tegra-2)
  • nVidia Jetson TK1 (tegra-124)
  • Acer Chromebook (tegra-124)
  • BeagleBoard xM (omap3)
  • Nokia n900 (omap3)
  • Nokia n950 prototype (omap3)
  • BeagleBone (am33xx)
  • BeagleBone Black (am33xx) x3
  • BeagleBone Green (am33xx)
  • PandaBoard ES Prototype (omap4)
  • PandaBoard ES B2 (omap4)
  • CubieBoard (allwinner-a10)
  • CubieTruck (allwinner-a20)
  • Banana Pi (allwinner-a20)
  • C.H.I.P. Alpha x2 (allwinner-r8)
  • Snowball (u8500)
  • Compulabs Utilite (imx6q)
  • WandBoard Quad revb (imx6q)
  • novena board (imx6q)
  • RIoTboard (imx6s)
  • UDOO Neo (imx6sx)
  • Origen (exynos-4)
  • OLPC XO 1.75 – a number of variants (mmp2) xNumerous
  • OLPC XO-4 including XO-Touch (mmp3) xNumerous
  • Linksys WRT1900AC (armada-xp)
  • Mirabox (armada-370)
  • ifc6410 (qcom)
  • Parallella Board (zynq7000)
  • Raspberry Pi 2 x3

The Cortex-M series for IoT sensors is currently:

  • TI SensorTag 2015
  • ARM mBed IoT starter kit
  • BeeWi SmartClim

Other random related bits:

  • BeagleBone Breadboard Prototyping Cape x2
  • BeagleBone CryptoCape
  • Original 256Mb Raspberry Pi model B
  • Grove starter kit for BeagleBone Green
  • Explorer HAT
  • PiGlo HAT
  • TI CC2531 802.15.4 USB dongle x3
  • numerous random sensors

So the list above is the devices that I use for hacking on. I count 41 without listing out the dozen or so ARM based XOs I have (various prototypes and models). I also don’t have in that list phones, tablets and two drones as I don’t really hack on those as it’s not like with the list above I don’t already have enough toys! So do I have a problem?

Fedora 21 and ARM device support

As we slowly meander our way towards the pointy end of the Fedora 21 release, with Alpha speeding up in the rear view mirror, the Fedora ARM team are starting to discuss the best way to deal with the blossoming amount of ARMv7 devices that can and do run out of the box on Fedora.

With our 3.16 kernel containing device tree blobs for 200+ devices, the Fedora 3.17 rawhide kernel already containing 230+, it’s truly impossible to actively test and support all of those devices. So much like previous releases we’ll be focusing on testing a group of “primary devices” with the remainder being considered as secondary. This doesn’t mean they won’t work, it just means they’re not necessarily a testing focus of the regular contributors or they might not be readily available to purchase.

So what makes a device primary? Well there’s a number of considerations we’ve put into the list. Firstly the device has to be widely available and well supported upstream. Some will notice that some of the devices are no longer widely available (yes Panda, Trimslice and Calxeda I’m looking at you!) and I did consider their removal from the list but given a lot of contributors have them I think it’s worthwhile keeping them around for the moment. The primary devices list won’t be release blocking, we don’t block x86 releases for specific single devices, so I don’t believe we should for ARMv7 either.

Astute readers will notice the proposed primary list of around two dozen devices is much larger than the core devices we supported in Fedora 20! YAY! is all I have to say about that 🙂

The list is not final, at the moment it’s a suggested list and one open for discussion to some degree and what we’ll be heading from Alpha to Beta with. I fully expect it to be tweaked as we go along, there might be cool new shiny Chromebooks 😉 that arrive on the scene and end up working nicely and are hence worth actively supporting (no EXYNOS Chromebooks I’m not looking at you!) and some devices on the list below that end up not making the grade. One thing is for sure the grade includes that they support Cute Embedded Nonsense Hacks ie DeviceTree… there’s no board support here.

Primary:

  • Wandboard (all models/revisions)
  • Utilite (all models)
  • Cubox-i (all models)
  • Hummingboard (all models)
  • RIoTBoard
  • BeagleBone Black
  • Tegra K1 Jetson
  • CubieBoard (all models)
  • Banana Pi
  • Trimslice
  • PandaBoard (all models)
  • Calxeda Highbank/Midway
  • VExpress (qemu)

Secondary:

  • BeagleBone White
  • Beagle xM
  • Novena
  • UDOO
  • AC100
  • Qualcomm (IFC6410, DragonBoard)
  • Various Marvell devices (Mirabox, AX3, CuBox)
  • Various Exynos devices
  • Other AllWinner devices (as per available u-boot/DT support)
  • Gumstix Overo series of devices
  • OMAP5 EVM
  • STE SnowBall

ARM64:

  • AMD Seattle
  • APM Mustang
  • VExpress (qemu)

So what can a user expect from the primary devices above? Will all the functionality of a device work? Well it depends on the specific device and the associated SoC. For example the AllWinner SoC GPU support is far from upstream so unfortunately there will be no graphical UX for those devices, the Tegra K1 support for the GPU isn’t quite there yet but we’re hoping by GA it will be. Some will be better than others in terms of certain features but for example the AllWinner devices would make good storage devices with their SoC attached SATA and Network, no ugly usb storage/network here, so they are useful to support as a primary device and can easily have feature enablement in the F-21 cycle with a “yum upgrade” to a newer kernel.

We’ll delve deeper into the specifics of each device and the final list closer to beta.

3.15 Fedora ARM kernel status

There’s been quite a bit of water under the bridge since my post on the 3.14 kernel status. With 3.15.x due to land in Fedora 20 shortly I thought I’d give an overview of changes for 3.15 and what’s happened since the last post.

From a shiny new devices and features point of view the 3.15 kernel is relatively boring on the ARM devices front, the advantage of that was that from a development point of view things tended to just work on Fedora. Running a diff between out 3.14 and 3.15 ARM kernel configs and checking our shipped Device Tree Blobs I get the following main changes:

  • Enabling of Marvell Dove platform. This primarily will be useful for people with the Original Cubox
  • SunXi MMC suuport. The enables initial basic support (Serial/MMC/network) for a number of AllWinner platforms
  • Zynq 7xxx platform improvements
  • OMAP DRM driver conversion to Device Tree (more on that below)
  • Initial Utilite support. It’s pretty basic with support for serial, MMC/SATA, and one of the two NICS. I plan on improving this soon
  • Added Device Tree support for a number of OMAP Overo devices
  • Added Device Tree support for a number of i.MX6 based devices

So while it was boring form a new device support point of view a kernel cycle for ARM is never really that boring! There was a lot of nice improvements generally under the hood and the march toward Device Tree is basically complete. I’m not aware of any device now that is supported not through DT in Fedora.

I mentioned above OMAP DRM. In the 3.14 post I mentioned I was sure we’d get Panda working soon. And we did! The main issue remaining was actually display support with 3.14 and with 3.15 that problem is now mostly closed because all the connectors and their associated drivers now support DT which meeans all the modules now load in the right order and things mostly just work. There’s some further improvments here in Xorg userspace in rawhide so I’d sugggest trying a nightly or the not far off Fedora 21 Alpha.

I’ve also had a few cycles to test Marvell mvebu support on my Mirabox and fixed a few kernel issues here so it now works. Unfortunately Marvell’s support of uboot, and hence Device Tree, is from the last decade and hence fairly horrible! I’ll save details of that for another post.

3.14 Fedora ARM kernel status

With 3.14.1 out Josh and Justin are preparing to land the 3.14 kernel into Fedora 20. So what does it mean in terms of ARM on Fedora. Well it’s an evolution. There’s of course the usual raft of new devices and some new SoCs, and best of all lots of improvements in support for existing devices. Even the return of some old favourites! Generally from the stash of devices Paul, I and others have that get regularly tested things are looking pretty good with 3.14.

New SoCs and device support:

  • Tegra 4/K1 support has been enabled. For 3.14 this doesn’t mean much as there’s not a wide level of devices out there that we ship device tree blobs for but it’s a good preparation for 3.15 as we should have pretty reasonable support for the nice new Tegra K1 dev board!
  • TI devices: The 3.14 finally brings working support for the OMAP5 EVM board, this will improve further in 3.15 but it boots and generally works. It also adds support for the original BeagleBone and the USB is now finally all back for the Beagle-XM devices so they go back into the fully working list! Also the DT bits for the Overo devices has started to land so if is interested in those I’d love feedback from people with those devices.
  • Freescale i.MX: To this lovely growing list we add initial support for the various Cubox-i devices and the hummingboard. Still no HDMI support but here’s hoping for 3.15
  • Xilinx Zynq 7000: The SD controller has finally landed for this which means they should be bootable to login but from there I’m not sure the status of the rest of the support. We ship DT for the zynq-zc702, zynq-zc706 and zynq-zed devices so if you’ve got one of those and have time to test feedback would be welcome.

Interesting bugs fixed:
Nothing particularly exciting comes to mind here. The fix for the Beagle-XM usb hub power up is nice, as was the final config option for the BeagleBone White. There’s obviously a deluge of other ARM driver fixes and improvements (PTP high precision time support for modular CPSW ethernet on the BeagleBone’s anyone?).

Outstanding bugs and issues:
So this is really the list of items I have outstanding for 3.14 so that I can spin some new images. Feedback on any of these and anything I might not be aware of would be very useful.

  • OMAP DRM display. In 3.11 a new display framework landed for the OMAP DRM driver and in 3.12 the old one was dropped. This broke X on devices like the Pandaboard for 3.12+ kernels. I know roughly the problematic area but I just need to get the cycles to debug this. Any help is welcome.
  • Serial over USB. While this isn’t a kernel bug but rather just needs me to hack together some scripts it’s a blocker for easy OOTB support for devices like the BeagleBone Black
  • Tegra DRM display. After a re-write it’s back and modular again in 3.14. Just need to ensure it’s working and ready to go
  • Testing… testing… testing 🙂

That’s mostly all I have on my list. If there’s anything I’m not aware of please do let me know and I’ll endevour to help out where possible. In particular I’m very interested in boot issues for devices that would would be supportable with new ARM images based on 3.14. From the 3.11 kernel that F-20 GA shipped with there’s been a lot of change and improvements, and while non boot enhancements are easy to do with a “yum update” issues with boot aren’t quite so easy to deal with!

Semi irregular Fedora ARM kernel status reports

I thought I’d start doing semi irregular ARM kernel status reports. I’ll do them as often as I think they might be useful and those who know me know I travel a lot and randomly and that ARM isn’t my $dayjob.

Few are bored or stupid enough to follow the 20 or so ARM kernel trees or have the regular insight as to what’s happening, what’s landed, what new devices might work and what bugs come and go that I do so I thought I’d try and dispense some of the more interesting bits of that information and how it relates to Fedora ARM to a wider audience by both the fedora-arm mailing list and my blog so those people that don’t sit on the IRC channel and those that like to lurk might have a better idea what’s going on.

The general format I plan to use is basically:

  • What’s new including SoCs, boards and new devices
  • Interesting bugs fixed
  • Outstanding bugs and issues
  • Random other insights

I don’t intend them to be long but rather short, sweet and to the point. They’ll probably come out when new major releases hit either rawhide or stable or something of particular interest lands. Feedback on both the format most other things is welcome as are questions and status of devices people might have had success or less so with.

I plan to have the first for 3.14 out later today and one for 3.15 RSN.

Booting PandaBoard with Fedora 20 GA

So I’ve always thought the fix for Fedora 20 booting on the PandaBoard devices would be a simple fix. I’ve always assumed it would be a minor kernel config change. I was wrong, it was even more simple once we worked it out with some help of our friends!

So to make it work you can do the following. First get a Fedora ARM Image, I used the Minimal Image, you’ll need the VFAT variant for which ever one you choose.

As root (or use sudo) run the following commands but remember to change the device path:

xzcat Fedora-Minimal-VFAT-armhfp-20-1-sda.raw.xz > /dev/mmcblk0; sync

Eject the card and put back it back in so it gets the new partition table etc and then copy the uboot over to the uboot partition. In my case I did (but change the path to your mount points):

cd /run/media/USERNAME/__/usr/share/uboot-panda/
cp MLO u-boot.img /run/media/USERNAME/uboot/
cp uEnv.txt.panda_es /run/media/USERNAME/uboot/uEnv.txt
umount /run/media/USERNAME/__/ /run/media/USERNAME/uboot/
sync

At this point you’ll likely want to use gparted or similar tools to expand the root filesystem. Once complete run screen or your favourite serial console application:

screen /dev/ttyUSB0 115200

Put the SD card into the panda, apply power, and boot through to u-boot but interrupt the boot process and get the Panda# u-boot prompt and enter the following set of commands:

setenv bootm_size 0x20000000
setenv bootargs console=${console} vram=${vram} root=LABEL=_/ ro rootwait
ext4load mmc 0:3 0x82000000 /boot/vmlinuz-3.11.10-301.fc20.armv7hl
ext4load mmc 0:3 0x88080000 /boot/uInitrd-3.11.10-301.fc20.armv7hl
ext4load mmc 0:3 0x88000000 /boot/dtb-3.11.10-301.fc20.armv7hl/omap4-panda-es.dtb
bootz 0x82000000 0x88080000 0x88000000

From there it should boot all the way through to the first boot configuration tool on the serial console, once you set a root password, create new user and set the timezone it will boot you thought to the login prompt.

That will get you to the point of being able to play. Networking works fine as does USB and a bunch of other core functionality, X doesn’t appear to with the first look and if you update the kernel (I’ve tested the shipping 3.11.10-301.fc20 and latest 3.113.13.6-200) you’ll just need to follow the process above with updated kernel details to reboot.

We’re working to get updated arm-boot-config to ensure the expected automated boot process plus some updates for X etc but I wanted to get this process out for those chomping at the bit to play while we review the changes! As always please report any issues (or success) to the Fedora ARM mailing list or IRC channel.

In terms of working devices I’ve personally tested my Panda-ES prototype (which never successfully booted anything later than 3.6.x) and Panda-ES B2, I’ve had reports of a number of variants of the vanilla Panda and ES working. I’m not sure about the B3 revisions with the Elpida DDR2 RAM but you might want to try the above with the Fedora 2014.01 uboot as it has the uboot components needed.

Lastly I’d like to explicitly call out Paul Whalen for his persistence on this one and Tom Rini who ultimately provided the tiny bit of glue we needed to solve this so simply!

I’m glad to be able to add this mainstay of the Fedora ARM project back into the supported list with fan fair, I just wish we’d found this prior to F-20 GA!

Update: It appears that X with at least the XFCE image works with the GA image.