Short history of ARMv7/armhfp/arm32 in Fedora

Back in mid November I proposed a change for Fedora 37 to retire ARMv7 as an architecture, FESCo accepted the proposal. Per the Fedora 36 schedule we branched Fedora 36 this week. Last night I enacted the last of the process to disable it in rawhide so to quote “It’s dead Jim”. The last release of Fedora to support ARMv7 AKA armhfp AKA arm32 will be Fedora 36 which will go end of life around June 2023.

I thought I’d cover a few of the things we achieved with Fedora ARM and some of the impact it’s had on the wider Linux on ARM ecosystem which people may not have realised.

First a little bit of ARM history in the Fedora ecosystem. The beginnings of ARM support actually precedes Fedora all the way back to 1998 with a fork of Red Hat Linux 4.2 and more officially with Red Hat Linux 5.1 on the Corel Netwinder (I always wanted one of those but they weren’t available in Aus).

In Fedora itself the earliest details I remember was that Marvell bootstrapped ARMv5 in Fedora 7 and continued to build and support it through to Fedora 12. This “software architecture” was known as softfp. It was optimised for the ARMv5 architecture which didn’t have a hard requirement on a floating point unit so emulated it when it was needed hence “software floating point”. In Fedora 13 Seneca College took over the ARMv5 infrastructure and building from Marvell. I officially got involved in the Fedora 14 build process and soon after was also contracted by OLPC to drive Fedora on OLPC for their ARM based XO laptops as well as work on their i686 devices to have a single OS for all of them.

In mid 2011, the Fedora 15 timeframe, a small Red Hat team started to do a ARMv7 hard floating point, AKA hardfp or armhfp, bootstrap as ARM’s new v7 mandated a floating point unit. The bootstrap included the core toolchain (binutils/gcc/glibc/elfutils and friends) and ultimately the entire distribution, I drove this effort from a community, build and packaging perspective. This required 100s of patches to upstream projects that made many assumptions about ARM only being softfp, but it also allowed us at the time to fix many general architecture assumptions in these projects. The hard floating point bootstrapping was useful for the wider community too, it was used by Nokia as the base of it’s hardfp efforts for Maemo, plus other distros used it as as it’s much easier/quicker if you already have a full distro running the architecture you wish to boostrap. What wasn’t generally known at the time was also the first new architecture that has been bootstrapped in the Fedora/RHEL ecosystem since x86_64 a long time before and it allowed Red Hat to refresh it’s memory on how to do this in preparation of the then unannounced aarch64 architecture and the POWER Little Endian intentions, basically it provided a cover story. We also worked to get other languages such as Fortran, golang, rust and others building and working on armhfp and those other architectures. The final piece of this was ARMv7 being promoted to a primary Fedora architecture in Fedora 20. This then later went on to my proposal to redefine secondary architectures in Fedora.

In the wider community of Linux Fedora ARM was the first distribution to adopt the kernel “multi platform” work enabling us to go from building 5 different kernels to support a handful of arm devices to a single kernel supporting 100s of devices in a very short period of time. I worked with closely Arnd Bergmann from Linaro on issues with the early pieces of the multiplatform work. In upstream U-Boot we posted the first distro_boot patches to support booting Linux in the same way across all the devices we actively supported so we didn’t need specially wrapped kernels and know exact offsets for every SoC or device. The distro_boot support evolved, working with SUSE, into UEFI support in U-Boot further standardising the ARM boot process by abstracting the pieces that were different and letting the firmware deal with them. This work ultimately evolved into EBBR and the ARM System Ready IR spec. In Fedora 34 we moved to soley supporting UEFI on both ARM architectures. A lot of Linux distros still have specific kernels for each device and use non standard boot methods for devices and hence have an image for each device/use-case they wish to use. This was something Fedora identified very early on as something that would not scale!

Fedora also leads a lot of things in the gcc toolchain stack across all our supported architectures, we’ve actively enabled a lot of security features and other things like LTO early on. As the Fedora gcc maintainers, employed by Red Hat, are also key upstream GCC maintainers we’re almost always the first distribution to rebase onto a new release before it’s a stable release, for example Fedora 36 had just had a mass rebuild against a gcc-12 pre release snapshot. This builds all of the 50k or more source packages with the pre-release of the new toolchain making for a much better release for the wider GCC community because this picks up a number of bugs/regressions in both the general support but also in the architectures Fedora supports which means the ARMv7 hardfp support in GCC has benefited from 100s of bugs we’ve detected in gcc/binutils/glibc etc before they land in a stable gcc release. With the retirement of ARMv7 in Fedora this is going to be something the wider ARMv7 community is going to have to pick up post the GCC-12 release.

Over the subsequent 11 years of ARMv7 support in Fedora, and much longer if you include the early ARMv5 the distribution has also enabled a number of other innovative features like support for containers, support for devices like the Raspberry Pi 2 and 3 in Fedora 25. as mentioned various toolchains, and fun things like robots. Of course we also lose some things too. Devices like the BeagleBone don’t yet have a 64 bit sibling, but there’s less and less 32 bit devices coming out and the use of armhfp is waning quickly and the maintenance cost is rising as the industry moves more generally to 64 bit even in embedded use cases and the fact is with devices like the $15 Raspberry Pi Zero 2W it makes less and less sense even if I do still actively run BeagleBones, a Panda-ES and 3 different i.MX6 devices.

So I engaged with the wider Arm ecosystem and it made sense to finally sunset our ARMv7 32 bit support. We’re of course leaving it in good shape with things like gcc-12, the latest rust and golang toolchains and 5.17 kernels, much newer by the time F-36 goes EOL in June 2023, it will be in good shape if people wish to use it as the basis of some form of continuing ARMv7 supported Linux distribution.

Sail off into the sunset friend, it’s been a fun 12 years of hacking on those projects!

Three ways to speed up dnf on arm devices

I have a large bunch of Arm Single Board Computers I use for testing a lot. Most of the testing ends up being pretty basic stuff like firmware, kernels, and the various bits of hardware peripherals that people use like storage, network, display and sound output, plus things like sensors and HAT support.

The problem is that these devices often aren’t the fastest in the world for various reasons so I want to be able to apply updates to the basic system as quickly as possible to find out the results. Over time I’ve worked out that these three things speed up dnf quite a bit for the sort of testing I wish to do are as follows:

  1. Disable modularity:
    sed -i 's/enabled=1/enabled=0/' /etc/yum.repos.d/fe*mod*
  2. Don’t install weak dependencies:
    echo "install_weak_deps=False" >> /etc/dnf/dnf.conf
  3. Disable dnf makecache. It never seems to be up to date when you need it anyway:
    systemctl disable dnf-makecache; systemctl mask dnf-makecache

You may need to re-do some of these each major update as they seem to want to force you to have them every time.

Raspberry Pi improvements in Fedora 29

So Fedora 29 is probably going to account for the largest single improvement to support on the Raspberry Pi support in Fedora since we added initial support in Fedora 25. It certainly wasn’t without issue, but after quite a bit of debug we’ve got the post release issues with the WiFi back to being stable!

WiFi improvements
The support for upstream NVRAM files and the ability to add those files to linux-firmware means we get WiFi support for the Raspberry Pi 3 Series of devices out of the box! No need to grab anything, it just works! Well mostly, we had some issues with WiFi being very intermittent, as well as a missed bug around aarch64 but now with the 4.19.10 kernel everything appears to be working and stable. This makes me very happy and it took longer than I had hoped but we’re there. This device specific NVRAM driver support will also help another bunch of cheap Arm and x86 based devices that ship with Broadcom/Cyprus based WiFi support moving forward.

ZRAM enabled by default
By supporting and enabling ZRAM swap by default we get a more responsive device and less wear on the MicroSD storage. Over all we’ve seen reasonable performance improvements and to no date no major issues.

GNOME performance improvements
In May 2018 the Raspberry Pi Foundation kindly hosted a GNOME Performance Hackfest in the lovely Cambridge. Over a couple of days we managed to fix a number of issues seen, review and document a number of issues and work on a number of ways of reducing the memory usage of GNOME. Of course this improvement is primarily seen constrained devices like the Raspberry Pi but ultimately less memory utilisation by GNOME even helps devices with decent amounts of RAM and CPUs too. The fixes didn’t arrive in time for Fedora 28, but a bunch have landed in Fedora 29 providing noticeable improvements, and the GNOME team is by no means done and there will be more coming in Fedora 30 and beyond! It was an excellent start and I expect there will be ongoing enhancements here into the future especially with devices like the Purism Phone which will have similar constraints.

Initial CPU frequency support
Another of the largest issues around the Raspberry Pi is complaints it was slow, part of the issue here is that there’s no upstream CPU Frequency driver which means all models of the Raspberry Pi run at a glacial, but safe, 600Mhz out of the box compared to the highest speed, which on the 3B+ is 1400Mhz. With Fedora 29 we’ve landed an experimental cpufreq driver which allows us to run the Raspberry Pi 3-Series at much closer to optimal speeds. While this is experimental it might not stay around if we find out it causes issues or ends up being a maintenance burden but to date it hasn’t yet appeared to have caused any issues.

HWmon Voltage Sensor
There’s a new driver that reports when the voltage supplied by the PSU drops below the required voltage. It can be a bit noisy in dmesg but one of the biggest support problems we have with the Raspberry Pis is people using a power supply that’s not powerful enough, this issue is more of a problem with Fedora 29 because with the support for running at faster frequencies due to the cpufreq driver it means we also draw more power and some PSUs that were previously fine now cause issues because they can’t supply enough current.

Enhanced support for config.txt
A lot of the hardware addons are supported in Raspbian are done by enabling things in the config.txt file, this in turn does things like loading DT overlays and merging them with the base DT to enable extra hardware like HAT support. We have enhanced the way Fedora works with this which enables us to be much closer to the way Raspbian handles these things. The advantage this has is that the documentation that’s written for Raspbian is then usable by Fedora in the wider Raspberry Pi ecosystem which in turn makes it easier for end users to get HW up and running due to less differences in process. There’s further enhancements to make here but every step closer is easier for everyone to enable and use their favourite HATs.

Improved bcm283x firmware support
In preparation for grub2 support we enhanced how we deal with the firmware that the Raspberry Pi uses for booting. This deals with the early startup. We never use to upgrade it by default to ensure things didn’t break, but it also meant most users also didn’t by default get the fixes and enhancements. Now we do. The config.txt is also handled directly which means if you never edit the file you now automatically get any changes we make, because rpm handles the file as a config file, if we change it you get a .rpmnew file so you won’t lose your changes.

Camera support
This wasn’t available in the Fedora 29 4.18 kernels, but with the rebase to the 4.19 kernel the support for the camera on the Raspberry Pi CSI Camera interface improved enough we could enable this in Fedora. The early 4.19 kernels don’t automatically detect and load support if the camera module is attached. There’s some patches in 4.20 in rawhide for this, and we’ll bring some of this to 4.19 soon, and we’re working with upstream to further improve the camera support. You’ll also want the latest bcm283x firmware which tweaks some of the config.txt and updates to a firmware with ISP fixes.

Another improvements
There was also a number of general Arm improvements which sped up crypto on the Raspberry Pi, improved the USB, fixed up some issues with the wired ethernet on the 3B+, power and a number of other fixes. As always there’s more coming. The 4.20 kernel rebase should also bring with it analog sound support early in the new year.

Conclusion
Overall I was pleased with the work that landed in Fedora over 2018 for the Raspberry Pi. The WiFi regression was disappointing, but now with that fixed in 4.19.10 we have WiFi support out of the box without users needing to download anything which moving forward will make things a lot more straight forward. The initial support for the camera makes it much more useful in numerous use cases and we’ll really polish up the HAT support in Fedora 30 which for me is the last remaining big ticket item for Raspberry Pi support. There’s still some annoying bits around the EDID detection in the display, but there’s work to improve that upstream, and also there’s work to land the media decode offloading upstream too which will also one of the few remaining bits.

The Raspberry Pi 3 B+ in Fedora

So I’m sure none of you are surprised to hear that I’ve been asked a bunch about support for the Raspberry Pi 3 B+ in Fedora. Well the good news is that it’ll be supported in Fedora 28. Most of the bits were there for the official Fedora 28 beta, it just needed a minor work around, but nightly images since Beta have had all the bits integrated so the upcoming Fedora 28 GA release will support the Raspberry Pi 3 B+ to the same levels as the original 3 B on both ARMv7 and aarch64. The Fedora Raspberry Pi FAQ has now been updated with all the details of both the RPi3+ and Fedora 28.

WiFi

As with the original 3 there’s files with the firmware we can’t redistribute. The details are documented in the Fedora Raspberry Pi FAQ.

You can grab the files like for the 3 although there’s now an extra one, which you don’t really need, but it gives you all the 802.11a frequencies:

$ sudo curl https://fedora.roving-it.com/brcmfmac43455-sdio.txt -o /lib/firmware/brcm/brcmfmac43455-sdio.txt
$ sudo curl https://fedora.roving-it.com/brcmfmac43455-sdio.clm_blob -o /lib/firmware/brcm/brcmfmac43455-sdio.clm_blob

I’ve also done a rpm of the files for both editions of the RPi3 (plus a few other brcm adapters used on ARM boards). You can either just grab the rpm or setup the repo if you want to get the latest if I update it:

$ sudo curl https://fedora.roving-it.com/wireless.repo -o /etc/yum.repos.d//wireless.repo
$ sudo dnf install brcm-firmware

Issues

Like all new things there’s often a few teething problems. Over all the support for the new RPi 3 B+ is relatively solid but like all new devices there’s some bits still bedding down, and the combination of a new device and new OS there’s been a few minor issues that have been seen in some circumstances. We pulled in the latest firmware just before GA to fix some issues but no doubt as it gets wider testing both in Fedora and in the wider Raspberry Pi community more issues may well come up.

The ones we’re aware of are:

  • The new USB hub and gigbit ethernet interface have seen a few issues in some cases. We’ve pulled in quite a few patches to help stabilise the NIC in Linux and it now mostly works in the vast majority of use cases.
  • The USB hub in u-boot, uEFI and grub on aarch64 can cause some issues. If there’s too many USB devices connected it sometimes won’t boot or will do so really slowly. The work around for the moment is to disconnect all the USB devices until Fedora has started to boot and then plug them in.

In Fedora we’ll deliver updates and fixes by the usual updates, in particular as fixes land upstream we’ll review and land them where useful into Fedora, more than likely via a kernel update. If you see a uboot-tools or a bcm283x-firmware update you’ll want to run the rpi-firmware-update command to update the firmware and then reboot for it to take effect.

Older releases

We won’t be supported the new device in the older releases. Why I hear you say? Well it’s possible but it needs update to the firmware, U-Boot and kernel to work. The Raspberry Pi foundation respun Raspbian to support it and basically it’s not straight forward. Much better to have a new shiny Fedora for a new shiny device!

Overview of aarch64 SBC support in Fedora 27

Support for ARM 64 bit (aarch64) Single Board Computers (SBCs) has been one of the most highly requested features along side the Raspberry Pi. It’s something I’ve been working towards almost as long too. Finally with Fedora 27 I felt we would have enough of the bits in place for a Minimum Viable Product (MVP).

You’ll note from the Fedora change linked above I was very cautious with what we planned to achieve. The change has a very focused list of images: Server, Workstation and Minimal and a limited list of devices: basically the Raspberry Pi 3, the 96boards Dragonboard 410c and HiKey, and a handful of AllWinner devices with a focus on the Pine64 series of boards. The reason for this was I knew there was going to be a lot of low level boot and kernel bits that needed focus and polish and the Fedora 27 cycle was severely limited time and resource wise so the plan was to focus on getting all the core bits into place for Fedora 27 and have a couple of well polished devices and then expand that rapidly for Fedora 28.

The key functionality we were aiming for was a well polished uEFI implementation in u-boot to enable a single install/boot path in Fedora on aarch64 using uEFI/shim/grub2 to boot Fedora on both SBCs and SBSA compliant aarch64 platforms. We now have that platform in place, primarily due to Herculean efforts of Rob Clark and Peter Jones, as well as many others who have provided insight into the deep dark details of the uEFI specification. Fedora 27 will ship with a quite heavily patched, well by Fedora’s standards anyway, u-boot 2017.09 which provides us the core of this functionality enabling us to use a vanilla upstream shim and grub2 to boot a standard Fedora. All this work is already upstream, or making it’s way there in 2017.11. In Fedora 28 there will be even more improvements that will enable us to do a bunch of other cool stuff (that’ll be a post for later!) and also enable much quicker upstream board enablement now all the core bits are in place.

So what do we actually support? Well all the usual bits that you would expect on a standard Fedora install, whether it be x86_64, ARMv7 or aarch64, like SELinux, containers, desktops and all the other bits. There’s a few bits and pieces that are a little rough around the edges but overall the feature is pretty robust. On a board by board feature set lets break the this down across the boards:

Raspberry Pi 3

The support for the Raspberry Pi3 is the equivalent to the ARMv7 support but with boot via uEFI/grub2. The memory isn’t quite as good as on 32-bit but that’s to be expected, overall it’s pretty reasonable for a device of the specs and cost. Like on 32 bit support we’re seeing regular improvements each release and throughout the releases. The aarch64 support for the RPi3 is just an evolution to this.

DragonBoard 410c

The support for the DragonBoard 410c is looking pretty decent. Qualcomm has been doing a pretty decent effort to get stuff upstream, we have firmwares for the GPU and for video decode/encode upstream as well, along with kernel drivers and the open freedreno 3D drivers, HDMI audio should work as well. The WiFi firmware isn’t yet upstream but I’ll document how/where to get that and hopefully that should be in linux-firmware soon as well. Overall I’m quite happy with the status of this device, although like all devices with 1Gb RAM it’s a little constrained, but that should make the newly announced 820c with 3Gb of RAM a decent device ;-). All the details for getting it running will soon be in the Fedora 96boards wiki page.

HiKey

Most features and functionality of the HiKey are supported, note this isn’t the HiKey960 (look to F-28 for support for that), except accelerated graphics due to the use of a MALI GPU. Other than that the functionality is pretty decent. You’ll likely want the latest tianocore firmware and the details for that can be found on the Fedora 96boards wiki page.

Pine64 (AllWinner A64 SoC)

We actually should have a number of devices based on the AllWinner A64 SoC working here but we’ve only tested the 3, 2Gb/1Gb/512Mb, Pine64 device sizes. The support for these devices is headless and you will need a serial console else you’re on your own as none of the display bits in the kernel have made it upstream, and of course the GPU is a MALI 400 series so when it does it won’t be fast. The support for the rest of the device is basic, it’s usable for a headless server style device, we support network, USB, KVM, RTC and a few other bits. Other than display we don’t yet support the SDIO attached wireless, sound, crypto offload or any of the other media interfaces. A lot of this is under review upstream so I think Fedora 28 should look much better for this series of devices and 4.15 might even bring very basic console output. Speaking of series of devices which ones should actually work other than the three Pine64 devices? Well the following A64 SoC devices have a Fedora built u-boot and kernel DT support so should work as well as the Pine64: BananaPi-m64, OrangePi Win, SoPine baseboard (PineBook boots if you’re happy with serial console), NanoPi-A64 and the A64-OLinuXino. We had some troubles with the AllWinner H5 SoC devices earlier in the cycle but I’ve had a couple of reports that it seems to be resolved so they should work too and that adds the Orange Pi PC2, Prime and Zero+ 2 as well as the NanoPi NEO2. So that’s around a dozen or so devices! 🙂

Other ARM64 SBCs

I’ve had reports that other aarch64 SBCs boot on Fedora just fine. I’ve not listed those where I can’t verify whether they boot with our uEFI enabled u-boot. Looking around on my desk I do have a number of devices that I expect us to be supporting in Fedora 28, or maybe even just enabling u-boot bits in a F-27 update.

Overall I’m pretty happy with the state of Aarch64 SBCs for Fedora 27 and what we’ve managed to achieve is such a short cycle!

Raspberry Pi improvements in Fedora 26

So since I landed support for the Raspberry Pi 2 and 3 just in time for Fedora 25 Beta it’s been a bit of a fun ride. The support for Raspberry Pi is mostly done in my spare time along side all the other responsibilities I have and it’s been interesting to see people’s feedback. Going into Fedora 25 I knew it wasn’t going to be perfect but the experience was going to be reasonable for newbies to get going without generally needing serial consoles and it met Fedora’s (and mine) exacting standards on free drivers. I think we achieved that quite well but I also learned a lot in the Fedora 25 cycle and what’s coming in Fedora 26 is quite a substantial jump forward.

Hardware for a good experience

So what have I learned about the first six months or so of Raspberry Pi in Fedora? Well there’s a couple of things that the user can do to ensure a decent starting experience themselves. The biggest FAQs I’ve dealt with on the various support forums are generally fixed by these three things:

  • A proper spec power supply. For the RPi2 this means at least 2 AMPs and for the RPi3 at least 2.5 AMPs. If you want to plug in USB WiFi dongle and a USB HDD you’ll likely want to add a little more! In most cases an old phone charger will not suffice.
  • A good quality Class 10 micro SD card. I generally use Samsung EVO or SanDisk Ultra cards.
  • A Raspberry Pi 2 or 3. Yes, it’s surprising how many people hope to run it on something else. SORRY (actually, I’m not!)!

What’s in Fedora 26 Final

So enough of what to do! Everyone wants to know what improvements arrived in the Fedora 26 Final with the 4.11.x kernels:

  • Pi3 WiFi: It’s been working in F-26 since Alpha and is surprisingly stable. There’s a file you need to grab to enable it. See details in the wiki here.
  • Performance: In the process of dealing with wifi I worked out one of the reasons we were seeing poor performance on the SD card. We’ve had some minor improvements in F-25 but this fix over doubles the performance for me on the SD card.
  • HDMI video: There’s been issues around certain monitors crashing the video (vc4) driver and people getting black screens during boot. While this isn’t perfect yet (ain’t hardware great!!) it’s greatly improved across numerous devices.
  • Composite video: We’ve had support for the composite video since 4.10 but I need people to help test this.
  • Sound: HDMI audio is supported, I’ve done minor testing with the one HDMI audio capable device I have. Analogue audio out isn’t upstream yet.
  • HAT support: We now have all the support needed to do overlays in the kernel/bootloader and dtc stack. I just need to test it some more, document it and work out how we can best distribute pre built overlays to ease consumption. There’s still no consensus on an Overlay Manager from upstream to auto load overlays based on EEPROM on the HATs. In a lot of cases you want to load the overlays from u-boot anyway for things like display. Look out for docs and blog posts on this soon!

What arrived with the 4.12 kernel rebase

  • Thermal support: so if the RPi runs too hot it’ll slow it down
  • More performance improvements and tweaks.

What’s coming in the 4.13 kernel rebase

  • Bluetooth support: upstream finally tracked down the issues here. It’s been a much requested features and I should have the bits in place soon!
  • More performance, stability and graphics improvements and tweaks.

What about Fedora 25?

Some of the above pieces will be coming to Fedora 25 with the 4.12 rebase. The focus of my spare time is Fedora 27 mostly now, with the above coming to F-26. Some components are a lot harder to back port without issues or a complex series of package updates to ensure smooth upgrade. The WiFi and performance improvements were the hardest as part of that change moves around the use of hardware blocks and drivers. I managed to stop both the RPi2 and RPi3 booting numerous times in testing before I properly realised the implications of the change. Getting these changes for users back into a stable release without issues is hard and time consuming to do across all the various use cases. I tried this with some fixes in 4.9 and ended up making the RPi3 very unstable. This cost me a lot of time to debug and fix and I don’t really want a repeat of that!!

Graphics device

One of the surprising side effects was the discovery of a device that is five years old is that Fedora suffered from early adopters issues. We were one of the first distributions to adopt a fully upstream open kernel and graphics stack and with that came a number of issues around monitor detection, especially older/cheaper models that aren’t 1920/1080 “Full HD” or via HDMI to VGA adapters. We’re still working through these with upstream and have improved the situation quite a bit in Fedora 26 overall but it takes time and reproducible use cases which with random hardware isn’t easy or quick! 🙁

Next up?

I’ll leave Fedora 27 features and functionality for another, this post has been sitting in my drafts folder since June so it’s time to get it out and like my development move on to Fedora 27!

WiFi on Raspberry Pi 3 for Fedora 26 Alpha

So I managed to land just about everything needed for the WiFi on the Raspberry Pi 3 for Fedora 26 Alpha (around 4.11 rc3). There’s one thing missing, because we can’t currently redistribute it, but it’s straight forward for the end user to do themselves once they’ve done the initial setup:

sudo curl https://raw.githubusercontent.com/RPi-Distro/firmware-nonfree/master/brcm80211/brcm/brcmfmac43430-sdio.txt -o /lib/firmware/brcm/brcmfmac43430-sdio.txt

Or you can also do it when you’re flashing the image if you mount the root filesystem but the above is likely easier. It’s been surprisingly stable in my testing.

Before you all ask, at the moment I don’t plan on pushing this to earlier Fedora releases, as the upgrade path is not trivial. I will also soon publish more details of some of the other new features coming for the Raspberry Pi to Fedora 26 but I thought you’d all like the WiFi details now. The wiki has also been updated to reflect the status of the WiFi.

PS: No this is not an April Fool’s joke (it’s well past midday in UK).

Updating Raspberry Pi firmware on Fedora

The upstream Raspberry Pi firmware/bootloader gets regular updates and improvements. In Fedora we ship that firmware in a package called bcm283x-firmware. I regularly follow the git repo of the upstream firmware and on occasion when I believe there’s reasonable changes that benefit Fedora I’ll prepare a new version, do some brief testing on my devices to make sure it boots and basic functionality hasn’t regressed at which point I’ll update the package and send it out to supported releases as an update.

Once the new bcm283x-firmware lands on your Raspberry Pi it doesn’t automatically update the firmware though. Why is that you ask? I don’t like to spring surprises on people where they end up with a device that might not boot or it might regress things they care about.

So how do you upgrade the firmware for the Raspberry Pi on Fedora? It’s simple! You simply run the command rpi-firmware-update and it’ll update the firmware and the u-boot to the latest one that’s shipped as a Fedora package. Then you just need to reboot to make it active.

The easiest way to work out which firmware you’re currently running is “dmesg | grep raspberrypi-firmware”

I tend to try and push out a new firmware update every month or so but if I see something that’s of interest or that fixes known issues I do it as needed.

Lipstick on a Pig AKA the Raspberry Pi 3

So while waiting for local scratch kernel builds for much more interesting devices I started looking around to see if I could find details of the kernel sources for the new BCM2837 SoC that is centre stage in the Raspberry Pi 3.

The problem is I couldn’t. What I did find is the hack the Raspberry Pi Foundation uses to boot the RPi3 on github.

So there is no source code release for the new BCM2837 SoC, just a device tree file. Someone said to me “They’re violating the GPL” and before people get out their pitch forks… they’re NOT because this is the code they ship, they are meeting their obligations there.

So for the lay person (yes, I know there’s a lot of deep level tech details I’m glossing over deep ARM architecturey people!!) basically what they are doing is booting this device as a ARMv7 device, and because the code isn’t built for ARMv8 (32 or 64 bit) they really just get the speed bump of a ARMv7 device running a bit faster, and possibly some better memory speeds and other general improvements.

So what does this mean for other distributions that wish to actually to support the Raspberry Pi 3 as a aarch64 device? You currently can NOT do so!. Why? Basically it boils down to two things:

  • Source code release for the kernel: To be honest I don’t think this should be large. People with low level knowledge of ARMv7 and the BCM283x could probably hack this up
  • Firmware support: I suspect there will need to be a new firmware that supports booting this as a aarch64 device. I obviously don’t know for sure but I’m guessing the firmware will need changes to actually properly boot this as a aarch64 device. I’ve little doubt there’s a bunch of hackery going on in there!

Of the above two, if my theory is correct, the firmware is the problematic one because it relies on the Raspberry Pi Foundation to do the work. This work for something that they feel, at the moment, gives them no particular gain but only confusion about multiple OSes. They are of course correct for their use case, basically like old school enterprise where you buy a bigger server to scale vertically because your app won’t scale horizontally, but this is another kick in the guts of the Open Source community they so heavily rely on! Oh well, it’s about as much as I expected from the Raspberry Pi Foundation as after all their devices are only just now becoming usable with upstream kernels and open userspace GPU drivers…. after a mere four years.

So what does this mean for Fedora? Basically the only way we’ll be able to support it in the short, possibly medium, term is like it’s sibling the Raspberry Pi 2 as an ARMv7 device but with added shitty wifi. Really, this device isn’t a cheap aarch64 device, it’s just like lipstick on a pig! If a cheap aarch64 device is what you want one of those go and buy a PINE64.

On the plus side the work needed to support it as a ARMv7 device at the same time as it’s sibling should just be some minor u-boot and kernel device tree patches on top of what I previously documented . Note I’ve not looked closely at this as yet, I’m still waiting for mine to arrive (YAY day 3 of 1 day shipping)! Frankly I’d sooner support it this way, an aarch64 device with terrible USB2 IO and 1GB of RAM won’t provide much, if any, of a perf bump over ARMv7, and then have the Raspberry Pi Foundation spend their time working with Broadcom on fixing the wifi and enabling distribution of the wifi firmware in linux-firmware as proper opensource broadcom wifi support would have a wider impact on the Open Source community the Foundation relies upon!

Supporting Fedora 24 on the Raspberry Pi 2

So I get asked the question dozens of times a day so I thought I’d outline the answer to the question “When is Fedora going to support the Raspberry Pi 2?” and “The kernel support is upstream in the 4.5 kernel, why isn’t it enabled in Fedora 24?”

Ultimately support in the kernel is great, it’s obviously a core blocker, and the first steps to supporting a new piece of hardware in Fedora. The thing is that when people say kernel support is easy they are partially right but it’s only a very small part of what’s needed to support a complex device such has an ARMv7 Single Board Computer for the average user, especially one as popular as the Raspberry Pi! To make the device work with Fedora we could just enable the kernel bits but it doesn’t make for a good user experience OOTB (Out Of The Box).

With a lot of ARMv7 devices these days a new device comes out and it just works with Fedora. It’s awesome when I read a report, or someone tells me “I tried device X with Fedora and it just worked”. That’s because of a lot of work Fedora, and others, have done to ensure upstream boot loaders and boot process just works with new devices. It’s taken a long time to get us to this point. The Raspberry Pi is sadly not like all the other ARMv7 or aarch64 devices. It doesn’t have a standard boot process, doesn’t use u-boot or uEFI, needs vFAT partitions, firmware, text config files and other things that none of our other supported devices need. All of these differences need to be taken into account.

So in terms of the support being upstream in 4.5 the answer to that is it’s “mostly” upstream, there’s still a bunch of patches we’d need to pull in to ensure a nice OOTB experience. This isn’t a blocker in my opinion, it’s something that’s relatively straight foward with most of the bits already headed upstream into 4.6 so it’s a short term issue.

For the rest of the bits what do I consider a nice out of the box experience? This:

  • A single image to support the Raspberry Pi and all out devices (more on that below)
  • Graphics and USB support from boot to login
  • Most basic peripherals working, at a minimum USB (inc keyboard/mouse), HDMI, wired ethernet, a decent selection of wireless USB dongles, storage (MMC and USB) and preferably sound (analog, digital HDMI, mic)
  • A means of easily creating a bootable SD card from at least the Fedora command line, and probably Windows or MacOSX
  • Good documentation, FAQ etc

Single Image:
We currently produce a lot of different images for ARMv7 like Workstation, Server, Minimal and various desktops. If we had to double the amount of images we make to add vFAT that would double the work needed by QA, rel-eng and also lead to confusion by end users as to which image is needed. I have no intention of doubling anyone’s work, or adding confusion for end users, there needs to be a proper engineering solution to this problem!

Serial Console
A number of people have said to me “just enable it and tell them to use the serial console” but having been working on ARMv7 for over 6 years now I know from experience that this leads to vast amounts of confusion by end users as to why “it’s not working” and it leads to a lot of time “providing support” to end users. With a device such as the Raspberry Pi this will become an order of magnitude worse which won’t provide users a good Fedora experience, and likely drive the people who are trying to support the device a even more nuts than normal!

Basic peripheral support
I feel that network, storage, display, input and sound, although I’m still on the fence about sound, are the minimum viable supported peripherals needed for a good OOTB experience for users.

I know that initially we won’t support HATs very well, you can already copy around device tree overlays in the /sys filesystem for basic support, but upstream still hasn’t finalised what a good experience will look like in this regard. I think we can live with this. This won’t stop the use of I2C or SPI devices connected to the 40 pin header, most of these should work just fine.

Image to card creation
Closely related to the single image and basic peripheral support I think the ability to easily create an image to use is important. I’m not sure we’re going to be able to easily solve the Windows/OSX problem, although to be honest I’ve not looked at what’s out there and we might be able to extend LiveUSBCreator here.

Raspberry Pi 3
So when will we support this? Well the kernel support hasn’t been released yet, at least that I could find at the time of writing this. That being said I don’t think it’ll be a particularly evasive or large patch set, the hardware around the Cortex-A53 is the same, so it’ll be just some glue and a pinctrl driver to make all the bits work together, likely not too dissimilar to other recent SoCs that have gained ARMv8 support. This won’t land in 4.6 as the major changes are already queued to land in that, so likely 4.7 will be the earliest upstream kernel. The wireless on the other hand could be more interesting.

So basically the work needed for support of the Raspberry Pi 3, with a proper 64 bit OS 😉 , shouldn’t be too hard once the kernel bits are upstream.

Firmwares
The other issue we had was the legal ability to redistribute the “GPU boot loader firmware” thankfully that problem was resolved about a year ago, although it did take us over 2 years to do so!

The other firmware issue which will cause problems with Raspberry Pi 3 support is that the Broadcom wireless is notoriously terrible, as any Linux running Mac user will contest to, and their wireless firmware isn’t re-distributable in the standard process of being included in the upstream linux-firmware. With luck this is something that the Raspberry Pi Foundation could assist Broadcom in improving!

Summary
So we’re well on the way to adding support. Assistance in the issues above would be very welcome, whether for supporting a single image or adding support for writing images for the Pi, to help us get the bits done sooner rather than later. Feel free to reach out to me on IRC (pbrobinson on Libera.Chat) or some other means.