Tag: rawhide

Short history of ARMv7/armhfp/arm32 in Fedora

Back in mid November I proposed a change for Fedora 37 to retire ARMv7 as an architecture, FESCo accepted the proposal. Per the Fedora 36 schedule we branched Fedora 36 this week. Last night I enacted the last of the process to disable it in rawhide so to quote “It’s dead Jim”. The last release of Fedora to support ARMv7 AKA armhfp AKA arm32 will be Fedora 36 which will go end of life around June 2023.

I thought I’d cover a few of the things we achieved with Fedora ARM and some of the impact it’s had on the wider Linux on ARM ecosystem which people may not have realised.

First a little bit of ARM history in the Fedora ecosystem. The beginnings of ARM support actually precedes Fedora all the way back to 1998 with a fork of Red Hat Linux 4.2 and more officially with Red Hat Linux 5.1 on the Corel Netwinder (I always wanted one of those but they weren’t available in Aus).

In Fedora itself the earliest details I remember was that Marvell bootstrapped ARMv5 in Fedora 7 and continued to build and support it through to Fedora 12. This “software architecture” was known as softfp. It was optimised for the ARMv5 architecture which didn’t have a hard requirement on a floating point unit so emulated it when it was needed hence “software floating point”. In Fedora 13 Seneca College took over the ARMv5 infrastructure and building from Marvell. I officially got involved in the Fedora 14 build process and soon after was also contracted by OLPC to drive Fedora on OLPC for their ARM based XO laptops as well as work on their i686 devices to have a single OS for all of them.

In mid 2011, the Fedora 15 timeframe, a small Red Hat team started to do a ARMv7 hard floating point, AKA hardfp or armhfp, bootstrap as ARM’s new v7 mandated a floating point unit. The bootstrap included the core toolchain (binutils/gcc/glibc/elfutils and friends) and ultimately the entire distribution, I drove this effort from a community, build and packaging perspective. This required 100s of patches to upstream projects that made many assumptions about ARM only being softfp, but it also allowed us at the time to fix many general architecture assumptions in these projects. The hard floating point bootstrapping was useful for the wider community too, it was used by Nokia as the base of it’s hardfp efforts for Maemo, plus other distros used it as as it’s much easier/quicker if you already have a full distro running the architecture you wish to boostrap. What wasn’t generally known at the time was also the first new architecture that has been bootstrapped in the Fedora/RHEL ecosystem since x86_64 a long time before and it allowed Red Hat to refresh it’s memory on how to do this in preparation of the then unannounced aarch64 architecture and the POWER Little Endian intentions, basically it provided a cover story. We also worked to get other languages such as Fortran, golang, rust and others building and working on armhfp and those other architectures. The final piece of this was ARMv7 being promoted to a primary Fedora architecture in Fedora 20. This then later went on to my proposal to redefine secondary architectures in Fedora.

In the wider community of Linux Fedora ARM was the first distribution to adopt the kernel “multi platform” work enabling us to go from building 5 different kernels to support a handful of arm devices to a single kernel supporting 100s of devices in a very short period of time. I worked with closely Arnd Bergmann from Linaro on issues with the early pieces of the multiplatform work. In upstream U-Boot we posted the first distro_boot patches to support booting Linux in the same way across all the devices we actively supported so we didn’t need specially wrapped kernels and know exact offsets for every SoC or device. The distro_boot support evolved, working with SUSE, into UEFI support in U-Boot further standardising the ARM boot process by abstracting the pieces that were different and letting the firmware deal with them. This work ultimately evolved into EBBR and the ARM System Ready IR spec. In Fedora 34 we moved to soley supporting UEFI on both ARM architectures. A lot of Linux distros still have specific kernels for each device and use non standard boot methods for devices and hence have an image for each device/use-case they wish to use. This was something Fedora identified very early on as something that would not scale!

Fedora also leads a lot of things in the gcc toolchain stack across all our supported architectures, we’ve actively enabled a lot of security features and other things like LTO early on. As the Fedora gcc maintainers, employed by Red Hat, are also key upstream GCC maintainers we’re almost always the first distribution to rebase onto a new release before it’s a stable release, for example Fedora 36 had just had a mass rebuild against a gcc-12 pre release snapshot. This builds all of the 50k or more source packages with the pre-release of the new toolchain making for a much better release for the wider GCC community because this picks up a number of bugs/regressions in both the general support but also in the architectures Fedora supports which means the ARMv7 hardfp support in GCC has benefited from 100s of bugs we’ve detected in gcc/binutils/glibc etc before they land in a stable gcc release. With the retirement of ARMv7 in Fedora this is going to be something the wider ARMv7 community is going to have to pick up post the GCC-12 release.

Over the subsequent 11 years of ARMv7 support in Fedora, and much longer if you include the early ARMv5 the distribution has also enabled a number of other innovative features like support for containers, support for devices like the Raspberry Pi 2 and 3 in Fedora 25. as mentioned various toolchains, and fun things like robots. Of course we also lose some things too. Devices like the BeagleBone don’t yet have a 64 bit sibling, but there’s less and less 32 bit devices coming out and the use of armhfp is waning quickly and the maintenance cost is rising as the industry moves more generally to 64 bit even in embedded use cases and the fact is with devices like the $15 Raspberry Pi Zero 2W it makes less and less sense even if I do still actively run BeagleBones, a Panda-ES and 3 different i.MX6 devices.

So I engaged with the wider Arm ecosystem and it made sense to finally sunset our ARMv7 32 bit support. We’re of course leaving it in good shape with things like gcc-12, the latest rust and golang toolchains and 5.17 kernels, much newer by the time F-36 goes EOL in June 2023, it will be in good shape if people wish to use it as the basis of some form of continuing ARMv7 supported Linux distribution.

Sail off into the sunset friend, it’s been a fun 12 years of hacking on those projects!

Three ways to speed up dnf on arm devices

I have a large bunch of Arm Single Board Computers I use for testing a lot. Most of the testing ends up being pretty basic stuff like firmware, kernels, and the various bits of hardware peripherals that people use like storage, network, display and sound output, plus things like sensors and HAT support.

The problem is that these devices often aren’t the fastest in the world for various reasons so I want to be able to apply updates to the basic system as quickly as possible to find out the results. Over time I’ve worked out that these three things speed up dnf quite a bit for the sort of testing I wish to do are as follows:

  1. Disable modularity:
    sed -i 's/enabled=1/enabled=0/' /etc/yum.repos.d/fe*mod*
  2. Don’t install weak dependencies:
    echo "install_weak_deps=False" >> /etc/dnf/dnf.conf
  3. Disable dnf makecache. It never seems to be up to date when you need it anyway:
    systemctl disable dnf-makecache; systemctl mask dnf-makecache

You may need to re-do some of these each major update as they seem to want to force you to have them every time.

Fedora on the UP Squared

With the IoT Working Group and Edition moving forward I’ve been looking for an x86_64 device suitable for testing IoT related use cases. I was originally planning on using the MinnowBoard or Joule but given Intel has killed those product lines off it was back to the drawing board. I eventually settled on the UP², in particular I chose the UP² Pentium-4GB-32GB-PACK as it had everything I wanted in one box.

Hardware

The on paper hardware specs show a recent generation Intel Apollo Lake core, reasonable memory and storage options, an onboard FPGA, USB-3, dual ethernet and various other bits. The kit comes with options for active or passive cooling and the later the heatsink is massive, for the moment I’m running it on the passive cooling. The case is OK, I wouldn’t rave about it though. The power connector on the other hand is terrible, the PSU cable doesn’t seat well into the board and I’ve bumped it already and had it lose power, the power button is also tiny, so small in fact I mistook it for a reset button.

Fedora support

As you would expect the support in Fedora 27 is decent, accelerated 4K graphics, wired RealTek ethernet NICs, Intel m.2 PCI-e WiFI/Bluetooth, although the later is only 4.2 for IoT I would have appreciated Bluetooth 5, all work out of the box as expected. The firmware is uEFI and in theory supports secure-boot but I couldn’t work out how to turn it on in the firmware menus as it was greyed out, it also has a TPM2 module I’ve not had time to investigate. They eagle eyed would also note that I mention Fedora 27 even though Fedora 28 has been out a few days. Well for some reason F-28 doesn’t boot, I tried the network installer and the Workstation live image, they both get to the grub menu, then I get no output and nothing for a moment then it resets and we start again. I need to investigate this further but Fedora 27 Workstation livecd booted and installed fine so that’s what it’s got for the moment. Ultimately this is going to a host to test Fedora IoT so while I tested the general support this is fine.

IoT support

There’s a number of reasons I chose this particular device as an IoT test device:

  • Reasonably priced with a reasonable feature set.
  • Intel hasn’t killed it off yet like they’ve done with the Joule platform and Minnowboard 3 so I could actually buy it 🙂
  • Multiple network interfaces, reasonable WiFi and Bluetooth support.
  • Industrial IO sensors via an onboard Intel Sensor Hub. lsiio is reporting 9 sensors of various types. I’ve not checked this further yet.
  • USB-3, a Raspberry Pi HAT compatible connector and other options to add IoT related functionality or interfaces.

FPGA support

I’ve not looked at the FPGA support at all. The upstream kernel now has a FPGA Manager Framework and there’s a bunch of Altera FPGA support there but I’m not sure how it maps to this device. I also have to investigate open source toolchains for FPGA bitstreams as a lot of them just aren’t, I’ll likely do the HW enablement side of things and leave the toolchain bits to people that understand them. I also have the 96boards Ultra96 board so FPGA investigation was already on my Fedora 29 To Do list, and a lot of other people seem quite interested in them of late, no idea why 😉

Flock Rochester

I’m not going to do a day by day outline of what I did at flock, if I did it would basically be “blah blah blah I talked a lot to a lot of people about a lot of tech topics” and anyone that’s ever met me would have guessed that! It was, as in the past, a great conference. A big shout out to the organisers for an excellent event with two excellent evening events! So I’m going to give a brief summary to my talks and link to slides and video recordings.

My first talk was an overview of the state of aarch64 and POWER as secondary architectures. The slides aren’t particularly interesting as they’re just words for discussion points. The video has all the interesting bits. A related talk was Dennis’s Standardising ARMv7 booting with a memorial quote by Jon Masters 😉

My second talk was about using Fedora as a base for IoT. Slides are here but the talk was quite a bit different to the slides and is more interesting so I suggest watching the video.

I also actively participated in Dennis’s Fedora Release Engineering going forward because well obviously I’m part of it 😉 and it was interesting for where we’re going, and even where we’ve come from in the last year or so 🙂

Finally I loved the Keynote Be an inspiration, not an impostor by Major Hayden. He’s published a follow up blog post with a FAQ too.

The least memorable bit was the terrible Amtrak ride back to New York City. On the plus side it makes the worst of the British National Rail service seem amazingly on time! NEVER AGAIN!

Fedora 23 on the Thinkpad X1 Carbon gen 3

So my corporate laptop for the last three years has been a Thinkpad x220, it’s getting a bit long in the teeth, when I got it the x230 was already out but the corporate standard hadn’t rebased. This time I happened to get a new device just after the new corporate refresh so I have the shiny X1 Carbon gen 3 with real track pad buttons.

Of course I grabbed the latest rawhide nightly boot.iso to install the latest shiny! How did I get on? Initially it was disaster, the kernel crashed before I even got to anaconda (will spend some time to recreate and log that soon). Oops! So I grabbed a couple of the earlier installers and with a bit of trial and error before long I was at the initial anaconda screen. From there the process was relatively boring! Up and running with Fedora Workstation installed, disk encrypted I started to play with it to see which bits were good, bad or ugly.

So first up with the good. Over all the vast majority of the hardware just worked out of the box. The GPU, USB3, wired ethernet, wireless ethernet (iwl 7265 AC), trackpad/thumb pointer, bluetooth mouse, camera, onboard speakers all work just fine, I even managed to enrol my finger in the finger print reader without issue. I must say I’m loving the 1920*1080 screen over the old 1366*768 of the x220 and I’m getting use to the chiclet keyboard layout. Overall F-23/rawhide is solid right out of the gate with pretty much everything working as you’d expect on a stable release. 🙂

It’s not all rosy though and some of the bad is that I’ve not managed to get the firmware upgraded to support 4K displays @ 60hz, thanks to Major and Sandro for the heads up on this, but it never seems to find the usb stick as a bootable image. Not a major issue in the short term as the OneLink Pro Dock I ordered is AWOL but I’ll want it soon as I’ve got a standing desk with a Dell 24 inch 4K monitor. The x220 and 4K monitor had never really worked overly well, the big issue was it only ran at 30hz refresh but I knew that when I got the monitor because I knew I was due for a refresh soon, it also had random glitches. I also need to workout how to adjust the acceleration of the thumb pointer thingy with libinput. Adjusting either the trackpad or mouse options in control panel doesn’t seem to have any effect, I never really used the trackpad on the x220 but I admit I’m getting use to the two finger scrolling for reading long pages. This should all be relatively easy to solve with a bit of poking!

The ugly seems to be the stability of the iwl 7265 wireless driver/firmware. It generally works but regularly shits itself. Some times minor by dropping SSH connections, sometimes majorly resulting in a need to unload/reload the modules or even to reboot! URGH! I’ve heard people complain about recent Intel Wireless stability but the “Advanced-N 6205 [Taylor Peak] (rev 34)” that was in the x220 was always solid. The solid lockups seems to be when pushing a reasonable amount of data via rsync/ssh. I do have a WRT1900AC router and I’ve connected to the 5 ghz 11ac so I’m wondering whether this combo is part of the issues. There is a newer firmware that hasn’t made it upstream yet. I need to do some more playing here testing the 11n 2.4ghz network as well as testing the newer firmware and possibly some patches that are on the wireless mailing list which I’m hoping will actually just land in 4.2 before long 🙂

There’s a few other things I need to play with some more. I’ve not tried external HDMI video/audio, the external display port, the DP to VGA converter (nice one was included in the box though) or the headphone/mic socket. None of the functionality of the dock has been tested yet simply because it’s yet to show up! It reports around 7.5 hours of battery life but I want to look at what the state of power consumption on these devices after mjg59’s post about it to see if I can’t get that well into the double digit hours.

Overall I’m pretty impressed with out of box experience of F-23/rawhide on the Thinkpad X1 Carbon gen 3 🙂

aarch64 (arm64) on Fedora 21 update

I realise that there’s not been an update as to the state of ARM aarch64 on Fedora for sometime. We should really shout out our achievements a lot more as over the last few months or so there’s been fairly extreme progress made but we all just tend to apply the nose to the grindstone and get one with it!

So where are we? Well we’ve got the vast majoriity of Fedora built. As we’re only building for rawhide/F-21 there’s a number of missing F-21 builds but with the mass rebuild well and truly underway on mainline we’re all hyped up and building on aarch64 too (appart from a power outage in the Huntsville facility that hosts some of the builders!) and we’re hot on the tail of mainline builds.

So we have HW virtualisation support and a bunch of virt bits on top of that, the vast majority of other services and low level support like storage platforms, all the various desktops and most of the various development stacks (even Ada!). It’s actually looking pretty good. In recent time we’ve added R, ghc and a raft of different language support. The java team have done a really good job getting java-1.8.0-openjdk up to spec too!

So appart for the mainline builds that haven’t yet been built for F-21 what are we missing? Well not much really now. Go and nodejs, the later due to missing v8 js engine support (it’s upstream in a newer v8 release but Fedora is stil trailing), and the things that depend on them are the main ones now missing so no Docker as yet either. There’s no mono, and while the core erlang is built I need to work out the correct order to finishing the bootstrap.

The rest of the job now becomes somewhat boring in that once the mass rebuild is complete we just need to go through packages and fix individual packages that are FTBFS and I strongly suspect most of those will have issues on mainline too.

On the hardware and virtualisation front the current rawhide kernel should boot just fine on the ARM foundation model emulator and the next release of qemu will have full system emulation of aarch64 to enable the running of aarch64 on x86_64 similar to what we can do with ARMv7 now. The 3.16 kernel should allow us to run on the currently and soon to be available HW without too many problems either as the last core bits of support should land upstream in the current merge window.

In short aarch64 on Fedora is looking awesome! By the time Fedora 21 Alpha lands in two months time we’ll basically be feature complete, and even now it’s extremely usable for those lucky enough to have hardware or patient enough to use software emulators.