Short history of ARMv7/armhfp/arm32 in Fedora

Back in mid November I proposed a change for Fedora 37 to retire ARMv7 as an architecture, FESCo accepted the proposal. Per the Fedora 36 schedule we branched Fedora 36 this week. Last night I enacted the last of the process to disable it in rawhide so to quote “It’s dead Jim”. The last release of Fedora to support ARMv7 AKA armhfp AKA arm32 will be Fedora 36 which will go end of life around June 2023.

I thought I’d cover a few of the things we achieved with Fedora ARM and some of the impact it’s had on the wider Linux on ARM ecosystem which people may not have realised.

First a little bit of ARM history in the Fedora ecosystem. The beginnings of ARM support actually precedes Fedora all the way back to 1998 with a fork of Red Hat Linux 4.2 and more officially with Red Hat Linux 5.1 on the Corel Netwinder (I always wanted one of those but they weren’t available in Aus).

In Fedora itself the earliest details I remember was that Marvell bootstrapped ARMv5 in Fedora 7 and continued to build and support it through to Fedora 12. This “software architecture” was known as softfp. It was optimised for the ARMv5 architecture which didn’t have a hard requirement on a floating point unit so emulated it when it was needed hence “software floating point”. In Fedora 13 Seneca College took over the ARMv5 infrastructure and building from Marvell. I officially got involved in the Fedora 14 build process and soon after was also contracted by OLPC to drive Fedora on OLPC for their ARM based XO laptops as well as work on their i686 devices to have a single OS for all of them.

In mid 2011, the Fedora 15 timeframe, a small Red Hat team started to do a ARMv7 hard floating point, AKA hardfp or armhfp, bootstrap as ARM’s new v7 mandated a floating point unit. The bootstrap included the core toolchain (binutils/gcc/glibc/elfutils and friends) and ultimately the entire distribution, I drove this effort from a community, build and packaging perspective. This required 100s of patches to upstream projects that made many assumptions about ARM only being softfp, but it also allowed us at the time to fix many general architecture assumptions in these projects. The hard floating point bootstrapping was useful for the wider community too, it was used by Nokia as the base of it’s hardfp efforts for Maemo, plus other distros used it as as it’s much easier/quicker if you already have a full distro running the architecture you wish to boostrap. What wasn’t generally known at the time was also the first new architecture that has been bootstrapped in the Fedora/RHEL ecosystem since x86_64 a long time before and it allowed Red Hat to refresh it’s memory on how to do this in preparation of the then unannounced aarch64 architecture and the POWER Little Endian intentions, basically it provided a cover story. We also worked to get other languages such as Fortran, golang, rust and others building and working on armhfp and those other architectures. The final piece of this was ARMv7 being promoted to a primary Fedora architecture in Fedora 20. This then later went on to my proposal to redefine secondary architectures in Fedora.

In the wider community of Linux Fedora ARM was the first distribution to adopt the kernel “multi platform” work enabling us to go from building 5 different kernels to support a handful of arm devices to a single kernel supporting 100s of devices in a very short period of time. I worked with closely Arnd Bergmann from Linaro on issues with the early pieces of the multiplatform work. In upstream U-Boot we posted the first distro_boot patches to support booting Linux in the same way across all the devices we actively supported so we didn’t need specially wrapped kernels and know exact offsets for every SoC or device. The distro_boot support evolved, working with SUSE, into UEFI support in U-Boot further standardising the ARM boot process by abstracting the pieces that were different and letting the firmware deal with them. This work ultimately evolved into EBBR and the ARM System Ready IR spec. In Fedora 34 we moved to soley supporting UEFI on both ARM architectures. A lot of Linux distros still have specific kernels for each device and use non standard boot methods for devices and hence have an image for each device/use-case they wish to use. This was something Fedora identified very early on as something that would not scale!

Fedora also leads a lot of things in the gcc toolchain stack across all our supported architectures, we’ve actively enabled a lot of security features and other things like LTO early on. As the Fedora gcc maintainers, employed by Red Hat, are also key upstream GCC maintainers we’re almost always the first distribution to rebase onto a new release before it’s a stable release, for example Fedora 36 had just had a mass rebuild against a gcc-12 pre release snapshot. This builds all of the 50k or more source packages with the pre-release of the new toolchain making for a much better release for the wider GCC community because this picks up a number of bugs/regressions in both the general support but also in the architectures Fedora supports which means the ARMv7 hardfp support in GCC has benefited from 100s of bugs we’ve detected in gcc/binutils/glibc etc before they land in a stable gcc release. With the retirement of ARMv7 in Fedora this is going to be something the wider ARMv7 community is going to have to pick up post the GCC-12 release.

Over the subsequent 11 years of ARMv7 support in Fedora, and much longer if you include the early ARMv5 the distribution has also enabled a number of other innovative features like support for containers, support for devices like the Raspberry Pi 2 and 3 in Fedora 25. as mentioned various toolchains, and fun things like robots. Of course we also lose some things too. Devices like the BeagleBone don’t yet have a 64 bit sibling, but there’s less and less 32 bit devices coming out and the use of armhfp is waning quickly and the maintenance cost is rising as the industry moves more generally to 64 bit even in embedded use cases and the fact is with devices like the $15 Raspberry Pi Zero 2W it makes less and less sense even if I do still actively run BeagleBones, a Panda-ES and 3 different i.MX6 devices.

So I engaged with the wider Arm ecosystem and it made sense to finally sunset our ARMv7 32 bit support. We’re of course leaving it in good shape with things like gcc-12, the latest rust and golang toolchains and 5.17 kernels, much newer by the time F-36 goes EOL in June 2023, it will be in good shape if people wish to use it as the basis of some form of continuing ARMv7 supported Linux distribution.

Sail off into the sunset friend, it’s been a fun 12 years of hacking on those projects!

Three ways to speed up dnf on arm devices

I have a large bunch of Arm Single Board Computers I use for testing a lot. Most of the testing ends up being pretty basic stuff like firmware, kernels, and the various bits of hardware peripherals that people use like storage, network, display and sound output, plus things like sensors and HAT support.

The problem is that these devices often aren’t the fastest in the world for various reasons so I want to be able to apply updates to the basic system as quickly as possible to find out the results. Over time I’ve worked out that these three things speed up dnf quite a bit for the sort of testing I wish to do are as follows:

  1. Disable modularity:
    sed -i 's/enabled=1/enabled=0/' /etc/yum.repos.d/fe*mod*
  2. Don’t install weak dependencies:
    echo "install_weak_deps=False" >> /etc/dnf/dnf.conf
  3. Disable dnf makecache. It never seems to be up to date when you need it anyway:
    systemctl disable dnf-makecache; systemctl mask dnf-makecache

You may need to re-do some of these each major update as they seem to want to force you to have them every time.

Fedora on the UDOO Neo

Some time ago I backed the UDOO Neo Kickstarter as it looked like a nifty, well featured, IoT device. I got the full option which came with 1Gb RAM and both wired and wireless Ethernet and some add-on sensors. It was a well run kickstarter campaign and the device was well packaged with a fab box. It has both a Cortex-A9 processor to run Fedora and a Cortex-M4 embedded processor to enable you to do Arduino style functionality which should be interesting to experiment with.

For various reasons it has sat around gathering dust, it’s been a bit of a long drawn out process with me randomly poking it as time allowed.. Primarily this was because there was no decent upstream U-Boot and kernel support, and I’d not had the time to hack that up myself from various downstream git repositories, but even without Fedora support their forked Ubuntu distro in the form of UDOObuntu has an experience that is truly terrible!

Late 2016 the problem of a lack of upstream support for U-Boot and kernel changed with initial basic support landing upstream for all three (Basic, Extended and Full) models so with a few cycles over a weekend it was time to dust it off to see if I could get Fedora 26 (did I mention this has been long running?) running on it and to see what worked.

The first thing for me to do was to setup a serial console for easy debugging. The UDOO Neo documentation is generally outstanding and the pins for the UART1 TTL are documented. Two things to note here is that the headers are female rather than the usual SBC male pins so I had to bodge my usual usb to serial TTL with some male-male jumper wires and you’ll need a ground for the TTL which is undocumented on their page, I used one of the GNDs as documented on connector J7 and all was good.

So after an initial set of fixes to the U-Boot support it saw my Fedora install and started to boot! Success! Well sort of, as mentioned above the initial support is rudimentary, it started to boot the kernel and very quickly managed to corrupt and destroy the filesystem not making it much beyond switch root. That wasn’t good. In the last week or two I’ve had a little time to look again, similar issues, it was better than it was a year or so ago but it still ended up with corruption. I reached out to one of the maintainers from NXP that deals with a bunch of the i.MX platforms and I got directed to a handful of patches, a test kernel and image later and a test boot… all the way to initial-setup! SUCCESS!

The core support for the i.MX6SX SoC and the UDOO Neo is pretty reasonable, with the MMC fixes it’s been very stable, all the core bits are working as expected, included wired and wireless network, thermal, cpufreq, crypto and it looks like the display should work fine. There’s a few quirks that I need to investigate further which should provide for a fun evening or weekend hacking. There has also been recently merged support for the i.MX6SX Cortex-M4 land upstream in Zephyr upstream for the 1.13 release, so getting that running and communication using Open-AMP between Fedora and Zephyr should also be an interesting addition. I think this will be a welcome addition to Fedora 29, and not a moment too soon!!

Why I’m not backing the Purism Librem 5 phone

NOTE: This is a post about my opinion on the device, hence the title of “Why I’m not backing….”, people have been explicitly asking me why I’ve not backed it, this documents it so I don’t need to keep repeating myself!

Numerous people have come up to me and asked “So will you get Fedora to run on your Librem phone?” and when my response is “No, I’ve not backed it” I get weird looks with a question of “Why?” I had thought it was time to do document my concerns with this laudable venture. This was certainly further qualified when I had to inform someone from the EFF that they’re delirious about the “it doesn’t need closed source firmware” on the i.MX chips that are being proposed. While I applaud the general principals and ideas of a fully open rights protecting phone I can’t help but feel that the group doing it either are being false at worst or naive at best with some of their statements.

Firmware

Their site claims “The i.MX 6/8 CPU will be completely free software without any binaries whatsoever!” while this could be sort of true if you want a hobbled device it’s not really the case at all. There’s a number of firmwares that are needed to make a number of pieces of functionality of a mobile device useful. Firstly the SDMA driver needs a firmware to run at any level of reasonable speed. Secondly the accelerated media decode, which will be required if you actually want to consume meda on your phone and have more than moments of battery life, also needs firmware. You’ll note on the media decode I don’t reference the actual firmware! Why? Because it’s not actually distributed in the linux-firmware repository so you have to request it from NXP or somewhere with appropriate signups (while I’m not 100% sure I believe having a driver in the linux kernel without required firmware in linux-firmware is a breach of the requirements of said driver in the upstream kernel).

i.MX6 or i.MX8 SoC

The i.MX 6 or i.MX 8 option concerns me. Basically make a choice! The core issue I have is the i.MX6 SoC is ancient, being announced back in Jan 2011, and based on a Cortex-A9 SoC. It doesn’t support USB-C so to ship as promised they’ll need to add a PCI-e attached USB-3 controller, charging circuitry, and maybe even a LVDS to Display Port option plus a chip to MUX the three through the actual USB-C port if they want to be able to ‘dock’ which will make power consumption on the phone even worse! If I’d spent $600 and received a phone in January 2019 based on an eight year old 32-bit chip design I’d be seriously pissed off. To be quite honest the i.MX8 is no spring chicken either, being announced in September 2013, it’s based on the already quite long in the teeth Cortex-A53 SoC design which was the original aarch64 “little” low end design but it does at least support USB-C on the SoC. The i.MX SoCs are generally quite nice, and the i.MX6 is quite well supported upstream, but the whole line of SoCs are more targeted towards embedded applications like cars, so they do have a long support cycle. They’re not a mobile phone focused SoC though, they also tend to be quite slow to get moving in the market/availability and the i.MX8 has little upstream support in the kernel as yet. Sure the etnaviv driver enables an open 3D accelerated driver, one that isn’t supported by the vendor so doesn’t enable all the features upstream, but you can run this today on Fedora 26/27 on the i.MX6 SoCs now but a free GPU driver is not the only reason to choose a platform.

General concerns

I can’t help but feel that there’s going to be a lot of disappointed people that will end up receiving an expensive sub standard device, probably late, that ends up not taking us much further along the road, to a fully open rights protecting phone, than the days of the Nokia n9xx series phones running Maemo. The HW, if based on an i.MX6, will certainly not be much further along that route and still stuck in the past in terms of HW. I personally believe the project would be better off engaging with the Qualcomm community team to use the Qualcomm 820c/600c SoCs because there’s an open driver that Qualcomm are working to improve, and while there’s a need for firmware for GPU and media offload, in reality it’s no better/worse than the i.MX devices with the bonus that their devices that are more current and aimed at mobile workloads with the vendor actively working to upsteam the enhance the support support of their SoCs.

Ultimately I think a device that closely resembles the specs of a phone of recent history, than that of ancient smart phone history, is likely to get a better following and hence a better software ecosystem than one that’s the same era as the Motorola DROID 4.

Update: So just to clarify a few things that were bought up in a couple of threads:

First this reddit thread:

  • I don’t question the value of hardware isolation of the various wireless interfaces that they claim, or the ability of them being able to deliver that bit on what ever SoC they choose, that is why it wasn’t addressed above.
  • I bought up USB-C because it is explicitly claimed as a feature.
  • I bought up display port because given USB-C above and their “It can be a desktop computer and phone all-in-one” claim, including a nice picture, then DP over USB-C is the only real option for that functionality.
  • Qualcomm chips were an example of other SoCs I believe might be a better fit, yes they do provide options in their APQ line without built in radios, they were just an example of another option, there are other possibilities, it wasn’t meant to provide a guarantee, it was an alternative example.
  • Yes, all ARM processors have onboard boot firmware, it’s generically referred to as “PBL” (Primary Boot Loader), so do most other processors.

Second this purism thread:

  • Sure you can disable the media engine with an e-FUSE but media is kind of useful for a lot of use cases like video config, audio calls and music, not just watching videos. If they choose this route, I would hope they leave the fuses unblown and document how a end user can do it else the $600 is a whole lot less useful for a lot of people

Ultimately the main points is I have to make about all of the above is two fold, firstly it’s my opinions and why I didn’t back the device, and secondly there appears to be some large discrepancies in the statements they make about the device and SoCs which I would not have expected and that is the key problem for me because it causes concerns over the ability to deliver a working and useful device to me. $600 is not a small amount of money, for me at least, to hedge on what *might* be a i.MX8 or might be an old 32 bit i.MX6.

My ARM grab bag device list

They say the first step of coming to terms with addiction is admitting you have a problem… I have a problem with collecting ARM devices… there I said it! How big is this problem you ask? How about I list them out and let you decide!

I’ll break the list down into categories as I believe it’s big enough to do so :-/

The aarch64 set of devices currently stands at:

  • 2x Applied Mustang (different x-gene SoC revs)
  • AMD Seattle
  • 96boards HiKey (hi6220)

The ARMv7 boards list is currently:

  • Compulabs Trimslice (tegra-2)
  • Toshiba AC100 (tegra-2)
  • nVidia Jetson TK1 (tegra-124)
  • Acer Chromebook (tegra-124)
  • BeagleBoard xM (omap3)
  • Nokia n900 (omap3)
  • Nokia n950 prototype (omap3)
  • BeagleBone (am33xx)
  • BeagleBone Black (am33xx) x3
  • BeagleBone Green (am33xx)
  • PandaBoard ES Prototype (omap4)
  • PandaBoard ES B2 (omap4)
  • CubieBoard (allwinner-a10)
  • CubieTruck (allwinner-a20)
  • Banana Pi (allwinner-a20)
  • C.H.I.P. Alpha x2 (allwinner-r8)
  • Snowball (u8500)
  • Compulabs Utilite (imx6q)
  • WandBoard Quad revb (imx6q)
  • novena board (imx6q)
  • RIoTboard (imx6s)
  • UDOO Neo (imx6sx)
  • Origen (exynos-4)
  • OLPC XO 1.75 – a number of variants (mmp2) xNumerous
  • OLPC XO-4 including XO-Touch (mmp3) xNumerous
  • Linksys WRT1900AC (armada-xp)
  • Mirabox (armada-370)
  • ifc6410 (qcom)
  • Parallella Board (zynq7000)
  • Raspberry Pi 2 x3

The Cortex-M series for IoT sensors is currently:

  • TI SensorTag 2015
  • ARM mBed IoT starter kit
  • BeeWi SmartClim

Other random related bits:

  • BeagleBone Breadboard Prototyping Cape x2
  • BeagleBone CryptoCape
  • Original 256Mb Raspberry Pi model B
  • Grove starter kit for BeagleBone Green
  • Explorer HAT
  • PiGlo HAT
  • TI CC2531 802.15.4 USB dongle x3
  • numerous random sensors

So the list above is the devices that I use for hacking on. I count 41 without listing out the dozen or so ARM based XOs I have (various prototypes and models). I also don’t have in that list phones, tablets and two drones as I don’t really hack on those as it’s not like with the list above I don’t already have enough toys! So do I have a problem?

Fedora 21 and ARM device support

As we slowly meander our way towards the pointy end of the Fedora 21 release, with Alpha speeding up in the rear view mirror, the Fedora ARM team are starting to discuss the best way to deal with the blossoming amount of ARMv7 devices that can and do run out of the box on Fedora.

With our 3.16 kernel containing device tree blobs for 200+ devices, the Fedora 3.17 rawhide kernel already containing 230+, it’s truly impossible to actively test and support all of those devices. So much like previous releases we’ll be focusing on testing a group of “primary devices” with the remainder being considered as secondary. This doesn’t mean they won’t work, it just means they’re not necessarily a testing focus of the regular contributors or they might not be readily available to purchase.

So what makes a device primary? Well there’s a number of considerations we’ve put into the list. Firstly the device has to be widely available and well supported upstream. Some will notice that some of the devices are no longer widely available (yes Panda, Trimslice and Calxeda I’m looking at you!) and I did consider their removal from the list but given a lot of contributors have them I think it’s worthwhile keeping them around for the moment. The primary devices list won’t be release blocking, we don’t block x86 releases for specific single devices, so I don’t believe we should for ARMv7 either.

Astute readers will notice the proposed primary list of around two dozen devices is much larger than the core devices we supported in Fedora 20! YAY! is all I have to say about that 🙂

The list is not final, at the moment it’s a suggested list and one open for discussion to some degree and what we’ll be heading from Alpha to Beta with. I fully expect it to be tweaked as we go along, there might be cool new shiny Chromebooks 😉 that arrive on the scene and end up working nicely and are hence worth actively supporting (no EXYNOS Chromebooks I’m not looking at you!) and some devices on the list below that end up not making the grade. One thing is for sure the grade includes that they support Cute Embedded Nonsense Hacks ie DeviceTree… there’s no board support here.

Primary:

  • Wandboard (all models/revisions)
  • Utilite (all models)
  • Cubox-i (all models)
  • Hummingboard (all models)
  • RIoTBoard
  • BeagleBone Black
  • Tegra K1 Jetson
  • CubieBoard (all models)
  • Banana Pi
  • Trimslice
  • PandaBoard (all models)
  • Calxeda Highbank/Midway
  • VExpress (qemu)

Secondary:

  • BeagleBone White
  • Beagle xM
  • Novena
  • UDOO
  • AC100
  • Qualcomm (IFC6410, DragonBoard)
  • Various Marvell devices (Mirabox, AX3, CuBox)
  • Various Exynos devices
  • Other AllWinner devices (as per available u-boot/DT support)
  • Gumstix Overo series of devices
  • OMAP5 EVM
  • STE SnowBall

ARM64:

  • AMD Seattle
  • APM Mustang
  • VExpress (qemu)

So what can a user expect from the primary devices above? Will all the functionality of a device work? Well it depends on the specific device and the associated SoC. For example the AllWinner SoC GPU support is far from upstream so unfortunately there will be no graphical UX for those devices, the Tegra K1 support for the GPU isn’t quite there yet but we’re hoping by GA it will be. Some will be better than others in terms of certain features but for example the AllWinner devices would make good storage devices with their SoC attached SATA and Network, no ugly usb storage/network here, so they are useful to support as a primary device and can easily have feature enablement in the F-21 cycle with a “yum upgrade” to a newer kernel.

We’ll delve deeper into the specifics of each device and the final list closer to beta.

3.15 Fedora ARM kernel status

There’s been quite a bit of water under the bridge since my post on the 3.14 kernel status. With 3.15.x due to land in Fedora 20 shortly I thought I’d give an overview of changes for 3.15 and what’s happened since the last post.

From a shiny new devices and features point of view the 3.15 kernel is relatively boring on the ARM devices front, the advantage of that was that from a development point of view things tended to just work on Fedora. Running a diff between out 3.14 and 3.15 ARM kernel configs and checking our shipped Device Tree Blobs I get the following main changes:

  • Enabling of Marvell Dove platform. This primarily will be useful for people with the Original Cubox
  • SunXi MMC suuport. The enables initial basic support (Serial/MMC/network) for a number of AllWinner platforms
  • Zynq 7xxx platform improvements
  • OMAP DRM driver conversion to Device Tree (more on that below)
  • Initial Utilite support. It’s pretty basic with support for serial, MMC/SATA, and one of the two NICS. I plan on improving this soon
  • Added Device Tree support for a number of OMAP Overo devices
  • Added Device Tree support for a number of i.MX6 based devices

So while it was boring form a new device support point of view a kernel cycle for ARM is never really that boring! There was a lot of nice improvements generally under the hood and the march toward Device Tree is basically complete. I’m not aware of any device now that is supported not through DT in Fedora.

I mentioned above OMAP DRM. In the 3.14 post I mentioned I was sure we’d get Panda working soon. And we did! The main issue remaining was actually display support with 3.14 and with 3.15 that problem is now mostly closed because all the connectors and their associated drivers now support DT which meeans all the modules now load in the right order and things mostly just work. There’s some further improvments here in Xorg userspace in rawhide so I’d sugggest trying a nightly or the not far off Fedora 21 Alpha.

I’ve also had a few cycles to test Marvell mvebu support on my Mirabox and fixed a few kernel issues here so it now works. Unfortunately Marvell’s support of uboot, and hence Device Tree, is from the last decade and hence fairly horrible! I’ll save details of that for another post.

3.14 Fedora ARM kernel status

With 3.14.1 out Josh and Justin are preparing to land the 3.14 kernel into Fedora 20. So what does it mean in terms of ARM on Fedora. Well it’s an evolution. There’s of course the usual raft of new devices and some new SoCs, and best of all lots of improvements in support for existing devices. Even the return of some old favourites! Generally from the stash of devices Paul, I and others have that get regularly tested things are looking pretty good with 3.14.

New SoCs and device support:

  • Tegra 4/K1 support has been enabled. For 3.14 this doesn’t mean much as there’s not a wide level of devices out there that we ship device tree blobs for but it’s a good preparation for 3.15 as we should have pretty reasonable support for the nice new Tegra K1 dev board!
  • TI devices: The 3.14 finally brings working support for the OMAP5 EVM board, this will improve further in 3.15 but it boots and generally works. It also adds support for the original BeagleBone and the USB is now finally all back for the Beagle-XM devices so they go back into the fully working list! Also the DT bits for the Overo devices has started to land so if is interested in those I’d love feedback from people with those devices.
  • Freescale i.MX: To this lovely growing list we add initial support for the various Cubox-i devices and the hummingboard. Still no HDMI support but here’s hoping for 3.15
  • Xilinx Zynq 7000: The SD controller has finally landed for this which means they should be bootable to login but from there I’m not sure the status of the rest of the support. We ship DT for the zynq-zc702, zynq-zc706 and zynq-zed devices so if you’ve got one of those and have time to test feedback would be welcome.

Interesting bugs fixed:
Nothing particularly exciting comes to mind here. The fix for the Beagle-XM usb hub power up is nice, as was the final config option for the BeagleBone White. There’s obviously a deluge of other ARM driver fixes and improvements (PTP high precision time support for modular CPSW ethernet on the BeagleBone’s anyone?).

Outstanding bugs and issues:
So this is really the list of items I have outstanding for 3.14 so that I can spin some new images. Feedback on any of these and anything I might not be aware of would be very useful.

  • OMAP DRM display. In 3.11 a new display framework landed for the OMAP DRM driver and in 3.12 the old one was dropped. This broke X on devices like the Pandaboard for 3.12+ kernels. I know roughly the problematic area but I just need to get the cycles to debug this. Any help is welcome.
  • Serial over USB. While this isn’t a kernel bug but rather just needs me to hack together some scripts it’s a blocker for easy OOTB support for devices like the BeagleBone Black
  • Tegra DRM display. After a re-write it’s back and modular again in 3.14. Just need to ensure it’s working and ready to go
  • Testing… testing… testing 🙂

That’s mostly all I have on my list. If there’s anything I’m not aware of please do let me know and I’ll endevour to help out where possible. In particular I’m very interested in boot issues for devices that would would be supportable with new ARM images based on 3.14. From the 3.11 kernel that F-20 GA shipped with there’s been a lot of change and improvements, and while non boot enhancements are easy to do with a “yum update” issues with boot aren’t quite so easy to deal with!