Zephyr RTOS 2.x on Fedora: Configure the build environment

It’s been a while since I’ve had time to play with the Zephyr RTOS project and the project has evolved greatly since so I thought I’d document the process I went through while playing with Zephyr 2.1 on Fedora using the Fedora native cross toolchains rather than the various ones suggested by the Zephyr Project docs.

I’m going to do a couple of posts in this series to break it up a little. This first one will be getting a generic build environment setup. I’ll go into more detail on the specific devices I’m playing with but the ones I have handy are ARM Cortex-M based so that’s what I’ll be focusing on even though Zephyr RTOS supports numerous architectures.

As before it’s worth reading the latest Zephyr Getting Started guide. This time around I’m using a AWS aarch64 a1.medium instance running a Fedora 31 cloud instance but I’ve also tested that a DigitalOcean Droplet with 2Gb RAM works with the later ZephyrRTOS releases too.

Once you have a Fedora instance up and running install the required dependencies:

# disable modularity, mostly just slows things down.
$ sudo sed -i 's/enabled=1/enabled=0/' /etc/yum.repos.d/fe*mod*

# install core utils, git and cross compilers
$ sudo dnf install git-core cmake ninja-build gperf dfu-util dtc \
xz file python3-pyelftools arm-none-eabi-gcc-cs python3-pip

# requirements for the west (Zephyr meta build tool)
$ sudo dnf install python3-colorama python3-configobj \
python3-configobj python3-docopt python3-pykwalify \
python3-dateutil python3-colorama python3-docopt \
python3-pykwalify python3-packaging

We now install west:

$ pip3 install --user --upgrade west

We now have key build dependencies installed so we can initialise west and clone key repos using west (this will take a little while to do an initial clone run):

$ west init zephyr
$ cd zephyr
$ west update

Setup the Zephyr environment for cross compiling with the distribution tools:

$ export ZEPHYR_TOOLCHAIN_VARIANT=cross-compile
$ export CROSS_COMPILE=/usr/bin/arm-none-eabi-
$ source zephyr-env.sh
$ mkdir builds

By default the above uses the git master branch of the Zephyr git repo. If you wish to use a stable branch you can just check it out. The latest stable release is, currently, 2.1 so to use this you can check out the stable branch:

$ git checkout v2.1-branch

With this we now have a Zephyr RTOS development environment setup for building for Arm Cortex-M based devices on Fedora using the distribution’s cross toolchains.

Raspberry Pi improvements in Fedora 29

So Fedora 29 is probably going to account for the largest single improvement to support on the Raspberry Pi support in Fedora since we added initial support in Fedora 25. It certainly wasn’t without issue, but after quite a bit of debug we’ve got the post release issues with the WiFi back to being stable!

WiFi improvements
The support for upstream NVRAM files and the ability to add those files to linux-firmware means we get WiFi support for the Raspberry Pi 3 Series of devices out of the box! No need to grab anything, it just works! Well mostly, we had some issues with WiFi being very intermittent, as well as a missed bug around aarch64 but now with the 4.19.10 kernel everything appears to be working and stable. This makes me very happy and it took longer than I had hoped but we’re there. This device specific NVRAM driver support will also help another bunch of cheap Arm and x86 based devices that ship with Broadcom/Cyprus based WiFi support moving forward.

ZRAM enabled by default
By supporting and enabling ZRAM swap by default we get a more responsive device and less wear on the MicroSD storage. Over all we’ve seen reasonable performance improvements and to no date no major issues.

GNOME performance improvements
In May 2018 the Raspberry Pi Foundation kindly hosted a GNOME Performance Hackfest in the lovely Cambridge. Over a couple of days we managed to fix a number of issues seen, review and document a number of issues and work on a number of ways of reducing the memory usage of GNOME. Of course this improvement is primarily seen constrained devices like the Raspberry Pi but ultimately less memory utilisation by GNOME even helps devices with decent amounts of RAM and CPUs too. The fixes didn’t arrive in time for Fedora 28, but a bunch have landed in Fedora 29 providing noticeable improvements, and the GNOME team is by no means done and there will be more coming in Fedora 30 and beyond! It was an excellent start and I expect there will be ongoing enhancements here into the future especially with devices like the Purism Phone which will have similar constraints.

Initial CPU frequency support
Another of the largest issues around the Raspberry Pi is complaints it was slow, part of the issue here is that there’s no upstream CPU Frequency driver which means all models of the Raspberry Pi run at a glacial, but safe, 600Mhz out of the box compared to the highest speed, which on the 3B+ is 1400Mhz. With Fedora 29 we’ve landed an experimental cpufreq driver which allows us to run the Raspberry Pi 3-Series at much closer to optimal speeds. While this is experimental it might not stay around if we find out it causes issues or ends up being a maintenance burden but to date it hasn’t yet appeared to have caused any issues.

HWmon Voltage Sensor
There’s a new driver that reports when the voltage supplied by the PSU drops below the required voltage. It can be a bit noisy in dmesg but one of the biggest support problems we have with the Raspberry Pis is people using a power supply that’s not powerful enough, this issue is more of a problem with Fedora 29 because with the support for running at faster frequencies due to the cpufreq driver it means we also draw more power and some PSUs that were previously fine now cause issues because they can’t supply enough current.

Enhanced support for config.txt
A lot of the hardware addons are supported in Raspbian are done by enabling things in the config.txt file, this in turn does things like loading DT overlays and merging them with the base DT to enable extra hardware like HAT support. We have enhanced the way Fedora works with this which enables us to be much closer to the way Raspbian handles these things. The advantage this has is that the documentation that’s written for Raspbian is then usable by Fedora in the wider Raspberry Pi ecosystem which in turn makes it easier for end users to get HW up and running due to less differences in process. There’s further enhancements to make here but every step closer is easier for everyone to enable and use their favourite HATs.

Improved bcm283x firmware support
In preparation for grub2 support we enhanced how we deal with the firmware that the Raspberry Pi uses for booting. This deals with the early startup. We never use to upgrade it by default to ensure things didn’t break, but it also meant most users also didn’t by default get the fixes and enhancements. Now we do. The config.txt is also handled directly which means if you never edit the file you now automatically get any changes we make, because rpm handles the file as a config file, if we change it you get a .rpmnew file so you won’t lose your changes.

Camera support
This wasn’t available in the Fedora 29 4.18 kernels, but with the rebase to the 4.19 kernel the support for the camera on the Raspberry Pi CSI Camera interface improved enough we could enable this in Fedora. The early 4.19 kernels don’t automatically detect and load support if the camera module is attached. There’s some patches in 4.20 in rawhide for this, and we’ll bring some of this to 4.19 soon, and we’re working with upstream to further improve the camera support. You’ll also want the latest bcm283x firmware which tweaks some of the config.txt and updates to a firmware with ISP fixes.

Another improvements
There was also a number of general Arm improvements which sped up crypto on the Raspberry Pi, improved the USB, fixed up some issues with the wired ethernet on the 3B+, power and a number of other fixes. As always there’s more coming. The 4.20 kernel rebase should also bring with it analog sound support early in the new year.

Conclusion
Overall I was pleased with the work that landed in Fedora over 2018 for the Raspberry Pi. The WiFi regression was disappointing, but now with that fixed in 4.19.10 we have WiFi support out of the box without users needing to download anything which moving forward will make things a lot more straight forward. The initial support for the camera makes it much more useful in numerous use cases and we’ll really polish up the HAT support in Fedora 30 which for me is the last remaining big ticket item for Raspberry Pi support. There’s still some annoying bits around the EDID detection in the display, but there’s work to improve that upstream, and also there’s work to land the media decode offloading upstream too which will also one of the few remaining bits.

Using ZRAM as swap on Fedora

One of the changes I did for Fedora 29 adding using ZRAM as swap on ARM. The use of compressed RAM for swap on constrained single board computer devices has performance advantages because the RAM is an order of faster than most of the attached storage and in the case of SD/emmc and related flash storage it also saves on the wear and tear of the flash there extending the life of the storage device.

The use of ZRAM as swap isn’t limited to constrained SBCs though, I also use it on my x86 laptop to great effect. It’s also very simple to setup.

# dnf install zram
# systemctl enable zram-swap.service
# reboot

And that’s it! Simple right? To see how it’s being used there are three commands that are useful:

# systemctl status zram-swap.service
● zram-swap.service - Enable compressed swap in memory using zram
   Loaded: loaded (/usr/lib/systemd/system/zram-swap.service; enabled; vendor preset: disabled)
   Active: active (exited) since Tue 2018-10-09 22:13:24 BST; 3 days ago
 Main PID: 1177 (code=exited, status=0/SUCCESS)
    Tasks: 0 (limit: 4915)
   Memory: 0B
   CGroup: /system.slice/zram-swap.service

Oct 09 22:13:24 localhost zramstart[1177]: Setting up swapspace version 1, size = 7.4 GiB (7960997888 bytes)
Oct 09 22:13:24 localhost zramstart[1177]: no label, UUID=d79b7cf6-41e7-4065-90a9-000811c654b4
Oct 09 22:13:24 localhost zramstart[1177]: Activated ZRAM swap device of 7961 MB
Oct 09 22:13:24 localhost systemd[1]: Started Enable compressed swap in memory using zram.
# swapon
NAME       TYPE      SIZE   USED PRIO
/dev/zram0 partition 7.4G 851.8M   -2
# zramctl
NAME       ALGORITHM DISKSIZE   DATA  COMPR  TOTAL STREAMS MOUNTPOINT
/dev/zram0 lz4           7.4G 848.3M 378.4M 389.9M       8 [SWAP]
#

When I was researching the use of ZRAM there was a lot of information online. A lot of implementations sliced up the zram into multiple slices to enable the balancing of the slices across CPUs, but this is outdated information as the zram support in recent kernels is now multi threaded so there’s no performance advantage to having multiple smaller swap devices any longer, and having a single larger swap space allows the kernel to be more effective in using it.

In Fedora all the pieces of the Fedora implementation are stored in the package source repo. So those that are interested in using zram for other use cases are free to test it. Bugs and RFEs can be reported as issues in pagure or in RHBZ like any other package.

Fedora on the UDOO Neo

Some time ago I backed the UDOO Neo Kickstarter as it looked like a nifty, well featured, IoT device. I got the full option which came with 1Gb RAM and both wired and wireless Ethernet and some add-on sensors. It was a well run kickstarter campaign and the device was well packaged with a fab box. It has both a Cortex-A9 processor to run Fedora and a Cortex-M4 embedded processor to enable you to do Arduino style functionality which should be interesting to experiment with.

For various reasons it has sat around gathering dust, it’s been a bit of a long drawn out process with me randomly poking it as time allowed.. Primarily this was because there was no decent upstream U-Boot and kernel support, and I’d not had the time to hack that up myself from various downstream git repositories, but even without Fedora support their forked Ubuntu distro in the form of UDOObuntu has an experience that is truly terrible!

Late 2016 the problem of a lack of upstream support for U-Boot and kernel changed with initial basic support landing upstream for all three (Basic, Extended and Full) models so with a few cycles over a weekend it was time to dust it off to see if I could get Fedora 26 (did I mention this has been long running?) running on it and to see what worked.

The first thing for me to do was to setup a serial console for easy debugging. The UDOO Neo documentation is generally outstanding and the pins for the UART1 TTL are documented. Two things to note here is that the headers are female rather than the usual SBC male pins so I had to bodge my usual usb to serial TTL with some male-male jumper wires and you’ll need a ground for the TTL which is undocumented on their page, I used one of the GNDs as documented on connector J7 and all was good.

So after an initial set of fixes to the U-Boot support it saw my Fedora install and started to boot! Success! Well sort of, as mentioned above the initial support is rudimentary, it started to boot the kernel and very quickly managed to corrupt and destroy the filesystem not making it much beyond switch root. That wasn’t good. In the last week or two I’ve had a little time to look again, similar issues, it was better than it was a year or so ago but it still ended up with corruption. I reached out to one of the maintainers from NXP that deals with a bunch of the i.MX platforms and I got directed to a handful of patches, a test kernel and image later and a test boot… all the way to initial-setup! SUCCESS!

The core support for the i.MX6SX SoC and the UDOO Neo is pretty reasonable, with the MMC fixes it’s been very stable, all the core bits are working as expected, included wired and wireless network, thermal, cpufreq, crypto and it looks like the display should work fine. There’s a few quirks that I need to investigate further which should provide for a fun evening or weekend hacking. There has also been recently merged support for the i.MX6SX Cortex-M4 land upstream in Zephyr upstream for the 1.13 release, so getting that running and communication using Open-AMP between Fedora and Zephyr should also be an interesting addition. I think this will be a welcome addition to Fedora 29, and not a moment too soon!!

Fedora on the UP Squared

With the IoT Working Group and Edition moving forward I’ve been looking for an x86_64 device suitable for testing IoT related use cases. I was originally planning on using the MinnowBoard or Joule but given Intel has killed those product lines off it was back to the drawing board. I eventually settled on the UP², in particular I chose the UP² Pentium-4GB-32GB-PACK as it had everything I wanted in one box.

Hardware

The on paper hardware specs show a recent generation Intel Apollo Lake core, reasonable memory and storage options, an onboard FPGA, USB-3, dual ethernet and various other bits. The kit comes with options for active or passive cooling and the later the heatsink is massive, for the moment I’m running it on the passive cooling. The case is OK, I wouldn’t rave about it though. The power connector on the other hand is terrible, the PSU cable doesn’t seat well into the board and I’ve bumped it already and had it lose power, the power button is also tiny, so small in fact I mistook it for a reset button.

Fedora support

As you would expect the support in Fedora 27 is decent, accelerated 4K graphics, wired RealTek ethernet NICs, Intel m.2 PCI-e WiFI/Bluetooth, although the later is only 4.2 for IoT I would have appreciated Bluetooth 5, all work out of the box as expected. The firmware is uEFI and in theory supports secure-boot but I couldn’t work out how to turn it on in the firmware menus as it was greyed out, it also has a TPM2 module I’ve not had time to investigate. They eagle eyed would also note that I mention Fedora 27 even though Fedora 28 has been out a few days. Well for some reason F-28 doesn’t boot, I tried the network installer and the Workstation live image, they both get to the grub menu, then I get no output and nothing for a moment then it resets and we start again. I need to investigate this further but Fedora 27 Workstation livecd booted and installed fine so that’s what it’s got for the moment. Ultimately this is going to a host to test Fedora IoT so while I tested the general support this is fine.

IoT support

There’s a number of reasons I chose this particular device as an IoT test device:

Reasonably priced with a reasonable feature set.
Intel hasn’t killed it off yet like they’ve done with the Joule platform and Minnowboard 3 so I could actually buy it 🙂
Multiple network interfaces, reasonable WiFi and Bluetooth support.
Industrial IO sensors via an onboard Intel Sensor Hub. lsiio is reporting 9 sensors of various types. I’ve not checked this further yet.
USB-3, a Raspberry Pi HAT compatible connector and other options to add IoT related functionality or interfaces.

FPGA support

I’ve not looked at the FPGA support at all. The upstream kernel now has a FPGA Manager Framework and there’s a bunch of Altera FPGA support there but I’m not sure how it maps to this device. I also have to investigate open source toolchains for FPGA bitstreams as a lot of them just aren’t, I’ll likely do the HW enablement side of things and leave the toolchain bits to people that understand them. I also have the 96boards Ultra96 board so FPGA investigation was already on my Fedora 29 To Do list, and a lot of other people seem quite interested in them of late, no idea why 😉

The Raspberry Pi 3 B+ in Fedora

So I’m sure none of you are surprised to hear that I’ve been asked a bunch about support for the Raspberry Pi 3 B+ in Fedora. Well the good news is that it’ll be supported in Fedora 28. Most of the bits were there for the official Fedora 28 beta, it just needed a minor work around, but nightly images since Beta have had all the bits integrated so the upcoming Fedora 28 GA release will support the Raspberry Pi 3 B+ to the same levels as the original 3 B on both ARMv7 and aarch64. The Fedora Raspberry Pi FAQ has now been updated with all the details of both the RPi3+ and Fedora 28.

WiFi

As with the original 3 there’s files with the firmware we can’t redistribute. The details are documented in the Fedora Raspberry Pi FAQ.

You can grab the files like for the 3 although there’s now an extra one, which you don’t really need, but it gives you all the 802.11a frequencies:

$ sudo curl https://fedora.roving-it.com/brcmfmac43455-sdio.txt -o /lib/firmware/brcm/brcmfmac43455-sdio.txt
$ sudo curl https://fedora.roving-it.com/brcmfmac43455-sdio.clm_blob -o /lib/firmware/brcm/brcmfmac43455-sdio.clm_blob

I’ve also done a rpm of the files for both editions of the RPi3 (plus a few other brcm adapters used on ARM boards). You can either just grab the rpm or setup the repo if you want to get the latest if I update it:

$ sudo curl https://fedora.roving-it.com/wireless.repo -o /etc/yum.repos.d//wireless.repo
$ sudo dnf install brcm-firmware

Issues

Like all new things there’s often a few teething problems. Over all the support for the new RPi 3 B+ is relatively solid but like all new devices there’s some bits still bedding down, and the combination of a new device and new OS there’s been a few minor issues that have been seen in some circumstances. We pulled in the latest firmware just before GA to fix some issues but no doubt as it gets wider testing both in Fedora and in the wider Raspberry Pi community more issues may well come up.

The ones we’re aware of are:

The new USB hub and gigbit ethernet interface have seen a few issues in some cases. We’ve pulled in quite a few patches to help stabilise the NIC in Linux and it now mostly works in the vast majority of use cases.
The USB hub in u-boot, uEFI and grub on aarch64 can cause some issues. If there’s too many USB devices connected it sometimes won’t boot or will do so really slowly. The work around for the moment is to disconnect all the USB devices until Fedora has started to boot and then plug them in.

In Fedora we’ll deliver updates and fixes by the usual updates, in particular as fixes land upstream we’ll review and land them where useful into Fedora, more than likely via a kernel update. If you see a uboot-tools or a bcm283x-firmware update you’ll want to run the rpi-firmware-update command to update the firmware and then reboot for it to take effect.

Older releases

We won’t be supported the new device in the older releases. Why I hear you say? Well it’s possible but it needs update to the firmware, U-Boot and kernel to work. The Raspberry Pi foundation respun Raspbian to support it and basically it’s not straight forward. Much better to have a new shiny Fedora for a new shiny device!

Fedora IoT Edition is go!

Tap tap tap… is this on?

So the Fedora Council has approved my proposal of IoT as a Council Objective. I did a presentation on my IoT proposal to the council a few weeks ago and we had an interesting and wide ranging discussion on IoT and what it means to Fedora. I was actually expecting IoT to be a Spin with a SIG to cover it but the Council decided it would be best to go the whole way and make it an Official Edition with a Working Group to back it! Amazing! One of the side effects of IoT being an accepted Objective is that the Objective Lead has a seat on the Council.

So I would say the real work starts now, but the reality is that there’s been no small amount of work I’ve been doing to get to this point, but there is also now a lot of work to do to get us to a release. We’re going to aim this initially for Fedora 29, with the intention to have a lightweight spin style process to get things up to speed as quick as possible between now and then.

So what will be happening over the coming weeks (and months)? We’ll be getting the working group in place, getting an initial monthly release process in place so that people can start to have something to kick the tires with and provide feedback and drive discussion. With those two big pieces in place we can start to grow the Fedora IoT community and work out the bits that work and bits that don’t work. Iterate early and iterate often as is often said!

So of course the big question is how do you get involved? We’ll be tracking all of the Working Group efforts in a number of places:

Fedora IoT Pagure Group: We’ll be using this for issue tracking, release milestones, and for git repositories to contain things like container recipes.
Fedora IoT mailing list: If you don’t have a FAS account you can subscribe by emailing (blank is fine) iot-subscribe AT lists.fedoraproject.org and the list server will reply with subscription options.
IRC on #fedora-iot
Fedora IoT Tracking bug: This will be primarily for tracking dependencies and component RFEs and issues.

The above list will change and evolve as we go, I expect the pagure group, mailing list and IRC to be the primary places of communication. There will of course be updates also on this blog, no doubt Fedora Magazine, FedoraIoT on twitter and elsewhere.

What will there be to do? Well lots, and that is still obviously in flux at the moment. The things that come to mind that we’ll definitely need to address will include, but certainly won’t be limited to, awesome docs, the actual OSTree Atomic host image which will be the key foundation, CI/CD pipelines to automate testing as much as possible, release processes including landing of features once they’re ready, containers and layers to add functionality, a selection of supported reference devices (see also CI/CD in this context too), various IoT frameworks, hardware enablement such as wireless standards and distinct from the supported reference HW, security (a single word can’t even begin to describe this iceberg!) and developer experience to name but a few but there’s so much more! Is everyone excited? Of course you are!

Securing home networks and IoT for family at holiday time

Many people head home to family at some point over the holiday season, whether that be like today for Thanksgiving in the US, Christian Christmas at the end of December or one of the many and varied holidays. During that time most people that are technical will be asked to help fix or setup various computer or internet related devices that family members that are not so technical have acquired or broken since the last time they ventured home. For me it use to be the regular upgrade/replacement of the Virus Scan and anti malware software. These days it tends to be patching of phones and tablets and all sorts of other devices.

So what can the average technical person do to help minimise risks to family members, or stop them from being part of a large botnet sometime in the future, without making the technology hard or even impossible for family to use, or to minimise the calls throughout the year.

Router

The first port of call should always be the router. Often these just get stuffed in the corner, on a bookshelf or somewhere out of site and forgotten. From a security point of view they are the most important, they are the thing that primarily protects everything else as they’re the ingress/egress point of the network. So what to do and change on these devices:

Upgrade the firmware to the latest supported version, and configure it to auto-upgrade if it’s an option. If the last firmware is ancient consider moving to a third party firmware like LEDE Project or an OpenWRT dirivative. Worst case scenario throw it away and give them a new one as their present.
Change the admin password.
Change the SSID and set a reasonable password.
Ensure that the admin interface isn’t available on the WAN link, do a port scan.
Turn off port forwarding and UPnP on the router.
Switch it to OpenDNS (208.67.222.222 208.67.220.220), Google Public DNS (8.8.8.8 4.4.4.4), the new Quad9, or even better a combination of them so if one service goes down or disappears their internet will still work.

Phones and Tablets

Ensure the phone is set to auto install new OS firmware releases, also ensure that apps are set to auto update and that if the provider, such as Google Play, has a malware scan option in their App store ensure that’s turned on so it’ll clean up any apps that are discovered to be problematic.

TVs, Bluerays and other Media Players

It’s surprising the number of these devices that have network connections and never get updated. In some cases the network functionality is rarely, if ever used, I know I’ve pretty much disconnected all Blu-ray players from networks, turned off the wireless if it has it, and not ever had a complaint. Often it’s better to replace some of old network media devices with ones that are actively maintained such as Google Chromecast, Amazon Fire, Roku etc. It’s also worth checking if any of these devices have the ability to connect to via ad-hoc means and disable that to limit connections to only those that are on the standard home network.

Various IoT devices

IoT devices should generally, if at all possible, be isolated on their own network. This is easy if as part of securing the router you moved it to LEDE or something similar above, and configure it to have a strict deny-by-default policy. Check the existing network for devices that are connected to it. In some cases there may have been a device connected to it some time ago that have long been forgotten about and are no longer in use, or the manufacturer has ceased to exist and they’re just a compromise waiting to happen masquerading as an expensive paperweight. Those that are in use might not be using the IoT/network functionality, if so turn the network off. Those that remain obviously ensure they’re running the latest firmware, set for auto update, and if possible move them to the IoT network. In some cases it might be possible or better to replace connected lighting if it’s some terrible WiFi/Bluetooth globe with something like the IKEA TRÅDFRI system as it has reasonable security, is of good quality and is affordable. Also don’t forget to check for things like doorbells, locks, cameras and other such devices.

Conslusion

Securing the router and associated DNS is by far and large the most important thing to do, it will help mitigate/protect most of the other problems that loom on the inside. But disconnecting, throwing away, replacement of old devices is sometimes the easiest way to fix them too, or else isolating them.

Let me know what else people do, and what I missed.

Overview of aarch64 SBC support in Fedora 27

Support for ARM 64 bit (aarch64) Single Board Computers (SBCs) has been one of the most highly requested features along side the Raspberry Pi. It’s something I’ve been working towards almost as long too. Finally with Fedora 27 I felt we would have enough of the bits in place for a Minimum Viable Product (MVP).

You’ll note from the Fedora change linked above I was very cautious with what we planned to achieve. The change has a very focused list of images: Server, Workstation and Minimal and a limited list of devices: basically the Raspberry Pi 3, the 96boards Dragonboard 410c and HiKey, and a handful of AllWinner devices with a focus on the Pine64 series of boards. The reason for this was I knew there was going to be a lot of low level boot and kernel bits that needed focus and polish and the Fedora 27 cycle was severely limited time and resource wise so the plan was to focus on getting all the core bits into place for Fedora 27 and have a couple of well polished devices and then expand that rapidly for Fedora 28.

The key functionality we were aiming for was a well polished uEFI implementation in u-boot to enable a single install/boot path in Fedora on aarch64 using uEFI/shim/grub2 to boot Fedora on both SBCs and SBSA compliant aarch64 platforms. We now have that platform in place, primarily due to Herculean efforts of Rob Clark and Peter Jones, as well as many others who have provided insight into the deep dark details of the uEFI specification. Fedora 27 will ship with a quite heavily patched, well by Fedora’s standards anyway, u-boot 2017.09 which provides us the core of this functionality enabling us to use a vanilla upstream shim and grub2 to boot a standard Fedora. All this work is already upstream, or making it’s way there in 2017.11. In Fedora 28 there will be even more improvements that will enable us to do a bunch of other cool stuff (that’ll be a post for later!) and also enable much quicker upstream board enablement now all the core bits are in place.

So what do we actually support? Well all the usual bits that you would expect on a standard Fedora install, whether it be x86_64, ARMv7 or aarch64, like SELinux, containers, desktops and all the other bits. There’s a few bits and pieces that are a little rough around the edges but overall the feature is pretty robust. On a board by board feature set lets break the this down across the boards:

Raspberry Pi 3

The support for the Raspberry Pi3 is the equivalent to the ARMv7 support but with boot via uEFI/grub2. The memory isn’t quite as good as on 32-bit but that’s to be expected, overall it’s pretty reasonable for a device of the specs and cost. Like on 32 bit support we’re seeing regular improvements each release and throughout the releases. The aarch64 support for the RPi3 is just an evolution to this.

DragonBoard 410c

The support for the DragonBoard 410c is looking pretty decent. Qualcomm has been doing a pretty decent effort to get stuff upstream, we have firmwares for the GPU and for video decode/encode upstream as well, along with kernel drivers and the open freedreno 3D drivers, HDMI audio should work as well. The WiFi firmware isn’t yet upstream but I’ll document how/where to get that and hopefully that should be in linux-firmware soon as well. Overall I’m quite happy with the status of this device, although like all devices with 1Gb RAM it’s a little constrained, but that should make the newly announced 820c with 3Gb of RAM a decent device ;-). All the details for getting it running will soon be in the Fedora 96boards wiki page.

HiKey

Most features and functionality of the HiKey are supported, note this isn’t the HiKey960 (look to F-28 for support for that), except accelerated graphics due to the use of a MALI GPU. Other than that the functionality is pretty decent. You’ll likely want the latest tianocore firmware and the details for that can be found on the Fedora 96boards wiki page.

Pine64 (AllWinner A64 SoC)

We actually should have a number of devices based on the AllWinner A64 SoC working here but we’ve only tested the 3, 2Gb/1Gb/512Mb, Pine64 device sizes. The support for these devices is headless and you will need a serial console else you’re on your own as none of the display bits in the kernel have made it upstream, and of course the GPU is a MALI 400 series so when it does it won’t be fast. The support for the rest of the device is basic, it’s usable for a headless server style device, we support network, USB, KVM, RTC and a few other bits. Other than display we don’t yet support the SDIO attached wireless, sound, crypto offload or any of the other media interfaces. A lot of this is under review upstream so I think Fedora 28 should look much better for this series of devices and 4.15 might even bring very basic console output. Speaking of series of devices which ones should actually work other than the three Pine64 devices? Well the following A64 SoC devices have a Fedora built u-boot and kernel DT support so should work as well as the Pine64: BananaPi-m64, OrangePi Win, SoPine baseboard (PineBook boots if you’re happy with serial console), NanoPi-A64 and the A64-OLinuXino. We had some troubles with the AllWinner H5 SoC devices earlier in the cycle but I’ve had a couple of reports that it seems to be resolved so they should work too and that adds the Orange Pi PC2, Prime and Zero+ 2 as well as the NanoPi NEO2. So that’s around a dozen or so devices! 🙂

Other ARM64 SBCs

I’ve had reports that other aarch64 SBCs boot on Fedora just fine. I’ve not listed those where I can’t verify whether they boot with our uEFI enabled u-boot. Looking around on my desk I do have a number of devices that I expect us to be supporting in Fedora 28, or maybe even just enabling u-boot bits in a F-27 update.

Overall I’m pretty happy with the state of Aarch64 SBCs for Fedora 27 and what we’ve managed to achieve is such a short cycle!

Raspberry Pi improvements in Fedora 26

So since I landed support for the Raspberry Pi 2 and 3 just in time for Fedora 25 Beta it’s been a bit of a fun ride. The support for Raspberry Pi is mostly done in my spare time along side all the other responsibilities I have and it’s been interesting to see people’s feedback. Going into Fedora 25 I knew it wasn’t going to be perfect but the experience was going to be reasonable for newbies to get going without generally needing serial consoles and it met Fedora’s (and mine) exacting standards on free drivers. I think we achieved that quite well but I also learned a lot in the Fedora 25 cycle and what’s coming in Fedora 26 is quite a substantial jump forward.

Hardware for a good experience

So what have I learned about the first six months or so of Raspberry Pi in Fedora? Well there’s a couple of things that the user can do to ensure a decent starting experience themselves. The biggest FAQs I’ve dealt with on the various support forums are generally fixed by these three things:

A proper spec power supply. For the RPi2 this means at least 2 AMPs and for the RPi3 at least 2.5 AMPs. If you want to plug in USB WiFi dongle and a USB HDD you’ll likely want to add a little more! In most cases an old phone charger will not suffice.
A good quality Class 10 micro SD card. I generally use Samsung EVO or SanDisk Ultra cards.
A Raspberry Pi 2 or 3. Yes, it’s surprising how many people hope to run it on something else. SORRY (actually, I’m not!)!

What’s in Fedora 26 Final

So enough of what to do! Everyone wants to know what improvements arrived in the Fedora 26 Final with the 4.11.x kernels:

Pi3 WiFi: It’s been working in F-26 since Alpha and is surprisingly stable. There’s a file you need to grab to enable it. See details in the wiki here.
Performance: In the process of dealing with wifi I worked out one of the reasons we were seeing poor performance on the SD card. We’ve had some minor improvements in F-25 but this fix over doubles the performance for me on the SD card.
HDMI video: There’s been issues around certain monitors crashing the video (vc4) driver and people getting black screens during boot. While this isn’t perfect yet (ain’t hardware great!!) it’s greatly improved across numerous devices.
Composite video: We’ve had support for the composite video since 4.10 but I need people to help test this.
Sound: HDMI audio is supported, I’ve done minor testing with the one HDMI audio capable device I have. Analogue audio out isn’t upstream yet.
HAT support: We now have all the support needed to do overlays in the kernel/bootloader and dtc stack. I just need to test it some more, document it and work out how we can best distribute pre built overlays to ease consumption. There’s still no consensus on an Overlay Manager from upstream to auto load overlays based on EEPROM on the HATs. In a lot of cases you want to load the overlays from u-boot anyway for things like display. Look out for docs and blog posts on this soon!

What arrived with the 4.12 kernel rebase

Thermal support: so if the RPi runs too hot it’ll slow it down
More performance improvements and tweaks.

What’s coming in the 4.13 kernel rebase

Bluetooth support: upstream finally tracked down the issues here. It’s been a much requested features and I should have the bits in place soon!
More performance, stability and graphics improvements and tweaks.

What about Fedora 25?

Some of the above pieces will be coming to Fedora 25 with the 4.12 rebase. The focus of my spare time is Fedora 27 mostly now, with the above coming to F-26. Some components are a lot harder to back port without issues or a complex series of package updates to ensure smooth upgrade. The WiFi and performance improvements were the hardest as part of that change moves around the use of hardware blocks and drivers. I managed to stop both the RPi2 and RPi3 booting numerous times in testing before I properly realised the implications of the change. Getting these changes for users back into a stable release without issues is hard and time consuming to do across all the various use cases. I tried this with some fixes in 4.9 and ended up making the RPi3 very unstable. This cost me a lot of time to debug and fix and I don’t really want a repeat of that!!

Graphics device

One of the surprising side effects was the discovery of a device that is five years old is that Fedora suffered from early adopters issues. We were one of the first distributions to adopt a fully upstream open kernel and graphics stack and with that came a number of issues around monitor detection, especially older/cheaper models that aren’t 1920/1080 “Full HD” or via HDMI to VGA adapters. We’re still working through these with upstream and have improved the situation quite a bit in Fedora 26 overall but it takes time and reproducible use cases which with random hardware isn’t easy or quick! 🙁

Next up?

I’ll leave Fedora 27 features and functionality for another, this post has been sitting in my drafts folder since June so it’s time to get it out and like my development move on to Fedora 27!