The state of open source accelerated graphics on ARM devices

I’ve been meaning to write about the state of accelerated open source graphics options for a while now to give an update on a blog post I wrote over 5 years ago in January 2012, before the Raspberry Pi even existed! Reading back through that post it was pretty dark times for any form of GUI on ARM devices but with the massive changes in ARM devices and the massive change in SBCs (Single Board Computers) heralded by things like the Raspberry Pi have things improved at all? The answer is generally yes!

The bad

Looking back at that post the MALI situation is still just as dire with ARM still steadfastly refusing to budge. The the LIMA reverse engineering effort started with promise, but went up in smoke with a fairly public community break down, I don’t envision that situation improving any time soon although just recently there appears to be some forward movement happening finally after a long silence. This only covers the MALI-400 series and any newer GPU is a completely different architecture/IP. Even with sessions recently at Linaro Connect titled Whatโ€™s happening with ARM Mali drivers I don’t see fast change here.

The Imagination Technologies PowerVR is still just as dire as situation as it was five years ago. The company’s incompetent management recently managed to avoid being bought by Apple which in turn, because they’ve screwed the open source community while milking the Apple cash cow, essentially means they’re screwed. I suspect they’ll either open source to try and remain a relevant contender or die in a tire fire. Only time will tell there, in the mean time any ARM SoC that has this IP on board is useless for anything graphical so I’d tend to avoid it, thankfully there seems to be less of them these days.

The good

Despite the two bad examples above there’s actually been a lot of good change in the last five years. We now have a number of options for fully accelerated 2D/3D graphics on ARM SoCs and I run GNOME Shell on Wayland, yes the full open source shiny, on a number of different devices regularly.

NVIDIA true to the rumours did open up all the graphics on the Tegra series of hardware. The new Tegra K/X series have GPUs similar to their x86 offerings with Kepler/Maxwell/Pascal GPU cores but NVIDIA supports these devices by contributing to the nouveau open driver rather than the closed x86 driver. The performance on 32 bit TK1 devices has been decent for a number of releases of Fedora and improves all the time, we’ll be supporting the X series (X1/X2) with their Maxwell/Pascal GPUs in Fedora 27.

In the old post I brushed past Vivante with a mere mention of Marvell and Freescale (now NXP). The Vivante GPUs ship in NXP i.MX6 and i.MX4, some Marvell chips and some TI chips. There was a reverse engineering effort called etnaviv that must have started not long after I wrote that post and after a number of years of development support landed upstream in the kernel late 2015, and in mesa in the 7.1 release allowing us to support fully accelerated Wayland in Fedora 26! Did anyone notice? I didn’t really yell about it as much as I should have! It supports fully accelerated 3D in mesa/wayland, is pretty stable and is improving all the time, well done to all the contributors to that effort!

Another I brushed past in the old post was the Qualcomm Snapdragon SoC. They ship with a Adreno GPU. This was previously closed source, with the SoC primarily used by phone/tablet manufacturers I suspect they didn’t care… until Rob Clark (and no doubt there were other contributors) decided to reverse engineer the driver with the open freedreno driver. This is now the default driver with even Qualcomm contributing to it. We’ll support this in Fedora 27, initially with the 96boards Dragonboard 410c using the freedreno driver, but I doubt it’ll be the last Qualcomm based device we support. The Snapdragon 835 SoC, the device in all the high end Android phones this year and the ARM Windows 10 laptops, is really nice with decent performance, I’d love to be able to support a device with that SoC!

Raspberry Pi, as I mentioned in the introduction, wasn’t even out in when I wrote the original post. When it fist launched there wasn’t an open driver but 5 years later there is, sponsored by Broadcom no less. We introduced initial support for the Raspberry Pi with the open vc4 driver by Eric Anholt in Fedora 25 and it’s improving regularly. It supports fully accelerated 3D in mesa/wayland, and 2D via glamor in mesa.

So in conclusion we have improved by A LOT! We now have numerous different GPUs with open drivers to choose from in all price ranges that support fully accelerated 2D/3D desktops from four different vendors on both ARMv7 and aarch64. The media acceleration offload is also looking quite good, but that’s one for another post. The biggest holdout is MALI, and that would need two open drivers or ARM to come to the table, LIMA might work out for the 400 series, but that won’t work on the newer midguard series. With support in a number of drivers for the shiny new Wayland there’s an increasing number of devices people can use to enjoy the latest desktops fully accelerated!

Raspberry Pi improvements in Fedora 26

So since I landed support for the Raspberry Pi 2 and 3 just in time for Fedora 25 Beta it’s been a bit of a fun ride. The support for Raspberry Pi is mostly done in my spare time along side all the other responsibilities I have and it’s been interesting to see people’s feedback. Going into Fedora 25 I knew it wasn’t going to be perfect but the experience was going to be reasonable for newbies to get going without generally needing serial consoles and it met Fedora’s (and mine) exacting standards on free drivers. I think we achieved that quite well but I also learned a lot in the Fedora 25 cycle and what’s coming in Fedora 26 is quite a substantial jump forward.

Hardware for a good experience

So what have I learned about the first six months or so of Raspberry Pi in Fedora? Well there’s a couple of things that the user can do to ensure a decent starting experience themselves. The biggest FAQs I’ve dealt with on the various support forums are generally fixed by these three things:

  • A proper spec power supply. For the RPi2 this means at least 2 AMPs and for the RPi3 at least 2.5 AMPs. If you want to plug in USB WiFi dongle and a USB HDD you’ll likely want to add a little more! In most cases an old phone charger will not suffice.
  • A good quality Class 10 micro SD card. I generally use Samsung EVO or SanDisk Ultra cards.
  • A Raspberry Pi 2 or 3. Yes, it’s surprising how many people hope to run it on something else. SORRY (actually, I’m not!)!

What’s in Fedora 26 Final

So enough of what to do! Everyone wants to know what improvements arrived in the Fedora 26 Final with the 4.11.x kernels:

  • Pi3 WiFi: It’s been working in F-26 since Alpha and is surprisingly stable. There’s a file you need to grab to enable it. See details in the wiki here.
  • Performance: In the process of dealing with wifi I worked out one of the reasons we were seeing poor performance on the SD card. We’ve had some minor improvements in F-25 but this fix over doubles the performance for me on the SD card.
  • HDMI video: There’s been issues around certain monitors crashing the video (vc4) driver and people getting black screens during boot. While this isn’t perfect yet (ain’t hardware great!!) it’s greatly improved across numerous devices.
  • Composite video: We’ve had support for the composite video since 4.10 but I need people to help test this.
  • Sound: HDMI audio is supported, I’ve done minor testing with the one HDMI audio capable device I have. Analogue audio out isn’t upstream yet.
  • HAT support: We now have all the support needed to do overlays in the kernel/bootloader and dtc stack. I just need to test it some more, document it and work out how we can best distribute pre built overlays to ease consumption. There’s still no consensus on an Overlay Manager from upstream to auto load overlays based on EEPROM on the HATs. In a lot of cases you want to load the overlays from u-boot anyway for things like display. Look out for docs and blog posts on this soon!

What arrived with the 4.12 kernel rebase

  • Thermal support: so if the RPi runs too hot it’ll slow it down
  • More performance improvements and tweaks.

What’s coming in the 4.13 kernel rebase

  • Bluetooth support: upstream finally tracked down the issues here. It’s been a much requested features and I should have the bits in place soon!
  • More performance, stability and graphics improvements and tweaks.

What about Fedora 25?

Some of the above pieces will be coming to Fedora 25 with the 4.12 rebase. The focus of my spare time is Fedora 27 mostly now, with the above coming to F-26. Some components are a lot harder to back port without issues or a complex series of package updates to ensure smooth upgrade. The WiFi and performance improvements were the hardest as part of that change moves around the use of hardware blocks and drivers. I managed to stop both the RPi2 and RPi3 booting numerous times in testing before I properly realised the implications of the change. Getting these changes for users back into a stable release without issues is hard and time consuming to do across all the various use cases. I tried this with some fixes in 4.9 and ended up making the RPi3 very unstable. This cost me a lot of time to debug and fix and I don’t really want a repeat of that!!

Graphics device

One of the surprising side effects was the discovery of a device that is five years old is that Fedora suffered from early adopters issues. We were one of the first distributions to adopt a fully upstream open kernel and graphics stack and with that came a number of issues around monitor detection, especially older/cheaper models that aren’t 1920/1080 “Full HD” or via HDMI to VGA adapters. We’re still working through these with upstream and have improved the situation quite a bit in Fedora 26 overall but it takes time and reproducible use cases which with random hardware isn’t easy or quick! ๐Ÿ™

Next up?

I’ll leave Fedora 27 features and functionality for another, this post has been sitting in my drafts folder since June so it’s time to get it out and like my development move on to Fedora 27!

Configuring HTTP/2 with Apache on Fedora

HTTP/2 is the new version of the well known HTTP protocol which has been at the venerable 1.1 since late last century. Version 2 was derived out of Google’s SPDY protocol and it’s a binary protocol over the text based 1.1. It introduces a bunch of improvements including reducing latency, multiplexing, and server push. There’s some useful improvements that will be great for things like apps that use WebSockets. The Apache httpd daemon has included complete support for HTTP/2 since the 2.4.17 release in the form of mod_http2.

First you should configure your site with SSL, I suggest using LetsEncrypt/certbot as documented in this Fedora Magazine article.

Then you need to make sure the module is loaded, at least in Fedora 25 this is enabled in /etc/httpd/conf.modules.d/00-base.conf by default:

LoadModule http2_module modules/

Then you just need to enable the protocol in either the general configuration or in specific VirtualHost directives for specific sites:

# for a https server
Protocols h2 http/1.1

# for a http server
Protocols h2c http/1.1

Then it’s just a systemctl restart httpd to make the changes take effect.

To test whether you’re serving over HTTP/2 you can test using this HTTP/2 testing site or with the OpenSSL client (check for “ALPN protocol: h2” in the output) with the following command:

openssl s_client -alpn h2 -connect HOSTNAME:443

Note: HTTP/2 is not currently supported in the httpd shipped in RHEL.

Getting started with Zephyr on Fedora

So while Fedora is great for a lot of IoT use cases it can’t be used everywhere, such as on tiny micro controllers such as an ARM Cortex-M series or Intel Quark micro controllers, but that doesn’t mean that Fedora doesn’t make a fantastic developer platform for working with these devices.

I have a handful of Zephyr capable devices (BBC Micro:bit, NXP FRDM-K64F, 96Boards Carbon, TI CC3200 LaunchPad) so how can you get a build environment up and running quickly so you can start doing real development as quickly as possible.

In testing this I used a Digital Ocean cloud instance for a build host. Wherever you choose to build it make sure you have at least 2GB of RAM available as from my experience you need at least 2GB for building a Zephyr image.

From there we diverge a little from the upstream notes by installing the Fedora ARM cross compiler (only tested with ARM, not sure of state of other targets) and developer tools:

sudo dnf install git-core gcc gcc-arm-linux-gnu glibc-static libstdc++-static make dfu-util dtc python3-PyYAML

Next up we clone the upstream Zephyr git repository:

git clone zephyr-project

If we want to use a particular stable branch we now switch to the chosen branch. I’m using the latest stable release branch:

cd zephyr-project; git checkout v1.7-branch

Set up the cross compiler variables:

cd $ZEPHYR_BASE/samples/hello_world

Select and build our target:

make CROSS_COMPILE="/usr/bin/arm-linux-gnu-" DTC=/usr/bin/dtc BOARD=96b_carbon

If we’re developing this on our local machine we can now just directly flash the new build straight to the device. To do this we connect a micro USB cable to the USB OTG port on the Carbon and to your computer. The board should power on. Force the board into DFU mode by keeping the BOOT0 switch pressed while pressing and releasing the RST switch.

Confirm DFU can see the device:

$ sudo dfu-util -l
dfu-util 0.9

Copyright 2005-2009 Weston Schmidt, Harald Welte and OpenMoko Inc.
Copyright 2010-2016 Tormod Volden and Stefan Schmidt
This program is Free Software and has ABSOLUTELY NO WARRANTY
Please report bugs to

Found DFU: [0483:df11] ver=2200, devnum=8, cfg=1, intf=0, path="2-1", alt=3, name="@Device Feature/0xFFFF0000/01*004 e", serial="123456789"
Found DFU: [0483:df11] ver=2200, devnum=8, cfg=1, intf=0, path="2-1", alt=2, name="@OTP Memory /0x1FFF7800/01*512 e,01*016 e", serial="123456789"
Found DFU: [0483:df11] ver=2200, devnum=8, cfg=1, intf=0, path="2-1", alt=1, name="@Option Bytes  /0x1FFFC000/01*016 e", serial="123456789"
Found DFU: [0483:df11] ver=2200, devnum=8, cfg=1, intf=0, path="2-1", alt=0, name="@Internal Flash  /0x08000000/04*016Kg,01*064Kg,03*128Kg", serial="123456789"

Flash our build onto the device:

sudo dfu-util -d [0483:df11] -a 0 -D outdir/96b_carbon/zephyr.bin -s 0x08000000

Now connect another micro USB cable to the UART port and run a console:

sudo screen /dev/ttyUSB0 115200

Hit the reset button and you should see the following output:

***** BOOTING ZEPHYR OS v1.7.1 - BUILD: Jun  6 2017 14:07:24 *****
Hello World! arm

Now we have a basic development environment setup, know we can build, flash and run a release on the 96boards Carbon next time we can do something more advanced ๐Ÿ˜‰

Update (2017-06-13): Minor updates to dependency installs and make command

WiFi on Raspberry Pi 3 for Fedora 26 Alpha

So I managed to land just about everything needed for the WiFi on the Raspberry Pi 3 for Fedora 26 Alpha (around 4.11 rc3). There’s one thing missing, because we can’t currently redistribute it, but it’s straight forward for the end user to do themselves once they’ve done the initial setup:

sudo curl -o /lib/firmware/brcm/brcmfmac43430-sdio.txt

Or you can also do it when you’re flashing the image if you mount the root filesystem but the above is likely easier. It’s been surprisingly stable in my testing.

Before you all ask, at the moment I don’t plan on pushing this to earlier Fedora releases, as the upgrade path is not trivial. I will also soon publish more details of some of the other new features coming for the Raspberry Pi to Fedora 26 but I thought you’d all like the WiFi details now. The wiki has also been updated to reflect the status of the WiFi.

PS: No this is not an April Fool’s joke (it’s well past midday in UK).

Updating Raspberry Pi firmware on Fedora

The upstream Raspberry Pi firmware/bootloader gets regular updates and improvements. In Fedora we ship that firmware in a package called bcm283x-firmware. I regularly follow the git repo of the upstream firmware and on occasion when I believe there’s reasonable changes that benefit Fedora I’ll prepare a new version, do some brief testing on my devices to make sure it boots and basic functionality hasn’t regressed at which point I’ll update the package and send it out to supported releases as an update.

Once the new bcm283x-firmware lands on your Raspberry Pi it doesn’t automatically update the firmware though. Why is that you ask? I don’t like to spring surprises on people where they end up with a device that might not boot or it might regress things they care about.

So how do you upgrade the firmware for the Raspberry Pi on Fedora? It’s simple! You simply run the command rpi-firmware-update and it’ll update the firmware and the u-boot to the latest one that’s shipped as a Fedora package. Then you just need to reboot to make it active.

The easiest way to work out which firmware you’re currently running is “dmesg | grep raspberrypi-firmware”

I tend to try and push out a new firmware update every month or so but if I see something that’s of interest or that fixes known issues I do it as needed.

Useful commands for manipulating PDFs on Fedora (any linux really)

So it’s not much of a secret that the Red Hat expenses system is truly terrible. Not a well known is the EMEA accounts team still require what I call “Arts and Crafts sessions” (all receipts attached to bits of paper and scanned as a whole) even though there’s no legal requirement for paper receipts to be provided any more in the UK/Ireland/EU!

Anyway the system regularly routes the emailed PDFs to /dev/null for no apparent reason and then you have to scratch your head and try and work out what’s wrong.

Size: this is the regular issue, basically if the PDF is larger than a few Mb it barfs. Thankfully ghostscript comes to the rescue here.

gs -dNOPAUSE -dBATCH -sDEVICE=pdfwrite -dCompatibilityLevel=1.4 -dPDFSETTINGS=/screen -sOutputFile=new_file.pdf original_file.pdf

The -dPDFSETTINGS setting has a few options:

  • /screen selects low-resolution output and hence the lowest file-size.
  • /ebook selects medium-resolution output with a medium file-size.
  • /printer uses high-resolution option which is mainly used for printing PDFs.
  • /prepress) similar to /printer but gives you the largest files.

Too many pages: Airline bookings are the great ones here, they add pages of adds to a one page receipt. Two pages are either to “print” just the page you need, or use oowriter (Libre Office Writer) to open it, delete the pages and export as PDF again.

Multiple PDFs: In theory the system can handle multiple docs. My millage has varied a LOT here. Easy fix comes from the poppler-utils package:

pdfunite doc-1.pdf doc-2.pdf doc-3.pdf out-doc.pdf

PDF versions: I have found 1.4 be the most effective here. Ghostscript comes to the rescue again here:

gs -sDEVICE=pdfwrite -dCompatibilityLevel=1.4 -dPDFSETTINGS=/screen -dNOPAUSE -dQUIET -dBATCH -sOutputFile=output.pdf input.pdf

Adjust the level -dCompatibilityLevel to the version you need.

Connect to a wireless network using command line nmcli

I use a lot of minimal installs on various ARM devices. They’re good because they’re quick to download and you can test most of the functionality of the device to ensure it’s working or to quickly test specific functionality but of course it doesn’t have a GUI to use the nice graphical tools which are useful to quickly connect to a wifi network or other things.

This where nmcli comes in handy to quickly do anything you can do with the GUI. To connect to a wireless network I do:

Check you can see the wireless NIC and that the radio is enabled (basically “Airplane” mode):

# nmcli radio
enabled  enabled  enabled  enabled 
# nmcli device
wlan0   wifi      disconnected  --         
eth0    ethernet  unavailable   --         
lo      loopback  unmanaged     --         

Then to actually connect to a wireless AP:

# nmcli device wifi rescan
# nmcli device wifi list
# nmcli device wifi connect SSID-Name password wireless-password

And that should be enough to get you connected. You can list the connection with nmcli connection and various other options. It’s pretty straight forward. The only complaint I have is that it doesn’t prompt for a password if you leave it out so it means the AP password is stored on the command line history. Not a major given it’s quite easy to find where it’s stored anyway on the system but it would be a useful addition.

When creating updates remember to build for rawhide and Fedora 25 (devel)

When ever we branch for a new release of Fedora I, and others, end up spending a non trivial amount of time ensuring that there’s a clean upgrade path for packages. From the moment we branch you need to build new versions and bug fixes of packages for rawhide (currently what will become Fedora 26), for the current stabilising release (what will become Fedora 25) as well as what ever stable releases you need to push the fix for. For rawhide you don’t need to submit it as an update but for the current release that’s stabilising you do need to submit it as an update as it won’t just automagically get tagged into the release.

As a packager you should know this, it’s been like it for a VERY LONG TIME! Yet each cycle from the moment of branching right through to when a new release goes GA I still end up having to fix packages that “get downgraded” when people upgrade between releases!!

So far this cycle I’ve fixed about 20 odd with the latest being bash-completion (built but not submitted as an update for F-25) and certmonger (numerous fixes missing from F-25 and master branch).

The other silly packaging bug I end up having to fix quite a bit is NVR downgrades where even though it’s a newer package the way the NVR is handled makes rpm/dnf/yum think the newer package is a lesser version than the current version and hence you’re new shiny fix won’t actually make it to end users. I see this a lot where people push a beta/RC package to a devel (F-25/rawhide) release. Just something to be aware of, there’s lots of good docs around the way rpm/dnf/yum handles eNVR upgrades.

Fedora 24 released on all architectures simultaneously!!

So for the first time ever we’ve released Fedora 24 across both primary and alternate architectures pretty much simultaneously! That’s the three primary architectures, x86_64, ARMv7 and i686, plus the alternate architectures of aarch64, ppc64, ppc64le and s390x. This is the first time we’ve ever released SEVEN architectures on the same day!

Fedora 24 from a Release Engineering perspective has been fairly momentous, we’ve made the single biggest change to our tooling that composes the release since Fedora Core and Fedora Extras was merged in Fedora 7! The plans for this change had been long discussed and first started to move back in the Fedora 21 cycle with the hope it could be implemented in Fedora 22…. boy were we wrong! But on the flip slide, while it’s still not perfect, it has massively improved the process for the releases. We now FINALLY have a full compose every single night! No more Test Composes! It now means QA can automate tests against any bit of the compose output, it means the installer team can see the results of their changes the next day. For the average user this has no material visible impact, for those much closer into the process it has a hugely positive impact!

From the alternate “secondary” architecture perspective I’ve ticked of a massive amount of “goal check boxes” that I made for myself when I joined this role almost two years ago.

My big ticket item was “make the release process like primary”. Our release process was a lot more manual than “primary” but we had also a lot less “features”. Well with this release neither are now true. We’re not 100% of the primary process, but the difference is tiny. We have a full nightly compose now, like primary, where previously we basically had a “newRepo” process we now have a full installer stack with network and iso installs produced. This in itself is a massive improvement! To this we add docker to aarch64 and Power64, cloud to aarch64 (and more automation of cloud to Power64), and initial “tech preview” disk images to aarch64 for Single Board Computers (I’m sorry I really did want to nail down this feature but even I have to sleep!) like the Pine64.

Some of the “smaller” tick box improvements to non primary arches is that we’re now fully 100% ansible run for the aarch64 and Power64 infrastructure, and only have a few minor bits to work out for s390x. This means from the back end everything is 100% like primary and orchestrated as such, the hubs all looks the same and the experience is consistent. Changes are deployed simultaneously and hence consistently. YAY automation! We’ve also simplified the way the secondary content gets to the mirrors so it’s more consistent and faster!

On the not x86 architecture front we’ve got a whole bunch more exciting features planned, and improvements to make, for the Fedora 25 cycle and beyond, some of which will begin happening very soon. Watch this space, the second half of the year looks to me to be just as frantic as the first!!