Why I’m not backing the Purism Librem 5 phone

NOTE: This is a post about my opinion on the device, hence the title of “Why I’m not backing….”, people have been explicitly asking me why I’ve not backed it, this documents it so I don’t need to keep repeating myself!

Numerous people have come up to me and asked “So will you get Fedora to run on your Librem phone?” and when my response is “No, I’ve not backed it” I get weird looks with a question of “Why?” I had thought it was time to do document my concerns with this laudable venture. This was certainly further qualified when I had to inform someone from the EFF that they’re delirious about the “it doesn’t need closed source firmware” on the i.MX chips that are being proposed. While I applaud the general principals and ideas of a fully open rights protecting phone I can’t help but feel that the group doing it either are being false at worst or naive at best with some of their statements.

Firmware

Their site claims “The i.MX 6/8 CPU will be completely free software without any binaries whatsoever!” while this could be sort of true if you want a hobbled device it’s not really the case at all. There’s a number of firmwares that are needed to make a number of pieces of functionality of a mobile device useful. Firstly the SDMA driver needs a firmware to run at any level of reasonable speed. Secondly the accelerated media decode, which will be required if you actually want to consume meda on your phone and have more than moments of battery life, also needs firmware. You’ll note on the media decode I don’t reference the actual firmware! Why? Because it’s not actually distributed in the linux-firmware repository so you have to request it from NXP or somewhere with appropriate signups (while I’m not 100% sure I believe having a driver in the linux kernel without required firmware in linux-firmware is a breach of the requirements of said driver in the upstream kernel).

i.MX6 or i.MX8 SoC

The i.MX 6 or i.MX 8 option concerns me. Basically make a choice! The core issue I have is the i.MX6 SoC is ancient, being announced back in Jan 2011, and based on a Cortex-A9 SoC. It doesn’t support USB-C so to ship as promised they’ll need to add a PCI-e attached USB-3 controller, charging circuitry, and maybe even a LVDS to Display Port option plus a chip to MUX the three through the actual USB-C port if they want to be able to ‘dock’ which will make power consumption on the phone even worse! If I’d spent $600 and received a phone in January 2019 based on an eight year old 32-bit chip design I’d be seriously pissed off. To be quite honest the i.MX8 is no spring chicken either, being announced in September 2013, it’s based on the already quite long in the teeth Cortex-A53 SoC design which was the original aarch64 “little” low end design but it does at least support USB-C on the SoC. The i.MX SoCs are generally quite nice, and the i.MX6 is quite well supported upstream, but the whole line of SoCs are more targeted towards embedded applications like cars, so they do have a long support cycle. They’re not a mobile phone focused SoC though, they also tend to be quite slow to get moving in the market/availability and the i.MX8 has little upstream support in the kernel as yet. Sure the etnaviv driver enables an open 3D accelerated driver, one that isn’t supported by the vendor so doesn’t enable all the features upstream, but you can run this today on Fedora 26/27 on the i.MX6 SoCs now but a free GPU driver is not the only reason to choose a platform.

General concerns

I can’t help but feel that there’s going to be a lot of disappointed people that will end up receiving an expensive sub standard device, probably late, that ends up not taking us much further along the road, to a fully open rights protecting phone, than the days of the Nokia n9xx series phones running Maemo. The HW, if based on an i.MX6, will certainly not be much further along that route and still stuck in the past in terms of HW. I personally believe the project would be better off engaging with the Qualcomm community team to use the Qualcomm 820c/600c SoCs because there’s an open driver that Qualcomm are working to improve, and while there’s a need for firmware for GPU and media offload, in reality it’s no better/worse than the i.MX devices with the bonus that their devices that are more current and aimed at mobile workloads with the vendor actively working to upsteam the enhance the support support of their SoCs.

Ultimately I think a device that closely resembles the specs of a phone of recent history, than that of ancient smart phone history, is likely to get a better following and hence a better software ecosystem than one that’s the same era as the Motorola DROID 4.

Update: So just to clarify a few things that were bought up in a couple of threads:

First this reddit thread:

  • I don’t question the value of hardware isolation of the various wireless interfaces that they claim, or the ability of them being able to deliver that bit on what ever SoC they choose, that is why it wasn’t addressed above.
  • I bought up USB-C because it is explicitly claimed as a feature.
  • I bought up display port because given USB-C above and their “It can be a desktop computer and phone all-in-one” claim, including a nice picture, then DP over USB-C is the only real option for that functionality.
  • Qualcomm chips were an example of other SoCs I believe might be a better fit, yes they do provide options in their APQ line without built in radios, they were just an example of another option, there are other possibilities, it wasn’t meant to provide a guarantee, it was an alternative example.
  • Yes, all ARM processors have onboard boot firmware, it’s generically referred to as “PBL” (Primary Boot Loader), so do most other processors.

Second this purism thread:

  • Sure you can disable the media engine with an e-FUSE but media is kind of useful for a lot of use cases like video config, audio calls and music, not just watching videos. If they choose this route, I would hope they leave the fuses unblown and document how a end user can do it else the $600 is a whole lot less useful for a lot of people

Ultimately the main points is I have to make about all of the above is two fold, firstly it’s my opinions and why I didn’t back the device, and secondly there appears to be some large discrepancies in the statements they make about the device and SoCs which I would not have expected and that is the key problem for me because it causes concerns over the ability to deliver a working and useful device to me. $600 is not a small amount of money, for me at least, to hedge on what *might* be a i.MX8 or might be an old 32 bit i.MX6.

The state of open source accelerated graphics on ARM devices

I’ve been meaning to write about the state of accelerated open source graphics options for a while now to give an update on a blog post I wrote over 5 years ago in January 2012, before the Raspberry Pi even existed! Reading back through that post it was pretty dark times for any form of GUI on ARM devices but with the massive changes in ARM devices and the massive change in SBCs (Single Board Computers) heralded by things like the Raspberry Pi have things improved at all? The answer is generally yes!

The bad

Looking back at that post the MALI situation is still just as dire with ARM still steadfastly refusing to budge. The the LIMA reverse engineering effort started with promise, but went up in smoke with a fairly public community break down, I don’t envision that situation improving any time soon although just recently there appears to be some forward movement happening finally after a long silence. This only covers the MALI-400 series and any newer GPU is a completely different architecture/IP. Even with sessions recently at Linaro Connect titled Whatโ€™s happening with ARM Mali drivers I don’t see fast change here.

The Imagination Technologies PowerVR is still just as dire as situation as it was five years ago. The company’s incompetent management recently managed to avoid being bought by Apple which in turn, because they’ve screwed the open source community while milking the Apple cash cow, essentially means they’re screwed. I suspect they’ll either open source to try and remain a relevant contender or die in a tire fire. Only time will tell there, in the mean time any ARM SoC that has this IP on board is useless for anything graphical so I’d tend to avoid it, thankfully there seems to be less of them these days.

The good

Despite the two bad examples above there’s actually been a lot of good change in the last five years. We now have a number of options for fully accelerated 2D/3D graphics on ARM SoCs and I run GNOME Shell on Wayland, yes the full open source shiny, on a number of different devices regularly.

NVIDIA true to the rumours did open up all the graphics on the Tegra series of hardware. The new Tegra K/X series have GPUs similar to their x86 offerings with Kepler/Maxwell/Pascal GPU cores but NVIDIA supports these devices by contributing to the nouveau open driver rather than the closed x86 driver. The performance on 32 bit TK1 devices has been decent for a number of releases of Fedora and improves all the time, we’ll be supporting the X series (X1/X2) with their Maxwell/Pascal GPUs in Fedora 27.

In the old post I brushed past Vivante with a mere mention of Marvell and Freescale (now NXP). The Vivante GPUs ship in NXP i.MX6 and i.MX4, some Marvell chips and some TI chips. There was a reverse engineering effort called etnaviv that must have started not long after I wrote that post and after a number of years of development support landed upstream in the kernel late 2015, and in mesa in the 7.1 release allowing us to support fully accelerated Wayland in Fedora 26! Did anyone notice? I didn’t really yell about it as much as I should have! It supports fully accelerated 3D in mesa/wayland, is pretty stable and is improving all the time, well done to all the contributors to that effort!

Another I brushed past in the old post was the Qualcomm Snapdragon SoC. They ship with a Adreno GPU. This was previously closed source, with the SoC primarily used by phone/tablet manufacturers I suspect they didn’t care… until Rob Clark (and no doubt there were other contributors) decided to reverse engineer the driver with the open freedreno driver. This is now the default driver with even Qualcomm contributing to it. We’ll support this in Fedora 27, initially with the 96boards Dragonboard 410c using the freedreno driver, but I doubt it’ll be the last Qualcomm based device we support. The Snapdragon 835 SoC, the device in all the high end Android phones this year and the ARM Windows 10 laptops, is really nice with decent performance, I’d love to be able to support a device with that SoC!

Raspberry Pi, as I mentioned in the introduction, wasn’t even out in when I wrote the original post. When it fist launched there wasn’t an open driver but 5 years later there is, sponsored by Broadcom no less. We introduced initial support for the Raspberry Pi with the open vc4 driver by Eric Anholt in Fedora 25 and it’s improving regularly. It supports fully accelerated 3D in mesa/wayland, and 2D via glamor in mesa.

So in conclusion we have improved by A LOT! We now have numerous different GPUs with open drivers to choose from in all price ranges that support fully accelerated 2D/3D desktops from four different vendors on both ARMv7 and aarch64. The media acceleration offload is also looking quite good, but that’s one for another post. The biggest holdout is MALI, and that would need two open drivers or ARM to come to the table, LIMA might work out for the 400 series, but that won’t work on the newer midguard series. With support in a number of drivers for the shiny new Wayland there’s an increasing number of devices people can use to enjoy the latest desktops fully accelerated!

Raspberry Pi improvements in Fedora 26

So since I landed support for the Raspberry Pi 2 and 3 just in time for Fedora 25 Beta it’s been a bit of a fun ride. The support for Raspberry Pi is mostly done in my spare time along side all the other responsibilities I have and it’s been interesting to see people’s feedback. Going into Fedora 25 I knew it wasn’t going to be perfect but the experience was going to be reasonable for newbies to get going without generally needing serial consoles and it met Fedora’s (and mine) exacting standards on free drivers. I think we achieved that quite well but I also learned a lot in the Fedora 25 cycle and what’s coming in Fedora 26 is quite a substantial jump forward.

Hardware for a good experience

So what have I learned about the first six months or so of Raspberry Pi in Fedora? Well there’s a couple of things that the user can do to ensure a decent starting experience themselves. The biggest FAQs I’ve dealt with on the various support forums are generally fixed by these three things:

  • A proper spec power supply. For the RPi2 this means at least 2 AMPs and for the RPi3 at least 2.5 AMPs. If you want to plug in USB WiFi dongle and a USB HDD you’ll likely want to add a little more! In most cases an old phone charger will not suffice.
  • A good quality Class 10 micro SD card. I generally use Samsung EVO or SanDisk Ultra cards.
  • A Raspberry Pi 2 or 3. Yes, it’s surprising how many people hope to run it on something else. SORRY (actually, I’m not!)!

What’s in Fedora 26 Final

So enough of what to do! Everyone wants to know what improvements arrived in the Fedora 26 Final with the 4.11.x kernels:

  • Pi3 WiFi: It’s been working in F-26 since Alpha and is surprisingly stable. There’s a file you need to grab to enable it. See details in the wiki here.
  • Performance: In the process of dealing with wifi I worked out one of the reasons we were seeing poor performance on the SD card. We’ve had some minor improvements in F-25 but this fix over doubles the performance for me on the SD card.
  • HDMI video: There’s been issues around certain monitors crashing the video (vc4) driver and people getting black screens during boot. While this isn’t perfect yet (ain’t hardware great!!) it’s greatly improved across numerous devices.
  • Composite video: We’ve had support for the composite video since 4.10 but I need people to help test this.
  • Sound: HDMI audio is supported, I’ve done minor testing with the one HDMI audio capable device I have. Analogue audio out isn’t upstream yet.
  • HAT support: We now have all the support needed to do overlays in the kernel/bootloader and dtc stack. I just need to test it some more, document it and work out how we can best distribute pre built overlays to ease consumption. There’s still no consensus on an Overlay Manager from upstream to auto load overlays based on EEPROM on the HATs. In a lot of cases you want to load the overlays from u-boot anyway for things like display. Look out for docs and blog posts on this soon!

What arrived with the 4.12 kernel rebase

  • Thermal support: so if the RPi runs too hot it’ll slow it down
  • More performance improvements and tweaks.

What’s coming in the 4.13 kernel rebase

  • Bluetooth support: upstream finally tracked down the issues here. It’s been a much requested features and I should have the bits in place soon!
  • More performance, stability and graphics improvements and tweaks.

What about Fedora 25?

Some of the above pieces will be coming to Fedora 25 with the 4.12 rebase. The focus of my spare time is Fedora 27 mostly now, with the above coming to F-26. Some components are a lot harder to back port without issues or a complex series of package updates to ensure smooth upgrade. The WiFi and performance improvements were the hardest as part of that change moves around the use of hardware blocks and drivers. I managed to stop both the RPi2 and RPi3 booting numerous times in testing before I properly realised the implications of the change. Getting these changes for users back into a stable release without issues is hard and time consuming to do across all the various use cases. I tried this with some fixes in 4.9 and ended up making the RPi3 very unstable. This cost me a lot of time to debug and fix and I don’t really want a repeat of that!!

Graphics device

One of the surprising side effects was the discovery of a device that is five years old is that Fedora suffered from early adopters issues. We were one of the first distributions to adopt a fully upstream open kernel and graphics stack and with that came a number of issues around monitor detection, especially older/cheaper models that aren’t 1920/1080 “Full HD” or via HDMI to VGA adapters. We’re still working through these with upstream and have improved the situation quite a bit in Fedora 26 overall but it takes time and reproducible use cases which with random hardware isn’t easy or quick! ๐Ÿ™

Next up?

I’ll leave Fedora 27 features and functionality for another, this post has been sitting in my drafts folder since June so it’s time to get it out and like my development move on to Fedora 27!

Configuring HTTP/2 with Apache on Fedora

HTTP/2 is the new version of the well known HTTP protocol which has been at the venerable 1.1 since late last century. Version 2 was derived out of Google’s SPDY protocol and it’s a binary protocol over the text based 1.1. It introduces a bunch of improvements including reducing latency, multiplexing, and server push. There’s some useful improvements that will be great for things like apps that use WebSockets. The Apache httpd daemon has included complete support for HTTP/2 since the 2.4.17 release in the form of mod_http2.

First you should configure your site with SSL, I suggest using LetsEncrypt/certbot as documented in this Fedora Magazine article.

Then you need to make sure the module is loaded, at least in Fedora 25 this is enabled in /etc/httpd/conf.modules.d/00-base.conf by default:

LoadModule http2_module modules/mod_http2.so

Then you just need to enable the protocol in either the general configuration or in specific VirtualHost directives for specific sites:

# for a https server
Protocols h2 http/1.1

# for a http server
Protocols h2c http/1.1

Then it’s just a systemctl restart httpd to make the changes take effect.

To test whether you’re serving over HTTP/2 you can test using this HTTP/2 testing site or with the OpenSSL client (check for “ALPN protocol: h2” in the output) with the following command:

openssl s_client -alpn h2 -connect HOSTNAME:443

Note: HTTP/2 is not currently supported in the httpd shipped in RHEL.

Getting started with Zephyr on Fedora

So while Fedora is great for a lot of IoT use cases it can’t be used everywhere, such as on tiny micro controllers such as an ARM Cortex-M series or Intel Quark micro controllers, but that doesn’t mean that Fedora doesn’t make a fantastic developer platform for working with these devices.

I have a handful of Zephyr capable devices (BBC Micro:bit, NXP FRDM-K64F, 96Boards Carbon, TI CC3200 LaunchPad) so how can you get a build environment up and running quickly so you can start doing real development as quickly as possible.

In testing this I used a Digital Ocean cloud instance for a build host. Wherever you choose to build it make sure you have at least 2GB of RAM available as from my experience you need at least 2GB for building a Zephyr image.

From there we diverge a little from the upstream notes by installing the Fedora ARM cross compiler (only tested with ARM, not sure of state of other targets) and developer tools:

sudo dnf install git-core gcc gcc-arm-linux-gnu glibc-static libstdc++-static make dfu-util dtc python3-PyYAML

Next up we clone the upstream Zephyr git repository:

git clone https://gerrit.zephyrproject.org/r/zephyr zephyr-project

If we want to use a particular stable branch we now switch to the chosen branch. I’m using the latest stable release branch:

cd zephyr-project; git checkout v1.7-branch

Set up the cross compiler variables:

export GCCARMEMB_TOOLCHAIN_PATH="/usr"
source zephyr-env.sh
cd $ZEPHYR_BASE/samples/hello_world

Select and build our target:

make CROSS_COMPILE="/usr/bin/arm-linux-gnu-" DTC=/usr/bin/dtc BOARD=96b_carbon

If we’re developing this on our local machine we can now just directly flash the new build straight to the device. To do this we connect a micro USB cable to the USB OTG port on the Carbon and to your computer. The board should power on. Force the board into DFU mode by keeping the BOOT0 switch pressed while pressing and releasing the RST switch.

Confirm DFU can see the device:

$ sudo dfu-util -l
dfu-util 0.9

Copyright 2005-2009 Weston Schmidt, Harald Welte and OpenMoko Inc.
Copyright 2010-2016 Tormod Volden and Stefan Schmidt
This program is Free Software and has ABSOLUTELY NO WARRANTY
Please report bugs to http://sourceforge.net/p/dfu-util/tickets/

Found DFU: [0483:df11] ver=2200, devnum=8, cfg=1, intf=0, path="2-1", alt=3, name="@Device Feature/0xFFFF0000/01*004 e", serial="123456789"
Found DFU: [0483:df11] ver=2200, devnum=8, cfg=1, intf=0, path="2-1", alt=2, name="@OTP Memory /0x1FFF7800/01*512 e,01*016 e", serial="123456789"
Found DFU: [0483:df11] ver=2200, devnum=8, cfg=1, intf=0, path="2-1", alt=1, name="@Option Bytes  /0x1FFFC000/01*016 e", serial="123456789"
Found DFU: [0483:df11] ver=2200, devnum=8, cfg=1, intf=0, path="2-1", alt=0, name="@Internal Flash  /0x08000000/04*016Kg,01*064Kg,03*128Kg", serial="123456789"

Flash our build onto the device:

sudo dfu-util -d [0483:df11] -a 0 -D outdir/96b_carbon/zephyr.bin -s 0x08000000

Now connect another micro USB cable to the UART port and run a console:

sudo screen /dev/ttyUSB0 115200

Hit the reset button and you should see the following output:

***** BOOTING ZEPHYR OS v1.7.1 - BUILD: Jun  6 2017 14:07:24 *****
Hello World! arm

Now we have a basic development environment setup, know we can build, flash and run a release on the 96boards Carbon next time we can do something more advanced ๐Ÿ˜‰

Update (2017-06-13): Minor updates to dependency installs and make command

WiFi on Raspberry Pi 3 for Fedora 26 Alpha

So I managed to land just about everything needed for the WiFi on the Raspberry Pi 3 for Fedora 26 Alpha (around 4.11 rc3). There’s one thing missing, because we can’t currently redistribute it, but it’s straight forward for the end user to do themselves once they’ve done the initial setup:

sudo curl https://raw.githubusercontent.com/RPi-Distro/firmware-nonfree/master/brcm80211/brcm/brcmfmac43430-sdio.txt -o /lib/firmware/brcm/brcmfmac43430-sdio.txt

Or you can also do it when you’re flashing the image if you mount the root filesystem but the above is likely easier. It’s been surprisingly stable in my testing.

Before you all ask, at the moment I don’t plan on pushing this to earlier Fedora releases, as the upgrade path is not trivial. I will also soon publish more details of some of the other new features coming for the Raspberry Pi to Fedora 26 but I thought you’d all like the WiFi details now. The wiki has also been updated to reflect the status of the WiFi.

PS: No this is not an April Fool’s joke (it’s well past midday in UK).

Updating Raspberry Pi firmware on Fedora

The upstream Raspberry Pi firmware/bootloader gets regular updates and improvements. In Fedora we ship that firmware in a package called bcm283x-firmware. I regularly follow the git repo of the upstream firmware and on occasion when I believe there’s reasonable changes that benefit Fedora I’ll prepare a new version, do some brief testing on my devices to make sure it boots and basic functionality hasn’t regressed at which point I’ll update the package and send it out to supported releases as an update.

Once the new bcm283x-firmware lands on your Raspberry Pi it doesn’t automatically update the firmware though. Why is that you ask? I don’t like to spring surprises on people where they end up with a device that might not boot or it might regress things they care about.

So how do you upgrade the firmware for the Raspberry Pi on Fedora? It’s simple! You simply run the command rpi-firmware-update and it’ll update the firmware and the u-boot to the latest one that’s shipped as a Fedora package. Then you just need to reboot to make it active.

The easiest way to work out which firmware you’re currently running is “dmesg | grep raspberrypi-firmware”

I tend to try and push out a new firmware update every month or so but if I see something that’s of interest or that fixes known issues I do it as needed.

Useful commands for manipulating PDFs on Fedora (any linux really)

So it’s not much of a secret that the Red Hat expenses system is truly terrible. Not a well known is the EMEA accounts team still require what I call “Arts and Crafts sessions” (all receipts attached to bits of paper and scanned as a whole) even though there’s no legal requirement for paper receipts to be provided any more in the UK/Ireland/EU!

Anyway the system regularly routes the emailed PDFs to /dev/null for no apparent reason and then you have to scratch your head and try and work out what’s wrong.

Size: this is the regular issue, basically if the PDF is larger than a few Mb it barfs. Thankfully ghostscript comes to the rescue here.

gs -dNOPAUSE -dBATCH -sDEVICE=pdfwrite -dCompatibilityLevel=1.4 -dPDFSETTINGS=/screen -sOutputFile=new_file.pdf original_file.pdf

The -dPDFSETTINGS setting has a few options:

  • /screen selects low-resolution output and hence the lowest file-size.
  • /ebook selects medium-resolution output with a medium file-size.
  • /printer uses high-resolution option which is mainly used for printing PDFs.
  • /prepress) similar to /printer but gives you the largest files.

Too many pages: Airline bookings are the great ones here, they add pages of adds to a one page receipt. Two pages are either to “print” just the page you need, or use oowriter (Libre Office Writer) to open it, delete the pages and export as PDF again.

Multiple PDFs: In theory the system can handle multiple docs. My millage has varied a LOT here. Easy fix comes from the poppler-utils package:

pdfunite doc-1.pdf doc-2.pdf doc-3.pdf out-doc.pdf

PDF versions: I have found 1.4 be the most effective here. Ghostscript comes to the rescue again here:

gs -sDEVICE=pdfwrite -dCompatibilityLevel=1.4 -dPDFSETTINGS=/screen -dNOPAUSE -dQUIET -dBATCH -sOutputFile=output.pdf input.pdf

Adjust the level -dCompatibilityLevel to the version you need.

Connect to a wireless network using command line nmcli

I use a lot of minimal installs on various ARM devices. They’re good because they’re quick to download and you can test most of the functionality of the device to ensure it’s working or to quickly test specific functionality but of course it doesn’t have a GUI to use the nice graphical tools which are useful to quickly connect to a wifi network or other things.

This where nmcli comes in handy to quickly do anything you can do with the GUI. To connect to a wireless network I do:

Check you can see the wireless NIC and that the radio is enabled (basically “Airplane” mode):

# nmcli radio
WIFI-HW  WIFI     WWAN-HW  WWAN    
enabled  enabled  enabled  enabled 
# nmcli device
DEVICE  TYPE      STATE         CONNECTION 
wlan0   wifi      disconnected  --         
eth0    ethernet  unavailable   --         
lo      loopback  unmanaged     --         

Then to actually connect to a wireless AP:

# nmcli device wifi rescan
# nmcli device wifi list
# nmcli device wifi connect SSID-Name password wireless-password

And that should be enough to get you connected. You can list the connection with nmcli connection and various other options. It’s pretty straight forward. The only complaint I have is that it doesn’t prompt for a password if you leave it out so it means the AP password is stored on the command line history. Not a major given it’s quite easy to find where it’s stored anyway on the system but it would be a useful addition.

When creating updates remember to build for rawhide and Fedora 25 (devel)

When ever we branch for a new release of Fedora I, and others, end up spending a non trivial amount of time ensuring that there’s a clean upgrade path for packages. From the moment we branch you need to build new versions and bug fixes of packages for rawhide (currently what will become Fedora 26), for the current stabilising release (what will become Fedora 25) as well as what ever stable releases you need to push the fix for. For rawhide you don’t need to submit it as an update but for the current release that’s stabilising you do need to submit it as an update as it won’t just automagically get tagged into the release.

As a packager you should know this, it’s been like it for a VERY LONG TIME! Yet each cycle from the moment of branching right through to when a new release goes GA I still end up having to fix packages that “get downgraded” when people upgrade between releases!!

So far this cycle I’ve fixed about 20 odd with the latest being bash-completion (built but not submitted as an update for F-25) and certmonger (numerous fixes missing from F-25 and master branch).

The other silly packaging bug I end up having to fix quite a bit is NVR downgrades where even though it’s a newer package the way the NVR is handled makes rpm/dnf/yum think the newer package is a lesser version than the current version and hence you’re new shiny fix won’t actually make it to end users. I see this a lot where people push a beta/RC package to a devel (F-25/rawhide) release. Just something to be aware of, there’s lots of good docs around the way rpm/dnf/yum handles eNVR upgrades.