Fedora 24 released on all architectures simultaneously!!

So for the first time ever we’ve released Fedora 24 across both primary and alternate architectures pretty much simultaneously! That’s the three primary architectures, x86_64, ARMv7 and i686, plus the alternate architectures of aarch64, ppc64, ppc64le and s390x. This is the first time we’ve ever released SEVEN architectures on the same day!

Fedora 24 from a Release Engineering perspective has been fairly momentous, we’ve made the single biggest change to our tooling that composes the release since Fedora Core and Fedora Extras was merged in Fedora 7! The plans for this change had been long discussed and first started to move back in the Fedora 21 cycle with the hope it could be implemented in Fedora 22…. boy were we wrong! But on the flip slide, while it’s still not perfect, it has massively improved the process for the releases. We now FINALLY have a full compose every single night! No more Test Composes! It now means QA can automate tests against any bit of the compose output, it means the installer team can see the results of their changes the next day. For the average user this has no material visible impact, for those much closer into the process it has a hugely positive impact!

From the alternate “secondary” architecture perspective I’ve ticked of a massive amount of “goal check boxes” that I made for myself when I joined this role almost two years ago.

My big ticket item was “make the release process like primary”. Our release process was a lot more manual than “primary” but we had also a lot less “features”. Well with this release neither are now true. We’re not 100% of the primary process, but the difference is tiny. We have a full nightly compose now, like primary, where previously we basically had a “newRepo” process we now have a full installer stack with network and iso installs produced. This in itself is a massive improvement! To this we add docker to aarch64 and Power64, cloud to aarch64 (and more automation of cloud to Power64), and initial “tech preview” disk images to aarch64 for Single Board Computers (I’m sorry I really did want to nail down this feature but even I have to sleep!) like the Pine64.

Some of the “smaller” tick box improvements to non primary arches is that we’re now fully 100% ansible run for the aarch64 and Power64 infrastructure, and only have a few minor bits to work out for s390x. This means from the back end everything is 100% like primary and orchestrated as such, the hubs all looks the same and the experience is consistent. Changes are deployed simultaneously and hence consistently. YAY automation! We’ve also simplified the way the secondary content gets to the mirrors so it’s more consistent and faster!

On the not x86 architecture front we’ve got a whole bunch more exciting features planned, and improvements to make, for the Fedora 25 cycle and beyond, some of which will begin happening very soon. Watch this space, the second half of the year looks to me to be just as frantic as the first!!

Fedora 24 Alpha for aarch64 and POWER

So Fedora 24 Alpha is out for aarch64 and POWER. Keen followers will note that we were a couple of days behind the primary architecture’s Alpha release, which hasn’t been the case for the last few Fedora cycles where we’ve generally released on the same day.

The primary reason for the delay was the Pungi Refactor. While the pungi 4 change has been massive for primary architectures for the secondary architectures it’s the single biggest change to our release process EVER! Basically we’ve thrown the lot out and started again. When I started in release engineering over 18 months ago the number one goal that was set for me can be summarised as “Be more like primary. Make the whole secondary architecture as close to primary as possible!” and we’ve been continuously moving, albeit not as fast as I would have liked, in that general direction. With the arrival of pungi 4 for Fedora 24 we’re almost at that end goal in terms of the current way we do secondary architectures.

With Fedora 24 we’re also adding a lot more release engineering focused features and functionality to the secondary architectures. We have now have full nightly composes on rawhide and branched whereas previously we’d just produce a “Everything” repo. This allows ongoing continual testing on things so it’s easier to know when things regress. On PowerPC we’ve produced qcow2 cloud images to some degree since Fedora 22 but it was a bit of a manual process. These are now fully integrated into the pungi/koji process and, like on primary produced nightly, similarly they’ll be coming to aarch64 very shortly too. In Fedora 24 we’ve added Docker base images, they’re produced nighly on branched and rawhide for PowerPC now, and will be nightly for aarch64 at the same time the qcow2 cloud images arrive. Finally aarch64 will also soon have disk images like ARMv7 on primary to enable us to easily support the new shiny aarch64 Single Board Computers (SBCs) that are _FINALLY_ becoming available for the architecture, for Fedora 24 it’ll be a bit of a hack, but with Fedora 25 both ARMv7 and aarch64 will be able to move to koji based live-media-creator image build process but I’ll outline more of that in another post.

So the pungi refactor has been big for the secondary architectures. It’s required big changes in our infrastructure which is now mostly complete, there’s a few infrastructure cleanups and final changes that are in process, these will be done in the next few weeks in the lead up to Beta. We have a single host left to migrate to ansible (YAY!!) and some final moving around of resources. We’ll be changing the way we sync content out to mirrors too which will close out one of the final deltas of the rel-eng secondary process. Overall the last few weeks have been challenging getting all the bits in place, but by the time we hit Beta it’ll all be complete! The new processes lay the foundations for the secondary architectures to add functionality quicker than ever before, and by being almost identical to primary the “onboarding” of new people to use that process, or end users be able to consume the output of the rel-eng process is easier than ever before and that makes me happy! 🙂

Flock Rochester

I’m not going to do a day by day outline of what I did at flock, if I did it would basically be “blah blah blah I talked a lot to a lot of people about a lot of tech topics” and anyone that’s ever met me would have guessed that! It was, as in the past, a great conference. A big shout out to the organisers for an excellent event with two excellent evening events! So I’m going to give a brief summary to my talks and link to slides and video recordings.

My first talk was an overview of the state of aarch64 and POWER as secondary architectures. The slides aren’t particularly interesting as they’re just words for discussion points. The video has all the interesting bits. A related talk was Dennis’s Standardising ARMv7 booting with a memorial quote by Jon Masters 😉

My second talk was about using Fedora as a base for IoT. Slides are here but the talk was quite a bit different to the slides and is more interesting so I suggest watching the video.

I also actively participated in Dennis’s Fedora Release Engineering going forward because well obviously I’m part of it 😉 and it was interesting for where we’re going, and even where we’ve come from in the last year or so 🙂

Finally I loved the Keynote Be an inspiration, not an impostor by Major Hayden. He’s published a follow up blog post with a FAQ too.

The least memorable bit was the terrible Amtrak ride back to New York City. On the plus side it makes the worst of the British National Rail service seem amazingly on time! NEVER AGAIN!