Supporting Fedora 24 on the Raspberry Pi 2

So I get asked the question dozens of times a day so I thought I’d outline the answer to the question “When is Fedora going to support the Raspberry Pi 2?” and “The kernel support is upstream in the 4.5 kernel, why isn’t it enabled in Fedora 24?”

Ultimately support in the kernel is great, it’s obviously a core blocker, and the first steps to supporting a new piece of hardware in Fedora. The thing is that when people say kernel support is easy they are partially right but it’s only a very small part of what’s needed to support a complex device such has an ARMv7 Single Board Computer for the average user, especially one as popular as the Raspberry Pi! To make the device work with Fedora we could just enable the kernel bits but it doesn’t make for a good user experience OOTB (Out Of The Box).

With a lot of ARMv7 devices these days a new device comes out and it just works with Fedora. It’s awesome when I read a report, or someone tells me “I tried device X with Fedora and it just worked”. That’s because of a lot of work Fedora, and others, have done to ensure upstream boot loaders and boot process just works with new devices. It’s taken a long time to get us to this point. The Raspberry Pi is sadly not like all the other ARMv7 or aarch64 devices. It doesn’t have a standard boot process, doesn’t use u-boot or uEFI, needs vFAT partitions, firmware, text config files and other things that none of our other supported devices need. All of these differences need to be taken into account.

So in terms of the support being upstream in 4.5 the answer to that is it’s “mostly” upstream, there’s still a bunch of patches we’d need to pull in to ensure a nice OOTB experience. This isn’t a blocker in my opinion, it’s something that’s relatively straight foward with most of the bits already headed upstream into 4.6 so it’s a short term issue.

For the rest of the bits what do I consider a nice out of the box experience? This:

  • A single image to support the Raspberry Pi and all out devices (more on that below)
  • Graphics and USB support from boot to login
  • Most basic peripherals working, at a minimum USB (inc keyboard/mouse), HDMI, wired ethernet, a decent selection of wireless USB dongles, storage (MMC and USB) and preferably sound (analog, digital HDMI, mic)
  • A means of easily creating a bootable SD card from at least the Fedora command line, and probably Windows or MacOSX
  • Good documentation, FAQ etc

Single Image:
We currently produce a lot of different images for ARMv7 like Workstation, Server, Minimal and various desktops. If we had to double the amount of images we make to add vFAT that would double the work needed by QA, rel-eng and also lead to confusion by end users as to which image is needed. I have no intention of doubling anyone’s work, or adding confusion for end users, there needs to be a proper engineering solution to this problem!

Serial Console
A number of people have said to me “just enable it and tell them to use the serial console” but having been working on ARMv7 for over 6 years now I know from experience that this leads to vast amounts of confusion by end users as to why “it’s not working” and it leads to a lot of time “providing support” to end users. With a device such as the Raspberry Pi this will become an order of magnitude worse which won’t provide users a good Fedora experience, and likely drive the people who are trying to support the device a even more nuts than normal!

Basic peripheral support
I feel that network, storage, display, input and sound, although I’m still on the fence about sound, are the minimum viable supported peripherals needed for a good OOTB experience for users.

I know that initially we won’t support HATs very well, you can already copy around device tree overlays in the /sys filesystem for basic support, but upstream still hasn’t finalised what a good experience will look like in this regard. I think we can live with this. This won’t stop the use of I2C or SPI devices connected to the 40 pin header, most of these should work just fine.

Image to card creation
Closely related to the single image and basic peripheral support I think the ability to easily create an image to use is important. I’m not sure we’re going to be able to easily solve the Windows/OSX problem, although to be honest I’ve not looked at what’s out there and we might be able to extend LiveUSBCreator here.

Raspberry Pi 3
So when will we support this? Well the kernel support hasn’t been released yet, at least that I could find at the time of writing this. That being said I don’t think it’ll be a particularly evasive or large patch set, the hardware around the Cortex-A53 is the same, so it’ll be just some glue and a pinctrl driver to make all the bits work together, likely not too dissimilar to other recent SoCs that have gained ARMv8 support. This won’t land in 4.6 as the major changes are already queued to land in that, so likely 4.7 will be the earliest upstream kernel. The wireless on the other hand could be more interesting.

So basically the work needed for support of the Raspberry Pi 3, with a proper 64 bit OS 😉 , shouldn’t be too hard once the kernel bits are upstream.

Firmwares
The other issue we had was the legal ability to redistribute the “GPU boot loader firmware” thankfully that problem was resolved about a year ago, although it did take us over 2 years to do so!

The other firmware issue which will cause problems with Raspberry Pi 3 support is that the Broadcom wireless is notoriously terrible, as any Linux running Mac user will contest to, and their wireless firmware isn’t re-distributable in the standard process of being included in the upstream linux-firmware. With luck this is something that the Raspberry Pi Foundation could assist Broadcom in improving!

Summary
So we’re well on the way to adding support. Assistance in the issues above would be very welcome, whether for supporting a single image or adding support for writing images for the Pi, to help us get the bits done sooner rather than later. Feel free to reach out to me on IRC (pbrobinson on FreeNode) or some other means.

My ARM grab bag device list

They say the first step of coming to terms with addiction is admitting you have a problem… I have a problem with collecting ARM devices… there I said it! How big is this problem you ask? How about I list them out and let you decide!

I’ll break the list down into categories as I believe it’s big enough to do so :-/

The aarch64 set of devices currently stands at:

  • 2x Applied Mustang (different x-gene SoC revs)
  • AMD Seattle
  • 96boards HiKey (hi6220)

The ARMv7 boards list is currently:

  • Compulabs Trimslice (tegra-2)
  • Toshiba AC100 (tegra-2)
  • nVidia Jetson TK1 (tegra-124)
  • Acer Chromebook (tegra-124)
  • BeagleBoard xM (omap3)
  • Nokia n900 (omap3)
  • Nokia n950 prototype (omap3)
  • BeagleBone (am33xx)
  • BeagleBone Black (am33xx) x3
  • BeagleBone Green (am33xx)
  • PandaBoard ES Prototype (omap4)
  • PandaBoard ES B2 (omap4)
  • CubieBoard (allwinner-a10)
  • CubieTruck (allwinner-a20)
  • Banana Pi (allwinner-a20)
  • C.H.I.P. Alpha x2 (allwinner-r8)
  • Snowball (u8500)
  • Compulabs Utilite (imx6q)
  • WandBoard Quad revb (imx6q)
  • novena board (imx6q)
  • RIoTboard (imx6s)
  • UDOO Neo (imx6sx)
  • Origen (exynos-4)
  • OLPC XO 1.75 – a number of variants (mmp2) xNumerous
  • OLPC XO-4 including XO-Touch (mmp3) xNumerous
  • Linksys WRT1900AC (armada-xp)
  • Mirabox (armada-370)
  • ifc6410 (qcom)
  • Parallella Board (zynq7000)
  • Raspberry Pi 2 x3

The Cortex-M series for IoT sensors is currently:

  • TI SensorTag 2015
  • ARM mBed IoT starter kit
  • BeeWi SmartClim

Other random related bits:

  • BeagleBone Breadboard Prototyping Cape x2
  • BeagleBone CryptoCape
  • Original 256Mb Raspberry Pi model B
  • Grove starter kit for BeagleBone Green
  • Explorer HAT
  • PiGlo HAT
  • TI CC2531 802.15.4 USB dongle x3
  • numerous random sensors

So the list above is the devices that I use for hacking on. I count 41 without listing out the dozen or so ARM based XOs I have (various prototypes and models). I also don’t have in that list phones, tablets and two drones as I don’t really hack on those as it’s not like with the list above I don’t already have enough toys! So do I have a problem?

Flock Rochester

I’m not going to do a day by day outline of what I did at flock, if I did it would basically be “blah blah blah I talked a lot to a lot of people about a lot of tech topics” and anyone that’s ever met me would have guessed that! It was, as in the past, a great conference. A big shout out to the organisers for an excellent event with two excellent evening events! So I’m going to give a brief summary to my talks and link to slides and video recordings.

My first talk was an overview of the state of aarch64 and POWER as secondary architectures. The slides aren’t particularly interesting as they’re just words for discussion points. The video has all the interesting bits. A related talk was Dennis’s Standardising ARMv7 booting with a memorial quote by Jon Masters 😉

My second talk was about using Fedora as a base for IoT. Slides are here but the talk was quite a bit different to the slides and is more interesting so I suggest watching the video.

I also actively participated in Dennis’s Fedora Release Engineering going forward because well obviously I’m part of it 😉 and it was interesting for where we’re going, and even where we’ve come from in the last year or so 🙂

Finally I loved the Keynote Be an inspiration, not an impostor by Major Hayden. He’s published a follow up blog post with a FAQ too.

The least memorable bit was the terrible Amtrak ride back to New York City. On the plus side it makes the worst of the British National Rail service seem amazingly on time! NEVER AGAIN!

Fedora 22 on ARM and aarch64

So with Fedora 22 well and truly out for both ARMv7 and aarch64 lets have a look at the release in general and also at the 4.0 kernel it ships.

Firstly I’d like to shout out to the AMAZING job done by the web team on the new sites for Spins, ARM and Labs. They really do look awesome!

It’s all to easy to get bogged down in the actual technical components that make up the actual release and not forget that there’s work that goes on all over the place that contributes to making the release easy to use even before you begin the download.

Anyway! On to ARMv7. We shipped GA with the 4.0.4 kernel and u-boot 2015.01. This combination gives us improved support for numerous devices primarily through new DT support and improvements to drivers. The 4.0 kernel from an ARM HW support PoV really was a minor revision. With the fully packaged and updated fedora-arm-installer it’s even easier to get a device up and running.

We were hoping to get enough time to rebase this to 2015.04 but we just plain ran out of time, although we do have a plan to be able to update the u-boot when writing an image to card/stick without the need to regenerate the images. More details on the improvements we’re planning for fedora-arm-installer for another post!

From an aarch64 point of view the big change, although one an average user won’t notice, was we went from carrying a rather large (around 65K lines) enablement patch for the kernel to a small collection of 4 bug/problem specific patches! This is a massive change where in the F-21 cycle we had a gigantic architecture enablement patch! This makes it much more straight forward and less stressful for both myself as the architecture maintainer and the core Fedora kernel developers. In terms of the rest of aarch64 we still support VExpress, APM Storm platform primarily with the Mustang boards, and the AMD Seattle platform.

Overall the Fedora 22 release on the ARM platforms is a really nice release, there’s been some good changes there to enable easier and quicker updates in the future and easier means of adding decent support to new devices mid cycle. AArch64 is maturing and hopefully we’ll start to see some more platforms land and the architecture start to depart the niche status!

3.19 Fedora ARM kernel status

I’ve been a bit lazy on the ARM kernel status updates. There wasn’t one at all for 3.18 but the fact was, that while there was lots of under the hood improvements for ARM/aarch64, the new device support or improvements from a user’s point of view was positively boring so I never bothered!

That said the 3.19 kernel is now on it’s way to the stable Fedora releases and there’s some bit of interest there 🙂

Beginning with aarch64 there’s been a raft of code support landing upstream for the core platforms we support (VExpress, APM Storm, and AMD Seattle) which means the enablement patch set has shrunk massively. The core missing bit from this is primarily the ACPI patches for the server standards. There’s also been a lot of stability improvements for various device drivers particularly on the APM Storm SoC (which massively helps the high network and IO traffic we generate when doing composes in release engineering!). Other improvements include support for seccomp. The upstream support for aarch64 is really starting to settle down nicely which is good because there’s devices finally starting to get to the point where they’ll be more widely available and affordable 🙂

On to ARMv7 changes. In terms of new supported SoCs the support for AllWinner A-23 (aka sun8i) is the most interesting in terms of new devices. There was also a lot of general SoC improvements and cleanups. The largest here is probably Rockchips, QCom and ZYNQ with notable mentions to Tegra, OMAP and i.MX6 too. In terms of new devices we now solely support DeviceTree devices and the built .dtb files we ship that are possible to support with the kernel jumped from 250 to 265 devices. Of course it doesn’t mean we’re testing all of those devices but we’re testing devices across all main SoC groups to ensure at least the core support works. Of course feedback for what works and doesn’t is always welcome. In this cycle there was also significant driver work with special mention to Hans and his significant movement on the Allwinner devices.

I’ll do a longer post for 4.0 and the new u-boot we’ll be supporting in Fedora 22 soon.

Fedora aarch64, device tree and u-boot support

A question that I’ve had a few times in the last couple of weeks is whether Fedora supports Cute Embedded Nonsense Hacks, also known as u-boot and device tree, on aarch64 (ARM64) platforms?

The answer is YES!, of course, why wouldn’t we?

I know people are well aware of Red Hat’s involvement in the Server Base System Architecture (SBSA) which mandates the use of UEFI 2.4 and ACPI 5.1 bindings and that the Red Hat Partner Early Access Program uses that standard to enable easy booting and support of server platforms running on aarch64 platforms but the fact is that is not Fedora.

Fedora plans to support the SBSA to enable easy use of Fedora on aarch64 server platforms. But we also plan to support the current standard u-boot with device tree boot options. The fact of the matter is that a lot of non server based aarch64 platforms will continue to use these options and so we’ll continue to actively support them. Just like Fedora support Xen when the Fedora derived enterprise product does not. Basically it’s not hard for us to continue these options and with the improved generic distro support in u-boot, which we’ve actively participated in and driven, testing of Cute Embedded Nonsense Hacks on aarch64 should be easy and straight forward.

Of course the support of both SBSA based uEFI/ACPI or u-boot/DTB isn’t perfect on aarch64 yet so if you’ve got access to aarch64 systems on either platforms I would love testing and bug reports. If you’re a vendor that plans on using u-boot/DTB on aarch64 I would ask to ensure that you support the generic distro options because it’ll enable out of the box booting of at least Fedora, Debian and openSUSE to seamlessly just work on your devices.

3.17 Fedora ARM kernel status

With 3.17 due momentarily and Fedora 21 been delayed a little we’ll now be bumping the kernel that ships in F-21 GA. So lets have an overview of what improvements and changes are going to be there for ARM.

Overall 3.17 has been relatively boring in terms of shiny new hardware support for ARMv7. We’ve added support for a bunch of new devices through the addition of appropriate device tree bits. Some of the highlights there include a number of AllWinner devices such as the Banana Pi, a number of new FreeScale i.MX6 devices, some of RockChips devices, and the ZYNQ Parallella.

On the aarch64 side there’s been general improvements all over the place. Over all we don’t have any new platforms but there’s improvements to the three we do support (VExprees, APM X-Gene, AMD Seattle) but the VExpress Juno device should work and initial support for the ACPI 5.1 standard and improved uEFI both of which are part of ARM SBSA Server standard.

Along side 3.17, or at least very shortly there after, u-boot 2014.10 or at least a release candidate should land in F-21 as well. This release adds support for a lot of new devices, primarily AllWinner A-10/13/20 categories, as well as the Tegra Jetson K1, a few i.MX6 devices such as the RIoT Board and the newly upstream distro standards for booting. This makes it much easier for us to “just boot” Fedora ARM with a lot more devices making the experience of getting started a lot easier for most people with supported devices.

The combination of u-boot 2014.10 and the 3.17 kernel will be what we head towards Fedora 21 GA with and things are starting to come together nicely.

Flock 2014 revisited

So having almost recovered from the lack of sleep that is one of the guarantees of conferences in general, but definitely a Fedora one, I thought I would reflect on a few bits. I’m not going to cover all the talks as a lot of people have done that and all the talks are on the Flock 2014 YouTube channel for your viewing pleasure.

As others have mentioned the venue was great, easy to get to and from for transport and the hotel. Huge kudos to the organisers of the event! An event such as this takes a lot of time and energy, and with the dust barely settled the Flock 2015 Bid process is already under way so if you’re interested in hosting 2015 in North America…

My State of ARM and aarch64 in Fedora went well, I enjoyed it and the room was packed with as many people standing as there were sitting 🙂 There was lots of good questions and interest, both in the talk, and in the hallway in general.

I went to numerous excellent talks, too many to count or remember and I’m looking forward to catching up on a number of talks I missed due to schedule conflicts, via the videos, when I get some spare time. Of course the other major part of the conference is the hall way track. There I had too many conversations to recall and caught up with numerous old friends, met a number of people I’d been dealing with online and had never met in person, and of course met a whole bunch of new friends too! It’s amazing how much can be achieved when talking to someone on the walk between conference rooms!

One of the other major things I enjoyed about Flock this year was the overall positivity of everything about the conference, whether it be people’s general attitude, the presentation titles and the presentations themselves or people in the Litre Pub 😉 . And of course being one of our values I have to mention that catching up with so many good friends is always the sugar on top of the cake!

Flock 2014 Prague

So I’m at Flock in Prague. So far I’ve been to a bunch of interesting talks about Release Engineering, Secondary Architectures, Fedora Workstation, Docker and Infrastructure.

Of course then there’s the hallway track of which I always actively participate and it’s been always fabulous to meet a bunch of people in real life that I’ve been dealing with online on a regular basis, in some cases for years!

I’ll be around for the entire conference and if you’re interested in chatting about secondary architectures (not just ARM), Sugar, Cloud or just about anything else or just to say hi please come and find me!

Fedora 21 and ARM aarch64 status for alpha

With the Fedora 21 Alpha freeze looming in the rear view mirror, although the object wasn’t as close as it would appear, I thought it was high time that I gave a brief overview of the state of ARM aarch64 in Fedora. Some might assume the silence means not a lot has been happening but this is extremely far from the truth!

So lets start with a few statistics:

  • Builds the same with mainline: 14973 odd (yes, that’s nearly 15,000 Fedora source packages built on aarch64!)
  • Older builds: 217 (we have a built but it’s not the same NVR as mainline)
  • Missing builds: 352

So that’s looking pretty damn good! The main components that we’re missing that make up the missing builds comprise of two main groups.

The first is builds that are FTBFS on mainline and that’s basically, if it can’t be build on F-21 on mainline we have no chance of rebuilding the f21 tag.

The second reason is platforms that aren’t yet supported on the aarch64 architecture. The core group of these come down to mono (and anything that depends on mono), golang, v8 (mongodb/nodejs etc), pypy make up the majority of that list. We’re working with upstreams to hopefully fill those gaps before long.

There’s a few other minor stragglers that don’t really fit into either of the above. erlang just needs to be bootstrapped plus a few others like thunderbird, libreoffice and hadoop that need some attention which we’ll get to soon.

So the aarch64 userspace, while still not 100% there, is looking EXTREMELY good and there’s a number of people that are now putting it through it’s paces on a daily basis which in turn allows us to improve it as we go.

Hardware
As I indicated in my 3.16 kernel status we now have support for a number of hardware options to run the userspace. Some of them are emulated (qemu, ARM foundation model) and some actual physical (APM Mustang, AMD Seattle) if you’re lucky enough to have access. The support for these devices is improving all the time and support for kernel features are coming along pretty thick and fast.

So in summary the Fedora aarch64 is in very good shape for the Fedora 21 Alpha and will only improve as we apply polish along side x86 and ARMv7 in the lead up to Fedora 21 GA.