Semi irregular Fedora ARM kernel status reports

I thought I’d start doing semi irregular ARM kernel status reports. I’ll do them as often as I think they might be useful and those who know me know I travel a lot and randomly and that ARM isn’t my $dayjob.

Few are bored or stupid enough to follow the 20 or so ARM kernel trees or have the regular insight as to what’s happening, what’s landed, what new devices might work and what bugs come and go that I do so I thought I’d try and dispense some of the more interesting bits of that information and how it relates to Fedora ARM to a wider audience by both the fedora-arm mailing list and my blog so those people that don’t sit on the IRC channel and those that like to lurk might have a better idea what’s going on.

The general format I plan to use is basically:

  • What’s new including SoCs, boards and new devices
  • Interesting bugs fixed
  • Outstanding bugs and issues
  • Random other insights

I don’t intend them to be long but rather short, sweet and to the point. They’ll probably come out when new major releases hit either rawhide or stable or something of particular interest lands. Feedback on both the format most other things is welcome as are questions and status of devices people might have had success or less so with.

I plan to have the first for 3.14 out later today and one for 3.15 RSN.

Simple WOL with RHEL and Fedora

I have a number of systems I use at home, they run a combination of RHEL and Fedora and are a variety of architectures (well x86 and ARM). Travelling a lot for $dayjob I don’t always want them on needlessly chewing power but on the flip side if I’m away for two weeks and I’d like to remotely use one and it’s off it’s a little tough to power it up if you’re 12 hours travel away!

So the obvious solution to this problem to get the best of both worlds is to use Wake On LAN (WOL) for supported hardware which means I can leave devices off or suspended and then just bring them up to full power when I actually want to use them from where ever I happen to be at the time, whether it be on the couch across the room or the other side of the world.

There’s a few places where you need to configure the device you wish to wake up. If you want to wake from power off you need to configure the BIOS/Firmware to have WOL enabled, this differs from device to device so this bit is an exercise for the reader. The quickest way to determine if your hardware supports WOL and its enabled in the BIOS/Firmware is to check to see if when the device is turned off and cabled up to a switch is if there’s a link light on the NIC is still lit.

Once you’ve got the HW and BIOS/Firmware sorted the next bit is pretty straight forward. First enable WOL on boot for the NIC in /etc/sysconfig/network-scripts/ifcfg- by adding the following line to the end of the file:

ETHTOOL_OPTIONS="wol g"

Then run

# ethtool -s  wol g

to enable it now. Finally record the ether address from the output from the “ip link” command so you know what MAC address you need to use to wake it up.

On your gateway make sure you have the appropriate tool to initiate the remote wakeup. On Fedora and RHEL this is ether-wake and it’s in the net-tools package. It’s then as simple as running

# ether-wake -i local-interface mac-address

Kernel taint state descriptions

It seems I always have to search for the kernel taint descriptions. So for reference they’re located in kernel/panic.c of the kernel source.

The descriptions are as follows:

/**
 *	print_tainted - return a string to represent the kernel taint state.
 *
 *  'P' - Proprietary module has been loaded.
 *  'F' - Module has been forcibly loaded.
 *  'S' - SMP with CPUs not designed for SMP.
 *  'R' - User forced a module unload.
 *  'M' - System experienced a machine check exception.
 *  'B' - System has hit bad_page.
 *  'U' - Userspace-defined naughtiness.
 *  'D' - Kernel has oopsed before
 *  'A' - ACPI table overridden.
 *  'W' - Taint on warning.
 *  'C' - modules from drivers/staging are loaded.
 *  'I' - Working around severe firmware bug.
 *  'O' - Out-of-tree module has been loaded.
 *
 *	The string is overwritten by the next call to print_tainted().
 */

I run into W regularly but as I don’t tend to use binary drivers, third party firmwares, force module loads or use custom ACPI tables (I wonder if a T for custom/out of tree Device Tree is needed?) I don’t tend to see many of the others. I do use staging drivers though. The other one I see quite regularly that isn’t documented in the above comment is G which means “GPL module loaded” ie basically the opposite to P “Proprietary”.

ARM hardware support on Fedora 20

A number of people complain about our hardware support on ARM. I’m not sure people really understand the issues we have bringing the hardware that we try to support to ARM and the constraints we work in. I’m going to try and outline the process we’ve been through over the last couple of releases and the reasons why we stick to the Fedora processes as much as possible. The other thing that I think people quickly forget is that the hardware support is a very small percentage of the overall work that has gone into bringing ARMv7 up to the excellent state it is in. Unfortunately like so many other projects the last 10% is always the hardest.

Support for hardware primarily revolves around three core components. They are the boot-loader (bios in x86 parlance), the kernel and any user space components needed which generally focuses on X drivers, but is no way the only thing.

Going back to the near memory Fedora 18 shipped with the 3.6.10 kernel. The 3.6 kernel didn’t ship with what’s known as the “Unified Multi Platform ARM kernel” which now allows us to ship a single kernel to boot multiple different SOCs (System on a Chip) by numerous manufacturers. The initial multi platform kernel support landed in 3.7 and initially allowed us to merge the Versatile Express and Calxeda Highbank kernels together. It was only with the 3.10 kernel we actually started to ship a single unified kernel that supported all the SoCs we support. We currently enable around a dozen different SoCs in the mainline kernel and actually support HW running on around half of those.

Supporting the multi platform kernel hasn’t been an easy ride. As you would expect of Fedora we were the first to adopt it and a lot of other distros still haven’t. Debian is in the process of moving over to it for Jessie, I thought Open SuSE was using it but it appears they’re not and Ubuntu certainly isn’t. I’m not sure about Arch or any of the others. It’s been an interesting ride and while a single kernel is something we’ve wanted for some time the lack of testing by a wider audience has certainly made it more painful for us than it could of been, it would be particularly helpful if Linaro, the main developers, actually used multi platform kernels for all their builds.

An example of problems we’ve had with the multi platform kernel is the support for the OMAP4 based PandaBoard, since before Fedora 17 it’s been consistently our best supported device but with the 3.11 kernel and the move of OMAP4 devices to DeviceTree it doesn’t boot and there’s been a number of us try to get to the bottom of the problem to no avail. Similarly going back to 3.6/3.7 the i.MX series of SOCs caused us massive problems to the point we just stopped supporting them but with the i.MX6 it’s very quickly becoming out best supported class of devices with a collection of almost a dozen very nice and relatively cheap devices (starting at USD$45) like the WandBoard dev boards, Compulabs Utilite and Cubox-i. Due to the high level of churn in the upstream kernel it at times makes it really hard and we can, and do, spend a lot of time chasing down issues with a single SoC or even device.

So what hardware actually works in Fedora 20? In theory we support 100s of devices but in practice the testing in limited to a selection of devices that we actually have to be able to easily test. So what actually works:

  • Versatile Express via QEMU emulation using libvirt.
  • Calxeda Highbank and Midway servers
  • Compulabs Trimslice (tegra).
  • The three WandBoards (i.MX6) in particular the Quad. 3.12 will improve the support too.
  • The BeagleBones (am33xx). In 3.11 there’s issues with usb, 3.12 is looking better.
  • BeagleBoard xM (omap3). For network you need to use the usb OTG port.
  • Mirabox and other Marvell (mvebu) devices with appended DTB (due to old uboot shipped with device).

Added to the above we ship around 100 device tree files, in 3.13 that rises to over a 120. I’ve had reports of a number of people successfully using some of these devices without issues. So what doesn’t work so well out of the box?:

  • PandaBoards (omap4). As mentioned above these broke with 3.11, I’m hoping to get them working again soon but my testing to date hasn’t been fruitful though.
  • Compulabs Utilite (i.MX6), well not with the 3.11 shipping kernel, we have initial support in 3.12 but there’s also issues with their initial uboot release.
  • AllWinner devices, but there’ll be a high quality remix that supports a lot of these devices, we’re hoping to improve the mainline support a lot in the F-21 development cycle.
  • Any device that doesn’t support multi platform kernels
  • Any device which isn’t supported in the upstream kernel

I’ve had reports of a number of other devices that “just work” or mostly work and with time this will improve. One thing that we’ve tried very hard not to do is pull 100s of patches in. Fedora likes to be as close to upstream as possible with their kernels and having lots of patches or kernel variants just doesn’t work as we just don’t have the resources to deal with them. We do on occasion pull in patches to fix or improve things but we try to keep as close to mainline as possible so if you want a device supported the first thing you’ll be asked is “What’s it’s supported state on the upstream kernel?”.

So what’s on the ToDo list? In short.. LOTS! With 3.14 we should finally get usable AllWinner devices, 3.13 in theory should have a number of RockChip devices supported and both of those SoCs bring a huge amount of cheap devices along that we have the potential to support. As it makes sense to enable new support we will. I’m also planning on improving and testing the support we currently have. I like the BeagleBone expansion capes so I plan on testing and playing with those and in general improving the support of devices that I have (a post on those coming soon).

All in all while we’ve got a long way to go I believe our hardware support has improved fairly well over the last couple of years where in Fedora 17 we officially supported two devices.