Flock Rochester

I’m not going to do a day by day outline of what I did at flock, if I did it would basically be “blah blah blah I talked a lot to a lot of people about a lot of tech topics” and anyone that’s ever met me would have guessed that! It was, as in the past, a great conference. A big shout out to the organisers for an excellent event with two excellent evening events! So I’m going to give a brief summary to my talks and link to slides and video recordings.

My first talk was an overview of the state of aarch64 and POWER as secondary architectures. The slides aren’t particularly interesting as they’re just words for discussion points. The video has all the interesting bits. A related talk was Dennis’s Standardising ARMv7 booting with a memorial quote by Jon Masters 😉

My second talk was about using Fedora as a base for IoT. Slides are here but the talk was quite a bit different to the slides and is more interesting so I suggest watching the video.

I also actively participated in Dennis’s Fedora Release Engineering going forward because well obviously I’m part of it 😉 and it was interesting for where we’re going, and even where we’ve come from in the last year or so 🙂

Finally I loved the Keynote Be an inspiration, not an impostor by Major Hayden. He’s published a follow up blog post with a FAQ too.

The least memorable bit was the terrible Amtrak ride back to New York City. On the plus side it makes the worst of the British National Rail service seem amazingly on time! NEVER AGAIN!

Fedora 23 on the Thinkpad X1 Carbon gen 3

So my corporate laptop for the last three years has been a Thinkpad x220, it’s getting a bit long in the teeth, when I got it the x230 was already out but the corporate standard hadn’t rebased. This time I happened to get a new device just after the new corporate refresh so I have the shiny X1 Carbon gen 3 with real track pad buttons.

Of course I grabbed the latest rawhide nightly boot.iso to install the latest shiny! How did I get on? Initially it was disaster, the kernel crashed before I even got to anaconda (will spend some time to recreate and log that soon). Oops! So I grabbed a couple of the earlier installers and with a bit of trial and error before long I was at the initial anaconda screen. From there the process was relatively boring! Up and running with Fedora Workstation installed, disk encrypted I started to play with it to see which bits were good, bad or ugly.

So first up with the good. Over all the vast majority of the hardware just worked out of the box. The GPU, USB3, wired ethernet, wireless ethernet (iwl 7265 AC), trackpad/thumb pointer, bluetooth mouse, camera, onboard speakers all work just fine, I even managed to enrol my finger in the finger print reader without issue. I must say I’m loving the 1920*1080 screen over the old 1366*768 of the x220 and I’m getting use to the chiclet keyboard layout. Overall F-23/rawhide is solid right out of the gate with pretty much everything working as you’d expect on a stable release. 🙂

It’s not all rosy though and some of the bad is that I’ve not managed to get the firmware upgraded to support 4K displays @ 60hz, thanks to Major and Sandro for the heads up on this, but it never seems to find the usb stick as a bootable image. Not a major issue in the short term as the OneLink Pro Dock I ordered is AWOL but I’ll want it soon as I’ve got a standing desk with a Dell 24 inch 4K monitor. The x220 and 4K monitor had never really worked overly well, the big issue was it only ran at 30hz refresh but I knew that when I got the monitor because I knew I was due for a refresh soon, it also had random glitches. I also need to workout how to adjust the acceleration of the thumb pointer thingy with libinput. Adjusting either the trackpad or mouse options in control panel doesn’t seem to have any effect, I never really used the trackpad on the x220 but I admit I’m getting use to the two finger scrolling for reading long pages. This should all be relatively easy to solve with a bit of poking!

The ugly seems to be the stability of the iwl 7265 wireless driver/firmware. It generally works but regularly shits itself. Some times minor by dropping SSH connections, sometimes majorly resulting in a need to unload/reload the modules or even to reboot! URGH! I’ve heard people complain about recent Intel Wireless stability but the “Advanced-N 6205 [Taylor Peak] (rev 34)” that was in the x220 was always solid. The solid lockups seems to be when pushing a reasonable amount of data via rsync/ssh. I do have a WRT1900AC router and I’ve connected to the 5 ghz 11ac so I’m wondering whether this combo is part of the issues. There is a newer firmware that hasn’t made it upstream yet. I need to do some more playing here testing the 11n 2.4ghz network as well as testing the newer firmware and possibly some patches that are on the wireless mailing list which I’m hoping will actually just land in 4.2 before long 🙂

There’s a few other things I need to play with some more. I’ve not tried external HDMI video/audio, the external display port, the DP to VGA converter (nice one was included in the box though) or the headphone/mic socket. None of the functionality of the dock has been tested yet simply because it’s yet to show up! It reports around 7.5 hours of battery life but I want to look at what the state of power consumption on these devices after mjg59’s post about it to see if I can’t get that well into the double digit hours.

Overall I’m pretty impressed with out of box experience of F-23/rawhide on the Thinkpad X1 Carbon gen 3 🙂

Fedora 22 on ARM and aarch64

So with Fedora 22 well and truly out for both ARMv7 and aarch64 lets have a look at the release in general and also at the 4.0 kernel it ships.

Firstly I’d like to shout out to the AMAZING job done by the web team on the new sites for Spins, ARM and Labs. They really do look awesome!

It’s all to easy to get bogged down in the actual technical components that make up the actual release and not forget that there’s work that goes on all over the place that contributes to making the release easy to use even before you begin the download.

Anyway! On to ARMv7. We shipped GA with the 4.0.4 kernel and u-boot 2015.01. This combination gives us improved support for numerous devices primarily through new DT support and improvements to drivers. The 4.0 kernel from an ARM HW support PoV really was a minor revision. With the fully packaged and updated fedora-arm-installer it’s even easier to get a device up and running.

We were hoping to get enough time to rebase this to 2015.04 but we just plain ran out of time, although we do have a plan to be able to update the u-boot when writing an image to card/stick without the need to regenerate the images. More details on the improvements we’re planning for fedora-arm-installer for another post!

From an aarch64 point of view the big change, although one an average user won’t notice, was we went from carrying a rather large (around 65K lines) enablement patch for the kernel to a small collection of 4 bug/problem specific patches! This is a massive change where in the F-21 cycle we had a gigantic architecture enablement patch! This makes it much more straight forward and less stressful for both myself as the architecture maintainer and the core Fedora kernel developers. In terms of the rest of aarch64 we still support VExpress, APM Storm platform primarily with the Mustang boards, and the AMD Seattle platform.

Overall the Fedora 22 release on the ARM platforms is a really nice release, there’s been some good changes there to enable easier and quicker updates in the future and easier means of adding decent support to new devices mid cycle. AArch64 is maturing and hopefully we’ll start to see some more platforms land and the architecture start to depart the niche status!

3.14 Fedora ARM kernel status

With 3.14.1 out Josh and Justin are preparing to land the 3.14 kernel into Fedora 20. So what does it mean in terms of ARM on Fedora. Well it’s an evolution. There’s of course the usual raft of new devices and some new SoCs, and best of all lots of improvements in support for existing devices. Even the return of some old favourites! Generally from the stash of devices Paul, I and others have that get regularly tested things are looking pretty good with 3.14.

New SoCs and device support:

  • Tegra 4/K1 support has been enabled. For 3.14 this doesn’t mean much as there’s not a wide level of devices out there that we ship device tree blobs for but it’s a good preparation for 3.15 as we should have pretty reasonable support for the nice new Tegra K1 dev board!
  • TI devices: The 3.14 finally brings working support for the OMAP5 EVM board, this will improve further in 3.15 but it boots and generally works. It also adds support for the original BeagleBone and the USB is now finally all back for the Beagle-XM devices so they go back into the fully working list! Also the DT bits for the Overo devices has started to land so if is interested in those I’d love feedback from people with those devices.
  • Freescale i.MX: To this lovely growing list we add initial support for the various Cubox-i devices and the hummingboard. Still no HDMI support but here’s hoping for 3.15
  • Xilinx Zynq 7000: The SD controller has finally landed for this which means they should be bootable to login but from there I’m not sure the status of the rest of the support. We ship DT for the zynq-zc702, zynq-zc706 and zynq-zed devices so if you’ve got one of those and have time to test feedback would be welcome.

Interesting bugs fixed:
Nothing particularly exciting comes to mind here. The fix for the Beagle-XM usb hub power up is nice, as was the final config option for the BeagleBone White. There’s obviously a deluge of other ARM driver fixes and improvements (PTP high precision time support for modular CPSW ethernet on the BeagleBone’s anyone?).

Outstanding bugs and issues:
So this is really the list of items I have outstanding for 3.14 so that I can spin some new images. Feedback on any of these and anything I might not be aware of would be very useful.

  • OMAP DRM display. In 3.11 a new display framework landed for the OMAP DRM driver and in 3.12 the old one was dropped. This broke X on devices like the Pandaboard for 3.12+ kernels. I know roughly the problematic area but I just need to get the cycles to debug this. Any help is welcome.
  • Serial over USB. While this isn’t a kernel bug but rather just needs me to hack together some scripts it’s a blocker for easy OOTB support for devices like the BeagleBone Black
  • Tegra DRM display. After a re-write it’s back and modular again in 3.14. Just need to ensure it’s working and ready to go
  • Testing… testing… testing 🙂

That’s mostly all I have on my list. If there’s anything I’m not aware of please do let me know and I’ll endevour to help out where possible. In particular I’m very interested in boot issues for devices that would would be supportable with new ARM images based on 3.14. From the 3.11 kernel that F-20 GA shipped with there’s been a lot of change and improvements, and while non boot enhancements are easy to do with a “yum update” issues with boot aren’t quite so easy to deal with!

Semi irregular Fedora ARM kernel status reports

I thought I’d start doing semi irregular ARM kernel status reports. I’ll do them as often as I think they might be useful and those who know me know I travel a lot and randomly and that ARM isn’t my $dayjob.

Few are bored or stupid enough to follow the 20 or so ARM kernel trees or have the regular insight as to what’s happening, what’s landed, what new devices might work and what bugs come and go that I do so I thought I’d try and dispense some of the more interesting bits of that information and how it relates to Fedora ARM to a wider audience by both the fedora-arm mailing list and my blog so those people that don’t sit on the IRC channel and those that like to lurk might have a better idea what’s going on.

The general format I plan to use is basically:

  • What’s new including SoCs, boards and new devices
  • Interesting bugs fixed
  • Outstanding bugs and issues
  • Random other insights

I don’t intend them to be long but rather short, sweet and to the point. They’ll probably come out when new major releases hit either rawhide or stable or something of particular interest lands. Feedback on both the format most other things is welcome as are questions and status of devices people might have had success or less so with.

I plan to have the first for 3.14 out later today and one for 3.15 RSN.

Simple WOL with RHEL and Fedora

I have a number of systems I use at home, they run a combination of RHEL and Fedora and are a variety of architectures (well x86 and ARM). Travelling a lot for $dayjob I don’t always want them on needlessly chewing power but on the flip side if I’m away for two weeks and I’d like to remotely use one and it’s off it’s a little tough to power it up if you’re 12 hours travel away!

So the obvious solution to this problem to get the best of both worlds is to use Wake On LAN (WOL) for supported hardware which means I can leave devices off or suspended and then just bring them up to full power when I actually want to use them from where ever I happen to be at the time, whether it be on the couch across the room or the other side of the world.

There’s a few places where you need to configure the device you wish to wake up. If you want to wake from power off you need to configure the BIOS/Firmware to have WOL enabled, this differs from device to device so this bit is an exercise for the reader. The quickest way to determine if your hardware supports WOL and its enabled in the BIOS/Firmware is to check to see if when the device is turned off and cabled up to a switch is if there’s a link light on the NIC is still lit.

Once you’ve got the HW and BIOS/Firmware sorted the next bit is pretty straight forward. First enable WOL on boot for the NIC in /etc/sysconfig/network-scripts/ifcfg- by adding the following line to the end of the file:

ETHTOOL_OPTIONS="wol g"

Then run

# ethtool -s  wol g

to enable it now. Finally record the ether address from the output from the “ip link” command so you know what MAC address you need to use to wake it up.

On your gateway make sure you have the appropriate tool to initiate the remote wakeup. On Fedora and RHEL this is ether-wake and it’s in the net-tools package. It’s then as simple as running

# ether-wake -i local-interface mac-address

Kernel taint state descriptions

It seems I always have to search for the kernel taint descriptions. So for reference they’re located in kernel/panic.c of the kernel source.

The descriptions are as follows:

/**
 *	print_tainted - return a string to represent the kernel taint state.
 *
 *  'P' - Proprietary module has been loaded.
 *  'F' - Module has been forcibly loaded.
 *  'S' - SMP with CPUs not designed for SMP.
 *  'R' - User forced a module unload.
 *  'M' - System experienced a machine check exception.
 *  'B' - System has hit bad_page.
 *  'U' - Userspace-defined naughtiness.
 *  'D' - Kernel has oopsed before
 *  'A' - ACPI table overridden.
 *  'W' - Taint on warning.
 *  'C' - modules from drivers/staging are loaded.
 *  'I' - Working around severe firmware bug.
 *  'O' - Out-of-tree module has been loaded.
 *
 *	The string is overwritten by the next call to print_tainted().
 */

I run into W regularly but as I don’t tend to use binary drivers, third party firmwares, force module loads or use custom ACPI tables (I wonder if a T for custom/out of tree Device Tree is needed?) I don’t tend to see many of the others. I do use staging drivers though. The other one I see quite regularly that isn’t documented in the above comment is G which means “GPL module loaded” ie basically the opposite to P “Proprietary”.

ARM hardware support on Fedora 20

A number of people complain about our hardware support on ARM. I’m not sure people really understand the issues we have bringing the hardware that we try to support to ARM and the constraints we work in. I’m going to try and outline the process we’ve been through over the last couple of releases and the reasons why we stick to the Fedora processes as much as possible. The other thing that I think people quickly forget is that the hardware support is a very small percentage of the overall work that has gone into bringing ARMv7 up to the excellent state it is in. Unfortunately like so many other projects the last 10% is always the hardest.

Support for hardware primarily revolves around three core components. They are the boot-loader (bios in x86 parlance), the kernel and any user space components needed which generally focuses on X drivers, but is no way the only thing.

Going back to the near memory Fedora 18 shipped with the 3.6.10 kernel. The 3.6 kernel didn’t ship with what’s known as the “Unified Multi Platform ARM kernel” which now allows us to ship a single kernel to boot multiple different SOCs (System on a Chip) by numerous manufacturers. The initial multi platform kernel support landed in 3.7 and initially allowed us to merge the Versatile Express and Calxeda Highbank kernels together. It was only with the 3.10 kernel we actually started to ship a single unified kernel that supported all the SoCs we support. We currently enable around a dozen different SoCs in the mainline kernel and actually support HW running on around half of those.

Supporting the multi platform kernel hasn’t been an easy ride. As you would expect of Fedora we were the first to adopt it and a lot of other distros still haven’t. Debian is in the process of moving over to it for Jessie, I thought Open SuSE was using it but it appears they’re not and Ubuntu certainly isn’t. I’m not sure about Arch or any of the others. It’s been an interesting ride and while a single kernel is something we’ve wanted for some time the lack of testing by a wider audience has certainly made it more painful for us than it could of been, it would be particularly helpful if Linaro, the main developers, actually used multi platform kernels for all their builds.

An example of problems we’ve had with the multi platform kernel is the support for the OMAP4 based PandaBoard, since before Fedora 17 it’s been consistently our best supported device but with the 3.11 kernel and the move of OMAP4 devices to DeviceTree it doesn’t boot and there’s been a number of us try to get to the bottom of the problem to no avail. Similarly going back to 3.6/3.7 the i.MX series of SOCs caused us massive problems to the point we just stopped supporting them but with the i.MX6 it’s very quickly becoming out best supported class of devices with a collection of almost a dozen very nice and relatively cheap devices (starting at USD$45) like the WandBoard dev boards, Compulabs Utilite and Cubox-i. Due to the high level of churn in the upstream kernel it at times makes it really hard and we can, and do, spend a lot of time chasing down issues with a single SoC or even device.

So what hardware actually works in Fedora 20? In theory we support 100s of devices but in practice the testing in limited to a selection of devices that we actually have to be able to easily test. So what actually works:

  • Versatile Express via QEMU emulation using libvirt.
  • Calxeda Highbank and Midway servers
  • Compulabs Trimslice (tegra).
  • The three WandBoards (i.MX6) in particular the Quad. 3.12 will improve the support too.
  • The BeagleBones (am33xx). In 3.11 there’s issues with usb, 3.12 is looking better.
  • BeagleBoard xM (omap3). For network you need to use the usb OTG port.
  • Mirabox and other Marvell (mvebu) devices with appended DTB (due to old uboot shipped with device).

Added to the above we ship around 100 device tree files, in 3.13 that rises to over a 120. I’ve had reports of a number of people successfully using some of these devices without issues. So what doesn’t work so well out of the box?:

  • PandaBoards (omap4). As mentioned above these broke with 3.11, I’m hoping to get them working again soon but my testing to date hasn’t been fruitful though.
  • Compulabs Utilite (i.MX6), well not with the 3.11 shipping kernel, we have initial support in 3.12 but there’s also issues with their initial uboot release.
  • AllWinner devices, but there’ll be a high quality remix that supports a lot of these devices, we’re hoping to improve the mainline support a lot in the F-21 development cycle.
  • Any device that doesn’t support multi platform kernels
  • Any device which isn’t supported in the upstream kernel

I’ve had reports of a number of other devices that “just work” or mostly work and with time this will improve. One thing that we’ve tried very hard not to do is pull 100s of patches in. Fedora likes to be as close to upstream as possible with their kernels and having lots of patches or kernel variants just doesn’t work as we just don’t have the resources to deal with them. We do on occasion pull in patches to fix or improve things but we try to keep as close to mainline as possible so if you want a device supported the first thing you’ll be asked is “What’s it’s supported state on the upstream kernel?”.

So what’s on the ToDo list? In short.. LOTS! With 3.14 we should finally get usable AllWinner devices, 3.13 in theory should have a number of RockChip devices supported and both of those SoCs bring a huge amount of cheap devices along that we have the potential to support. As it makes sense to enable new support we will. I’m also planning on improving and testing the support we currently have. I like the BeagleBone expansion capes so I plan on testing and playing with those and in general improving the support of devices that I have (a post on those coming soon).

All in all while we’ve got a long way to go I believe our hardware support has improved fairly well over the last couple of years where in Fedora 17 we officially supported two devices.