Semi irregular Fedora ARM kernel status reports

I thought I’d start doing semi irregular ARM kernel status reports. I’ll do them as often as I think they might be useful and those who know me know I travel a lot and randomly and that ARM isn’t my $dayjob.

Few are bored or stupid enough to follow the 20 or so ARM kernel trees or have the regular insight as to what’s happening, what’s landed, what new devices might work and what bugs come and go that I do so I thought I’d try and dispense some of the more interesting bits of that information and how it relates to Fedora ARM to a wider audience by both the fedora-arm mailing list and my blog so those people that don’t sit on the IRC channel and those that like to lurk might have a better idea what’s going on.

The general format I plan to use is basically:

  • What’s new including SoCs, boards and new devices
  • Interesting bugs fixed
  • Outstanding bugs and issues
  • Random other insights

I don’t intend them to be long but rather short, sweet and to the point. They’ll probably come out when new major releases hit either rawhide or stable or something of particular interest lands. Feedback on both the format most other things is welcome as are questions and status of devices people might have had success or less so with.

I plan to have the first for 3.14 out later today and one for 3.15 RSN.

Booting PandaBoard with Fedora 20 GA

So I’ve always thought the fix for Fedora 20 booting on the PandaBoard devices would be a simple fix. I’ve always assumed it would be a minor kernel config change. I was wrong, it was even more simple once we worked it out with some help of our friends!

So to make it work you can do the following. First get a Fedora ARM Image, I used the Minimal Image, you’ll need the VFAT variant for which ever one you choose.

As root (or use sudo) run the following commands but remember to change the device path:

xzcat Fedora-Minimal-VFAT-armhfp-20-1-sda.raw.xz > /dev/mmcblk0; sync

Eject the card and put back it back in so it gets the new partition table etc and then copy the uboot over to the uboot partition. In my case I did (but change the path to your mount points):

cd /run/media/USERNAME/__/usr/share/uboot-panda/
cp MLO u-boot.img /run/media/USERNAME/uboot/
cp uEnv.txt.panda_es /run/media/USERNAME/uboot/uEnv.txt
umount /run/media/USERNAME/__/ /run/media/USERNAME/uboot/
sync

At this point you’ll likely want to use gparted or similar tools to expand the root filesystem. Once complete run screen or your favourite serial console application:

screen /dev/ttyUSB0 115200

Put the SD card into the panda, apply power, and boot through to u-boot but interrupt the boot process and get the Panda# u-boot prompt and enter the following set of commands:

setenv bootm_size 0x20000000
setenv bootargs console=${console} vram=${vram} root=LABEL=_/ ro rootwait
ext4load mmc 0:3 0x82000000 /boot/vmlinuz-3.11.10-301.fc20.armv7hl
ext4load mmc 0:3 0x88080000 /boot/uInitrd-3.11.10-301.fc20.armv7hl
ext4load mmc 0:3 0x88000000 /boot/dtb-3.11.10-301.fc20.armv7hl/omap4-panda-es.dtb
bootz 0x82000000 0x88080000 0x88000000

From there it should boot all the way through to the first boot configuration tool on the serial console, once you set a root password, create new user and set the timezone it will boot you thought to the login prompt.

That will get you to the point of being able to play. Networking works fine as does USB and a bunch of other core functionality, X doesn’t appear to with the first look and if you update the kernel (I’ve tested the shipping 3.11.10-301.fc20 and latest 3.113.13.6-200) you’ll just need to follow the process above with updated kernel details to reboot.

We’re working to get updated arm-boot-config to ensure the expected automated boot process plus some updates for X etc but I wanted to get this process out for those chomping at the bit to play while we review the changes! As always please report any issues (or success) to the Fedora ARM mailing list or IRC channel.

In terms of working devices I’ve personally tested my Panda-ES prototype (which never successfully booted anything later than 3.6.x) and Panda-ES B2, I’ve had reports of a number of variants of the vanilla Panda and ES working. I’m not sure about the B3 revisions with the Elpida DDR2 RAM but you might want to try the above with the Fedora 2014.01 uboot as it has the uboot components needed.

Lastly I’d like to explicitly call out Paul Whalen for his persistence on this one and Tom Rini who ultimately provided the tiny bit of glue we needed to solve this so simply!

I’m glad to be able to add this mainstay of the Fedora ARM project back into the supported list with fan fair, I just wish we’d found this prior to F-20 GA!

Update: It appears that X with at least the XFCE image works with the GA image.

Simple WOL with RHEL and Fedora

I have a number of systems I use at home, they run a combination of RHEL and Fedora and are a variety of architectures (well x86 and ARM). Travelling a lot for $dayjob I don’t always want them on needlessly chewing power but on the flip side if I’m away for two weeks and I’d like to remotely use one and it’s off it’s a little tough to power it up if you’re 12 hours travel away!

So the obvious solution to this problem to get the best of both worlds is to use Wake On LAN (WOL) for supported hardware which means I can leave devices off or suspended and then just bring them up to full power when I actually want to use them from where ever I happen to be at the time, whether it be on the couch across the room or the other side of the world.

There’s a few places where you need to configure the device you wish to wake up. If you want to wake from power off you need to configure the BIOS/Firmware to have WOL enabled, this differs from device to device so this bit is an exercise for the reader. The quickest way to determine if your hardware supports WOL and its enabled in the BIOS/Firmware is to check to see if when the device is turned off and cabled up to a switch is if there’s a link light on the NIC is still lit.

Once you’ve got the HW and BIOS/Firmware sorted the next bit is pretty straight forward. First enable WOL on boot for the NIC in /etc/sysconfig/network-scripts/ifcfg- by adding the following line to the end of the file:

ETHTOOL_OPTIONS="wol g"

Then run 

# ethtool -s  wol g

to enable it now. Finally record the ether address from the output from the “ip link” command so you know what MAC address you need to use to wake it up.

On your gateway make sure you have the appropriate tool to initiate the remote wakeup. On Fedora and RHEL this is ether-wake and it’s in the net-tools package. It’s then as simple as running 

# ether-wake -i local-interface mac-address

puppetmaster on apache with passenger in 5 mins

Config management, DevOPs and all those sorts of time saving things are all well and good for readily repeatable tasks that happen on the same systems on a regular basis. But as a consultant my tasks are rarely the same and change from day to day depending on the customer, their mood or the phase of the moon. So while I’ve used puppet a lot I’ve only deployed a puppet platform once or twice and every time I do I end up digging around trying to work out the latest best practice.

So primarily to remind myself so I have a quick guide for the next time here’s my quick puppetmaster in five minutes using just rpm packages (so they’re easily upgraded and checked for security issues etc) on RHEL-6 or Fedora.

First ensure you have proper forward and reverse DNS records for the puppet master server and all puppet clients.

  1. Set SELinux to Permissive. This is a FIXME as there appears to be an issue running puppetmaster on rails on apache. A quick check it seems to be related to comms between httpd and rails. RHBZ 730837 has more info but I need to investigate but it should be straight forward . 
    sed 's/SELINUX=enforcing/SELINUX=permissive/' /etc/sysconfig/selinux
setenforce permissve
  2. On Fedora you already have all the repos required out of the box. On RHEL I’m using the upstream PuppetLabs repos so I used the following: 
    rhn-channel -a -c rhel-x86_64-server-optional-6
rpm -ivh http://yum.puppetlabs.com/el/6/products/x86_64/puppetlabs-release-6-10.noarch.rpm
rpm -ivh http://dl.fedoraproject.org/pub/epel/6/x86_64/epel-release-6-8.noarch.rpm
yum-config-manager --enable epel puppetlabs-products puppetlabs-deps
  3. Install the needed packages 
    yum install puppet-server httpd mod_ssl rubygem-rack mod_passenger
  4. Start the puppetmaster and set it to run on startup. While we won’t be using WEBrick we start it initially so it will initialise the default puppet config and setup things like certificates. 
    service puppetmaster start
puppet resource service puppetmaster ensure=running enable=true
  5. Configure the puppetmaster Rails app 
    mkdir -p /usr/share/puppet/rack/puppetmasterd
mkdir /usr/share/puppet/rack/puppetmasterd/public /usr/share/puppet/rack/puppetmasterd/tmp
cp /usr/share/puppet/ext/rack/config.ru /usr/share/puppet/rack/puppetmasterd/
chown -R puppet.puppet /usr/share/puppet/rack
restorecon -Rv /usr/share/puppet
mkdir /var/run/passenger
restorecon -Rv /var/run/passenger
  6. Configure apache by adding the config below and updating FQDN.pem to your local certificate name.
    vim /etc/httpd/conf.d/puppetmaster.conf

    # RHEL/CentOS:
# And the passenger performance tuning settings:
PassengerHighPerformance On
PassengerUseGlobalQueue On
# Set this to about 1.5 times the number of CPU cores in your master:
PassengerMaxPoolSize 6
# Recycle master processes after they service 1000 requests
PassengerMaxRequests 1000
# Stop processes if they sit idle for 10 minutes
PassengerPoolIdleTime 600
PassengerTempDir /var/run/passenger

Listen 8140
<VirtualHost *:8140>
    SSLEngine On

    # Only allow high security cryptography. Alter if needed for compatibility.
    SSLProtocol             All -SSLv2
    SSLCipherSuite          HIGH:!ADH:RC4+RSA:-MEDIUM:-LOW:-EXP
    SSLCertificateFile      /var/lib/puppet/ssl/certs/FQDN.pem
    SSLCertificateKeyFile   /var/lib/puppet/ssl/private_keys/FQDN.pem
    SSLCertificateChainFile /var/lib/puppet/ssl/ca/ca_crt.pem
    SSLCACertificateFile    /var/lib/puppet/ssl/ca/ca_crt.pem
    SSLCARevocationFile     /var/lib/puppet/ssl/ca/ca_crl.pem
    SSLVerifyClient         optional
    SSLVerifyDepth          1
    SSLOptions              +StdEnvVars +ExportCertData

    # These request headers are used to pass the client certificate
    # authentication information on to the puppet master process
    RequestHeader set X-SSL-Subject %{SSL_CLIENT_S_DN}e
    RequestHeader set X-Client-DN %{SSL_CLIENT_S_DN}e
    RequestHeader set X-Client-Verify %{SSL_CLIENT_VERIFY}e

    RackAutoDetect On
    DocumentRoot /usr/share/puppet/rack/puppetmasterd/public/
    <Directory /usr/share/puppet/rack/puppetmasterd/>
        Options None
        AllowOverride None
        Order Allow,Deny
        Allow from All
    </Directory>
</VirtualHost>

    restorecon -Rv /etc/httpd/conf.d

  7. Disable the puppetmaster so it runs via Rails on Apache 
    chkconfig puppetmaster off; service puppetmaster stop
chkconfig httpd on; service httpd restart
  8. Setup a basic puppet config files:
    • autosign.conf

      vim /etc/puppet/autosign.conf

      *.infra.example.com
    • site.pp
      vim /etc/puppet/manifests/site.pp

      # /etc/puppet/manifests/site.pp

import "nodes"

# The filebucket option allows for file backups to the server
filebucket { main: server => 'puppetmaster.infra.example.com' }

# Set global defaults - including backing up all files to the main filebucket and adds a global path
File { backup => main }
Exec { path => "/usr/bin:/usr/sbin:/bin:/sbin" }
    • modules.pp
      vim /etc/puppet/manifests/modules.pp

      # /etc/puppet/manifests/modules.pp

import "dns-client"
import "ntp-client"
    • nodes.pp
      vim /etc/puppet/manifests/nodes.pp

      # /etc/puppet/manifests/nodes.pp

node 'common' {
        include dns-client
        include ntp-client
}

node default inherits common {
}

node 'puppetmaster.infra.example.com' inherits common {
}

    With the above you obviously need to have the dns-client and ntp-client modules in place in /etc/puppet/modules/ or update the config files to reflect the modules you have installed!

  9. Relabel all the various puppet directories to ensure we’re good with selinux:
    restorecon -Rv /etc/puppet
  10. Reboot to ensure it all works when it comes back online 😉

P.S. I’ll update this post once I’ve had time to work out the SELinux issues, pointers welcome in the comments below 🙂

SSH escape sequences

A lot of people don’t know that ssh has a number of escape sequences that can be run. I’ve often used the ~. to kill a stuck session but I recently discovered, by mistyping the aforementioned option, there is a number of other useful options too:

Supported escape sequences:
  ~.  - terminate connection (and any multiplexed sessions)
  ~B  - send a BREAK to the remote system
  ~C  - open a command line
  ~R  - Request rekey (SSH protocol 2 only)
  ~^Z - suspend ssh
  ~#  - list forwarded connections
  ~&  - background ssh (when waiting for connections to terminate)
  ~?  - this message
  ~~  - send the escape character by typing it twice
(Note that escapes are only recognized immediately after newline.)

Kernel taint state descriptions

It seems I always have to search for the kernel taint descriptions. So for reference they’re located in kernel/panic.c of the kernel source.

The descriptions are as follows:

/**
 *	print_tainted - return a string to represent the kernel taint state.
 *
 *  'P' - Proprietary module has been loaded.
 *  'F' - Module has been forcibly loaded.
 *  'S' - SMP with CPUs not designed for SMP.
 *  'R' - User forced a module unload.
 *  'M' - System experienced a machine check exception.
 *  'B' - System has hit bad_page.
 *  'U' - Userspace-defined naughtiness.
 *  'D' - Kernel has oopsed before
 *  'A' - ACPI table overridden.
 *  'W' - Taint on warning.
 *  'C' - modules from drivers/staging are loaded.
 *  'I' - Working around severe firmware bug.
 *  'O' - Out-of-tree module has been loaded.
 *
 *	The string is overwritten by the next call to print_tainted().
 */

I run into W regularly but as I don’t tend to use binary drivers, third party firmwares, force module loads or use custom ACPI tables (I wonder if a T for custom/out of tree Device Tree is needed?) I don’t tend to see many of the others. I do use staging drivers though. The other one I see quite regularly that isn’t documented in the above comment is G which means “GPL module loaded” ie basically the opposite to P “Proprietary”.

ARM hardware support on Fedora 20

A number of people complain about our hardware support on ARM. I’m not sure people really understand the issues we have bringing the hardware that we try to support to ARM and the constraints we work in. I’m going to try and outline the process we’ve been through over the last couple of releases and the reasons why we stick to the Fedora processes as much as possible. The other thing that I think people quickly forget is that the hardware support is a very small percentage of the overall work that has gone into bringing ARMv7 up to the excellent state it is in. Unfortunately like so many other projects the last 10% is always the hardest.

Support for hardware primarily revolves around three core components. They are the boot-loader (bios in x86 parlance), the kernel and any user space components needed which generally focuses on X drivers, but is no way the only thing.

Going back to the near memory Fedora 18 shipped with the 3.6.10 kernel. The 3.6 kernel didn’t ship with what’s known as the “Unified Multi Platform ARM kernel” which now allows us to ship a single kernel to boot multiple different SOCs (System on a Chip) by numerous manufacturers. The initial multi platform kernel support landed in 3.7 and initially allowed us to merge the Versatile Express and Calxeda Highbank kernels together. It was only with the 3.10 kernel we actually started to ship a single unified kernel that supported all the SoCs we support. We currently enable around a dozen different SoCs in the mainline kernel and actually support HW running on around half of those.

Supporting the multi platform kernel hasn’t been an easy ride. As you would expect of Fedora we were the first to adopt it and a lot of other distros still haven’t. Debian is in the process of moving over to it for Jessie, I thought Open SuSE was using it but it appears they’re not and Ubuntu certainly isn’t. I’m not sure about Arch or any of the others. It’s been an interesting ride and while a single kernel is something we’ve wanted for some time the lack of testing by a wider audience has certainly made it more painful for us than it could of been, it would be particularly helpful if Linaro, the main developers, actually used multi platform kernels for all their builds.

An example of problems we’ve had with the multi platform kernel is the support for the OMAP4 based PandaBoard, since before Fedora 17 it’s been consistently our best supported device but with the 3.11 kernel and the move of OMAP4 devices to DeviceTree it doesn’t boot and there’s been a number of us try to get to the bottom of the problem to no avail. Similarly going back to 3.6/3.7 the i.MX series of SOCs caused us massive problems to the point we just stopped supporting them but with the i.MX6 it’s very quickly becoming out best supported class of devices with a collection of almost a dozen very nice and relatively cheap devices (starting at USD$45) like the WandBoard dev boards, Compulabs Utilite and Cubox-i. Due to the high level of churn in the upstream kernel it at times makes it really hard and we can, and do, spend a lot of time chasing down issues with a single SoC or even device.

So what hardware actually works in Fedora 20? In theory we support 100s of devices but in practice the testing in limited to a selection of devices that we actually have to be able to easily test. So what actually works:

  • Versatile Express via QEMU emulation using libvirt.
  • Calxeda Highbank and Midway servers
  • Compulabs Trimslice (tegra).
  • The three WandBoards (i.MX6) in particular the Quad. 3.12 will improve the support too.
  • The BeagleBones (am33xx). In 3.11 there’s issues with usb, 3.12 is looking better.
  • BeagleBoard xM (omap3). For network you need to use the usb OTG port.
  • Mirabox and other Marvell (mvebu) devices with appended DTB (due to old uboot shipped with device).

Added to the above we ship around 100 device tree files, in 3.13 that rises to over a 120. I’ve had reports of a number of people successfully using some of these devices without issues. So what doesn’t work so well out of the box?:

  • PandaBoards (omap4). As mentioned above these broke with 3.11, I’m hoping to get them working again soon but my testing to date hasn’t been fruitful though.
  • Compulabs Utilite (i.MX6), well not with the 3.11 shipping kernel, we have initial support in 3.12 but there’s also issues with their initial uboot release.
  • AllWinner devices, but there’ll be a high quality remix that supports a lot of these devices, we’re hoping to improve the mainline support a lot in the F-21 development cycle.
  • Any device that doesn’t support multi platform kernels
  • Any device which isn’t supported in the upstream kernel

I’ve had reports of a number of other devices that “just work” or mostly work and with time this will improve. One thing that we’ve tried very hard not to do is pull 100s of patches in. Fedora likes to be as close to upstream as possible with their kernels and having lots of patches or kernel variants just doesn’t work as we just don’t have the resources to deal with them. We do on occasion pull in patches to fix or improve things but we try to keep as close to mainline as possible so if you want a device supported the first thing you’ll be asked is “What’s it’s supported state on the upstream kernel?”.

So what’s on the ToDo list? In short.. LOTS! With 3.14 we should finally get usable AllWinner devices, 3.13 in theory should have a number of RockChip devices supported and both of those SoCs bring a huge amount of cheap devices along that we have the potential to support. As it makes sense to enable new support we will. I’m also planning on improving and testing the support we currently have. I like the BeagleBone expansion capes so I plan on testing and playing with those and in general improving the support of devices that I have (a post on those coming soon).

All in all while we’ve got a long way to go I believe our hardware support has improved fairly well over the last couple of years where in Fedora 17 we officially supported two devices.

Configuring filesystem “TRIM” options on Fedora or RHEL

The SATA TRIM option, or discard if you use enterprise SCSI/SAS, that everyone likes to ensure their whiz bang SSD supports actually needs some configuration on Linux. There’s a few tasks that need to be done and some depend on your partitioning configuration.

Filesystem mount options: You’ll need to be using ext4/xfs/btrfs and mount with the “-o discard” option. To do this automatically in /etc/fstab just add ,discard after the defaults option.

LVM config: In /etc/lvm/lvm.conf file set the issue_discards option to 1. So issue_discards = 1.

LUKS config: In /etc/crypttab file add discards to the end of the appropriate luks lines, likely it’ll look like the following: none discards. It’s worth nothing there are some security implications by enabling discards with an encrypted filesystem.

For the fstab changes to take effect you just need to reboot. For the LVM and crypttab changes to take effect you also need to regenerate the initrd (or just wait for the next kernel update 🙂 ).

The above will enable online discards. You can also do it in batches with the fstrim command which is as simple as fstrim mount-point.

Serial console options on the Beagle Bone Black

So unlike the original Beagle Bone, which had a built in USB serial adapter, the Beagle Bone Black only has a serial header and you have to buy a USB to serial adapter to get a real serial console.

There’s one other option for a pseudo serial console over the USB On the Go port but the problem with this is that it doesn’t work with u-boot so only works once the kernel has booted as we can setup the port. The enabling and use of the USB OtG in Fedora is still on my ToDo list to investigate for ARM but we can possibly enable it as both serial and usb network at some point in the future.

So for now we need to use the 6 pin header to connect a USB to serial adapter. The most important thing to note here is that it requires 3.3 volts for the data signals so don’t use some of the older 5 volt units. The best USB to Serial to use is the FTDI FT232RL but at $20 it’s almost half the price of the device. The advantage is that it just works and the 6 pin connect just plugs straight onto the board (black goes to PIN 1). I’ve also tried the Adafruit 4 Pin Cable (PL2303), which at $9.95, is less than half the price and as the 4 pins are on 4 separate 1 pin headers it can be used on a number of different devices as it doesn’t matter how the serial headers are pinned out. To connect to The BBBlack the black wire goes to PIN 1 (Ground), the green wire to PIN 4 (Receive) and the white wire goes to PIN 5 (Transmit). The red wire is power and isn’t needed. CircuitCo has a number of other Serial Cable Options listed and the appropriate configurations for them too.

Now for a serial console app on Fedora I usually use screen. So simply once you have your serial console connected to the BeagleBoneBlack just plug the usb port into you computer and run sudo screen /dev/ttyUSB0 115200 and then power up the Beagle Bone and you should soon see the output from u-boot and you’re on your way.

Nose back to the grindstone

Some may have noticed that I’ve not been around as much as usual in the last month, I’ve been on 4 weeks (originally meant to be 3) of holidays visiting my family back in Australia. It’d been 2.5 years since I’d last been home and well over a year since I’d had any form of break at all so I decided to disconnect myself as much as possible. I vaguely kept up to date on email replying to the urgent stuff and just deleting most stuff leaving the bits in between to deal with when I got back. So I’m now going through the large stack of outstanding stuff but if you’ve been expecting a response from me now you know why you’ve not had one and if you don’t get a response by the end of the week I suggest you poke me via IRC/Email again to follow me up. I have Fedora 20, Fedora ARM, Fedora in general and $dayjob on my list…. not necessarily in that order! I’m endeavouring to prioritise… we’ll see how that goes!

For those that don’t believe I can disconnect here’s a couple of photos for your viewing pleasure (click for full size) up on the family farm.

The farm at sunrise
The farm at sunrise
Me on a horse! Off to do sheep work
Me on a horse! Off to do sheep work