Skip to content
Jul 8 14

3.16 Fedora ARM kernel status

by nullr0ute

So 3.16 is has quite a few new features in terms of newly supported devices, also some what surprisingly this blog post will be out before 3.16! In terms of new device support all the SoCs listed here are exciting for a number of reasons for Fedora ARM. Aarch64 (ARM64) makes it’s first debut with support of real hardware although we’ve actually had kernel support enable for it for some time in Fedora even if only usable on the glacial Foundation emulator.

The 3.16 release is also very likely to be the kernel that ships with Fedora 21 GA and with the Alpha due in about a month we’re starting to polish and test all the platforms and devices we want to support for GA.

Anyway without any further a do let’s get into the gritty details:

  • NVIDIA Jetson TK1 support: While we’ve had the basics of this for a while all of the bits are there now.
  • EXYNOS support: This SoC is probably the most asked about platform and finally after a long wait the multiplatform support has landed upstream. We currently ship around 20 dtb files for exynos4 and 5 (Chromebook support anyone?). Testing is sought and feedback and greatly appreciated.
  • Qualcomm MSM 8×60, 8960 and 8974 support: While the multiplatform support for these devices landed upstream a few releases a go they’re now to the point they should be relatively usable so it’s time to get wider testing. This should be the beginning of supporting the venerable ifc6410 and dragonboard devices.
  • APM X-GENE support: One of our first aarch64 supported pieces of hardware. Similar to the QCom SoC the initial support has been upstream for a while but with 3.16 it becomes usable with the vast majority of basic support upstream so minimal patches are needed. More on aarch64 soon.
  • AMD Seattle support: The other of our aarch64 supported pieces of hardware if you’re lucky enough to get your mits on a device.

The other feature we’re starting to see mature is GPU and Graphics support. I’m not exactly sure yet as to what the final state of this functionality will be for Fedora 21 GA but we potentially will have suppport for:

  • nouveau/mesa support on the NVIDIA Tegra K1
  • freedreno/mesa support on the Qualcomm boards
  • etnaviv/mesa support on i.MX6 devices
  • improved modesetting support for a number of other devices. Some of this has already landed and is usable in rawhide now.

What covered above is just a high level overview of what’s new in the upcoming release. There’s been numerous other improvements in existing supported SoCs and devices all over the place that would take too long to cover off here but in short with all the shiny that’s landed in 3.16 what Fedora ARM will look like as part of the Fedora 21 GA release is quickly starting to take shape.

Jun 30 14

3.15 Fedora ARM kernel status

by nullr0ute

There’s been quite a bit of water under the bridge since my post on the 3.14 kernel status. With 3.15.x due to land in Fedora 20 shortly I thought I’d give an overview of changes for 3.15 and what’s happened since the last post.

From a shiny new devices and features point of view the 3.15 kernel is relatively boring on the ARM devices front, the advantage of that was that from a development point of view things tended to just work on Fedora. Running a diff between out 3.14 and 3.15 ARM kernel configs and checking our shipped Device Tree Blobs I get the following main changes:

  • Enabling of Marvell Dove platform. This primarily will be useful for people with the Original Cubox
  • SunXi MMC suuport. The enables initial basic support (Serial/MMC/network) for a number of AllWinner platforms
  • Zynq 7xxx platform improvements
  • OMAP DRM driver conversion to Device Tree (more on that below)
  • Initial Utilite support. It’s pretty basic with support for serial, MMC/SATA, and one of the two NICS. I plan on improving this soon
  • Added Device Tree support for a number of OMAP Overo devices
  • Added Device Tree support for a number of i.MX6 based devices

So while it was boring form a new device support point of view a kernel cycle for ARM is never really that boring! There was a lot of nice improvements generally under the hood and the march toward Device Tree is basically complete. I’m not aware of any device now that is supported not through DT in Fedora.

I mentioned above OMAP DRM. In the 3.14 post I mentioned I was sure we’d get Panda working soon. And we did! The main issue remaining was actually display support with 3.14 and with 3.15 that problem is now mostly closed because all the connectors and their associated drivers now support DT which meeans all the modules now load in the right order and things mostly just work. There’s some further improvments here in Xorg userspace in rawhide so I’d sugggest trying a nightly or the not far off Fedora 21 Alpha.

I’ve also had a few cycles to test Marvell mvebu support on my Mirabox and fixed a few kernel issues here so it now works. Unfortunately Marvell’s support of uboot, and hence Device Tree, is from the last decade and hence fairly horrible! I’ll save details of that for another post.

Jun 7 14

aarch64 (arm64) on Fedora 21 update

by nullr0ute

I realise that there’s not been an update as to the state of ARM aarch64 on Fedora for sometime. We should really shout out our achievements a lot more as over the last few months or so there’s been fairly extreme progress made but we all just tend to apply the nose to the grindstone and get one with it!

So where are we? Well we’ve got the vast majoriity of Fedora built. As we’re only building for rawhide/F-21 there’s a number of missing F-21 builds but with the mass rebuild well and truly underway on mainline we’re all hyped up and building on aarch64 too (appart from a power outage in the Huntsville facility that hosts some of the builders!) and we’re hot on the tail of mainline builds.

So we have HW virtualisation support and a bunch of virt bits on top of that, the vast majority of other services and low level support like storage platforms, all the various desktops and most of the various development stacks (even Ada!). It’s actually looking pretty good. In recent time we’ve added R, ghc and a raft of different language support. The java team have done a really good job getting java-1.8.0-openjdk up to spec too!

So appart for the mainline builds that haven’t yet been built for F-21 what are we missing? Well not much really now. Go and nodejs, the later due to missing v8 js engine support (it’s upstream in a newer v8 release but Fedora is stil trailing), and the things that depend on them are the main ones now missing so no Docker as yet either. There’s no mono, and while the core erlang is built I need to work out the correct order to finishing the bootstrap.

The rest of the job now becomes somewhat boring in that once the mass rebuild is complete we just need to go through packages and fix individual packages that are FTBFS and I strongly suspect most of those will have issues on mainline too.

On the hardware and virtualisation front the current rawhide kernel should boot just fine on the ARM foundation model emulator and the next release of qemu will have full system emulation of aarch64 to enable the running of aarch64 on x86_64 similar to what we can do with ARMv7 now. The 3.16 kernel should allow us to run on the currently and soon to be available HW without too many problems either as the last core bits of support should land upstream in the current merge window.

In short aarch64 on Fedora is looking awesome! By the time Fedora 21 Alpha lands in two months time we’ll basically be feature complete, and even now it’s extremely usable for those lucky enough to have hardware or patient enough to use software emulators.

May 12 14

Configure Firefox for Kerberos SSO on Fedora or RHEL

by nullr0ute

Single Sign On (SSO) is one of those nirvana things that rarely happens and a lot of people don’t understand (a central single username/password is not SSO people if you have to keep entering the password!!) but it’s very useful when it does work. With Firefox on Fedora or RHEL it works but it does take a little bit of config to do so efficiently.

  1. In the Firefox address bar type about:config to display the list of current configuration options.
  2. Filter the options with ‘negotiate’ to reduce the list of options.
  3. Double-click the network.negotiate-auth.trusted-uris entry and add the kerberos auth domain, eg: “.example.com” and click OK.
  4. Repeat the process for the network.negotiate-auth.delegation-uris entry, using the same domain.

Now you need to ensure you have a kerberos ticket. If you’ve configured an “Enterprise Login” using the GNOME Online Accounts check you have a ticket with the ‘klist’ command from a terminal window. If you don’t have a valid ticket run ‘kinit’ and from there you should be able to visit SSO enable sites and be auto logged in. Magic!

Apr 17 14

3.14 Fedora ARM kernel status

by nullr0ute

With 3.14.1 out Josh and Justin are preparing to land the 3.14 kernel into Fedora 20. So what does it mean in terms of ARM on Fedora. Well it’s an evolution. There’s of course the usual raft of new devices and some new SoCs, and best of all lots of improvements in support for existing devices. Even the return of some old favourites! Generally from the stash of devices Paul, I and others have that get regularly tested things are looking pretty good with 3.14.

New SoCs and device support:

  • Tegra 4/K1 support has been enabled. For 3.14 this doesn’t mean much as there’s not a wide level of devices out there that we ship device tree blobs for but it’s a good preparation for 3.15 as we should have pretty reasonable support for the nice new Tegra K1 dev board!
  • TI devices: The 3.14 finally brings working support for the OMAP5 EVM board, this will improve further in 3.15 but it boots and generally works. It also adds support for the original BeagleBone and the USB is now finally all back for the Beagle-XM devices so they go back into the fully working list! Also the DT bits for the Overo devices has started to land so if is interested in those I’d love feedback from people with those devices.
  • Freescale i.MX: To this lovely growing list we add initial support for the various Cubox-i devices and the hummingboard. Still no HDMI support but here’s hoping for 3.15
  • Xilinx Zynq 7000: The SD controller has finally landed for this which means they should be bootable to login but from there I’m not sure the status of the rest of the support. We ship DT for the zynq-zc702, zynq-zc706 and zynq-zed devices so if you’ve got one of those and have time to test feedback would be welcome.

Interesting bugs fixed:
Nothing particularly exciting comes to mind here. The fix for the Beagle-XM usb hub power up is nice, as was the final config option for the BeagleBone White. There’s obviously a deluge of other ARM driver fixes and improvements (PTP high precision time support for modular CPSW ethernet on the BeagleBone’s anyone?).

Outstanding bugs and issues:
So this is really the list of items I have outstanding for 3.14 so that I can spin some new images. Feedback on any of these and anything I might not be aware of would be very useful.

  • OMAP DRM display. In 3.11 a new display framework landed for the OMAP DRM driver and in 3.12 the old one was dropped. This broke X on devices like the Pandaboard for 3.12+ kernels. I know roughly the problematic area but I just need to get the cycles to debug this. Any help is welcome.
  • Serial over USB. While this isn’t a kernel bug but rather just needs me to hack together some scripts it’s a blocker for easy OOTB support for devices like the BeagleBone Black
  • Tegra DRM display. After a re-write it’s back and modular again in 3.14. Just need to ensure it’s working and ready to go
  • Testing… testing… testing :-)

That’s mostly all I have on my list. If there’s anything I’m not aware of please do let me know and I’ll endevour to help out where possible. In particular I’m very interested in boot issues for devices that would would be supportable with new ARM images based on 3.14. From the 3.11 kernel that F-20 GA shipped with there’s been a lot of change and improvements, and while non boot enhancements are easy to do with a “yum update” issues with boot aren’t quite so easy to deal with!

Apr 17 14

Semi irregular Fedora ARM kernel status reports

by nullr0ute

I thought I’d start doing semi irregular ARM kernel status reports. I’ll do them as often as I think they might be useful and those who know me know I travel a lot and randomly and that ARM isn’t my $dayjob.

Few are bored or stupid enough to follow the 20 or so ARM kernel trees or have the regular insight as to what’s happening, what’s landed, what new devices might work and what bugs come and go that I do so I thought I’d try and dispense some of the more interesting bits of that information and how it relates to Fedora ARM to a wider audience by both the fedora-arm mailing list and my blog so those people that don’t sit on the IRC channel and those that like to lurk might have a better idea what’s going on.

The general format I plan to use is basically:

  • What’s new including SoCs, boards and new devices
  • Interesting bugs fixed
  • Outstanding bugs and issues
  • Random other insights

I don’t intend them to be long but rather short, sweet and to the point. They’ll probably come out when new major releases hit either rawhide or stable or something of particular interest lands. Feedback on both the format most other things is welcome as are questions and status of devices people might have had success or less so with.

I plan to have the first for 3.14 out later today and one for 3.15 RSN.

Mar 12 14

Booting PandaBoard with Fedora 20 GA

by nullr0ute

So I’ve always thought the fix for Fedora 20 booting on the PandaBoard devices would be a simple fix. I’ve always assumed it would be a minor kernel config change. I was wrong, it was even more simple once we worked it out with some help of our friends!

So to make it work you can do the following. First get a Fedora ARM Image, I used the Minimal Image, you’ll need the VFAT variant for which ever one you choose.

As root (or use sudo) run the following commands but remember to change the device path:

xzcat Fedora-Minimal-VFAT-armhfp-20-1-sda.raw.xz > /dev/mmcblk0; sync

Eject the card and put back it back in so it gets the new partition table etc and then copy the uboot over to the uboot partition. In my case I did (but change the path to your mount points):

cd /run/media/USERNAME/__/usr/share/uboot-panda/
cp MLO u-boot.img /run/media/USERNAME/uboot/
cp uEnv.txt.panda_es /run/media/USERNAME/uboot/uEnv.txt
umount /run/media/USERNAME/__/ /run/media/USERNAME/uboot/
sync

At this point you’ll likely want to use gparted or similar tools to expand the root filesystem. Once complete run screen or your favourite serial console application:

screen /dev/ttyUSB0 115200

Put the SD card into the panda, apply power, and boot through to u-boot but interrupt the boot process and get the Panda# u-boot prompt and enter the following set of commands:

setenv bootm_size 0x20000000
setenv bootargs console=${console} vram=${vram} root=LABEL=_/ ro rootwait
ext4load mmc 0:3 0x82000000 /boot/vmlinuz-3.11.10-301.fc20.armv7hl
ext4load mmc 0:3 0x88080000 /boot/uInitrd-3.11.10-301.fc20.armv7hl
ext4load mmc 0:3 0x88000000 /boot/dtb-3.11.10-301.fc20.armv7hl/omap4-panda-es.dtb
bootz 0x82000000 0x88080000 0x88000000

From there it should boot all the way through to the first boot configuration tool on the serial console, once you set a root password, create new user and set the timezone it will boot you thought to the login prompt.

That will get you to the point of being able to play. Networking works fine as does USB and a bunch of other core functionality, X doesn’t appear to with the first look and if you update the kernel (I’ve tested the shipping 3.11.10-301.fc20 and latest 3.113.13.6-200) you’ll just need to follow the process above with updated kernel details to reboot.

We’re working to get updated arm-boot-config to ensure the expected automated boot process plus some updates for X etc but I wanted to get this process out for those chomping at the bit to play while we review the changes! As always please report any issues (or success) to the Fedora ARM mailing list or IRC channel.

In terms of working devices I’ve personally tested my Panda-ES prototype (which never successfully booted anything later than 3.6.x) and Panda-ES B2, I’ve had reports of a number of variants of the vanilla Panda and ES working. I’m not sure about the B3 revisions with the Elpida DDR2 RAM but you might want to try the above with the Fedora 2014.01 uboot as it has the uboot components needed.

Lastly I’d like to explicitly call out Paul Whalen for his persistence on this one and Tom Rini who ultimately provided the tiny bit of glue we needed to solve this so simply!

I’m glad to be able to add this mainstay of the Fedora ARM project back into the supported list with fan fair, I just wish we’d found this prior to F-20 GA!

Update: It appears that X with at least the XFCE image works with the GA image.

Feb 9 14

Simple WOL with RHEL and Fedora

by nullr0ute

I have a number of systems I use at home, they run a combination of RHEL and Fedora and are a variety of architectures (well x86 and ARM). Travelling a lot for $dayjob I don’t always want them on needlessly chewing power but on the flip side if I’m away for two weeks and I’d like to remotely use one and it’s off it’s a little tough to power it up if you’re 12 hours travel away!

So the obvious solution to this problem to get the best of both worlds is to use Wake On LAN (WOL) for supported hardware which means I can leave devices off or suspended and then just bring them up to full power when I actually want to use them from where ever I happen to be at the time, whether it be on the couch across the room or the other side of the world.

There’s a few places where you need to configure the device you wish to wake up. If you want to wake from power off you need to configure the BIOS/Firmware to have WOL enabled, this differs from device to device so this bit is an exercise for the reader. The quickest way to determine if your hardware supports WOL and its enabled in the BIOS/Firmware is to check to see if when the device is turned off and cabled up to a switch is if there’s a link light on the NIC is still lit.

Once you’ve got the HW and BIOS/Firmware sorted the next bit is pretty straight forward. First enable WOL on boot for the NIC in /etc/sysconfig/network-scripts/ifcfg- by adding the following line to the end of the file:

ETHTOOL_OPTIONS="wol g"

Then run

# ethtool -s  wol g

to enable it now. Finally record the ether address from the output from the “ip link” command so you know what MAC address you need to use to wake it up.

On your gateway make sure you have the appropriate tool to initiate the remote wakeup. On Fedora and RHEL this is ether-wake and it’s in the net-tools package. It’s then as simple as running

# ether-wake -i local-interface mac-address
Jan 30 14

puppetmaster on apache with passenger in 5 mins

by nullr0ute

Config management, DevOPs and all those sorts of time saving things are all well and good for readily repeatable tasks that happen on the same systems on a regular basis. But as a consultant my tasks are rarely the same and change from day to day depending on the customer, their mood or the phase of the moon. So while I’ve used puppet a lot I’ve only deployed a puppet platform once or twice and every time I do I end up digging around trying to work out the latest best practice.

So primarily to remind myself so I have a quick guide for the next time here’s my quick puppetmaster in five minutes using just rpm packages (so they’re easily upgraded and checked for security issues etc) on RHEL-6 or Fedora.

First ensure you have proper forward and reverse DNS records for the puppet master server and all puppet clients.

  1. Set SELinux to Permissive. This is a FIXME as there appears to be an issue running puppetmaster on rails on apache. A quick check it seems to be related to comms between httpd and rails. RHBZ 730837 has more info but I need to investigate but it should be straight forward .
    sed 's/SELINUX=enforcing/SELINUX=permissive/' /etc/sysconfig/selinux
    setenforce permissve
    
  2. On Fedora you already have all the repos required out of the box. On RHEL I’m using the upstream PuppetLabs repos so I used the following:
    rhn-channel -a -c rhel-x86_64-server-optional-6
    rpm -ivh http://yum.puppetlabs.com/el/6/products/x86_64/puppetlabs-release-6-10.noarch.rpm
    rpm -ivh http://dl.fedoraproject.org/pub/epel/6/x86_64/epel-release-6-8.noarch.rpm
    yum-config-manager --enable epel puppetlabs-products puppetlabs-deps
    
  3. Install the needed packages
    yum install puppet-server httpd mod_ssl rubygem-rack mod_passenger
    
  4. Start the puppetmaster and set it to run on startup. While we won’t be using WEBrick we start it initially so it will initialise the default puppet config and setup things like certificates.
    service puppetmaster start
    puppet resource service puppetmaster ensure=running enable=true
    
  5. Configure the puppetmaster Rails app
    mkdir -p /usr/share/puppet/rack/puppetmasterd
    mkdir /usr/share/puppet/rack/puppetmasterd/public /usr/share/puppet/rack/puppetmasterd/tmp
    cp /usr/share/puppet/ext/rack/config.ru /usr/share/puppet/rack/puppetmasterd/
    chown -R puppet.puppet /usr/share/puppet/rack
    restorecon -Rv /usr/share/puppet
    mkdir /var/run/passenger
    restorecon -Rv /var/run/passenger
    
  6. Configure apache by adding the config below and updating FQDN.pem to your local certificate name.
    vim /etc/httpd/conf.d/puppetmaster.conf

    # RHEL/CentOS:
    # And the passenger performance tuning settings:
    PassengerHighPerformance On
    PassengerUseGlobalQueue On
    # Set this to about 1.5 times the number of CPU cores in your master:
    PassengerMaxPoolSize 6
    # Recycle master processes after they service 1000 requests
    PassengerMaxRequests 1000
    # Stop processes if they sit idle for 10 minutes
    PassengerPoolIdleTime 600
    PassengerTempDir /var/run/passenger
    
    Listen 8140
    <VirtualHost *:8140>
        SSLEngine On
    
        # Only allow high security cryptography. Alter if needed for compatibility.
        SSLProtocol             All -SSLv2
        SSLCipherSuite          HIGH:!ADH:RC4+RSA:-MEDIUM:-LOW:-EXP
        SSLCertificateFile      /var/lib/puppet/ssl/certs/FQDN.pem
        SSLCertificateKeyFile   /var/lib/puppet/ssl/private_keys/FQDN.pem
        SSLCertificateChainFile /var/lib/puppet/ssl/ca/ca_crt.pem
        SSLCACertificateFile    /var/lib/puppet/ssl/ca/ca_crt.pem
        SSLCARevocationFile     /var/lib/puppet/ssl/ca/ca_crl.pem
        SSLVerifyClient         optional
        SSLVerifyDepth          1
        SSLOptions              +StdEnvVars +ExportCertData
    
        # These request headers are used to pass the client certificate
        # authentication information on to the puppet master process
        RequestHeader set X-SSL-Subject %{SSL_CLIENT_S_DN}e
        RequestHeader set X-Client-DN %{SSL_CLIENT_S_DN}e
        RequestHeader set X-Client-Verify %{SSL_CLIENT_VERIFY}e
    
        RackAutoDetect On
        DocumentRoot /usr/share/puppet/rack/puppetmasterd/public/
        <Directory /usr/share/puppet/rack/puppetmasterd/>
            Options None
            AllowOverride None
            Order Allow,Deny
            Allow from All
        </Directory>
    </VirtualHost>
    

    restorecon -Rv /etc/httpd/conf.d

  7. Disable the puppetmaster so it runs via Rails on Apache
    chkconfig puppetmaster off; service puppetmaster stop
    chkconfig httpd on; service httpd restart
    
  8. Setup a basic puppet config files:
    • autosign.conf

      vim /etc/puppet/autosign.conf

      *.infra.example.com
      
    • site.pp
      vim /etc/puppet/manifests/site.pp

      # /etc/puppet/manifests/site.pp
      
      import "nodes"
      
      # The filebucket option allows for file backups to the server
      filebucket { main: server => 'puppetmaster.infra.example.com' }
      
      # Set global defaults - including backing up all files to the main filebucket and adds a global path
      File { backup => main }
      Exec { path => "/usr/bin:/usr/sbin:/bin:/sbin" }
      
    • modules.pp
      vim /etc/puppet/manifests/modules.pp

      # /etc/puppet/manifests/modules.pp
      
      import "dns-client"
      import "ntp-client"
      
    • nodes.pp
      vim /etc/puppet/manifests/nodes.pp

      # /etc/puppet/manifests/nodes.pp
      
      node 'common' {
              include dns-client
              include ntp-client
      }
      
      node default inherits common {
      }
      
      node 'puppetmaster.infra.example.com' inherits common {
      }
      

    With the above you obviously need to have the dns-client and ntp-client modules in place in /etc/puppet/modules/ or update the config files to reflect the modules you have installed!

  9. Relabel all the various puppet directories to ensure we’re good with selinux:
    restorecon -Rv /etc/puppet
  10. Reboot to ensure it all works when it comes back online ;-)

P.S. I’ll update this post once I’ve had time to work out the SELinux issues, pointers welcome in the comments below :-)

Jan 27 14

SSH escape sequences

by nullr0ute

A lot of people don’t know that ssh has a number of escape sequences that can be run. I’ve often used the ~. to kill a stuck session but I recently discovered, by mistyping the aforementioned option, there is a number of other useful options too:

Supported escape sequences:
  ~.  - terminate connection (and any multiplexed sessions)
  ~B  - send a BREAK to the remote system
  ~C  - open a command line
  ~R  - Request rekey (SSH protocol 2 only)
  ~^Z - suspend ssh
  ~#  - list forwarded connections
  ~&  - background ssh (when waiting for connections to terminate)
  ~?  - this message
  ~~  - send the escape character by typing it twice
(Note that escapes are only recognized immediately after newline.)