Overview of aarch64 SBC support in Fedora 27

Support for ARM 64 bit (aarch64) Single Board Computers (SBCs) has been one of the most highly requested features along side the Raspberry Pi. It’s something I’ve been working towards almost as long too. Finally with Fedora 27 I felt we would have enough of the bits in place for a Minimum Viable Product (MVP).

You’ll note from the Fedora change linked above I was very cautious with what we planned to achieve. The change has a very focused list of images: Server, Workstation and Minimal and a limited list of devices: basically the Raspberry Pi 3, the 96boards Dragonboard 410c and HiKey, and a handful of AllWinner devices with a focus on the Pine64 series of boards. The reason for this was I knew there was going to be a lot of low level boot and kernel bits that needed focus and polish and the Fedora 27 cycle was severely limited time and resource wise so the plan was to focus on getting all the core bits into place for Fedora 27 and have a couple of well polished devices and then expand that rapidly for Fedora 28.

The key functionality we were aiming for was a well polished uEFI implementation in u-boot to enable a single install/boot path in Fedora on aarch64 using uEFI/shim/grub2 to boot Fedora on both SBCs and SBSA compliant aarch64 platforms. We now have that platform in place, primarily due to Herculean efforts of Rob Clark and Peter Jones, as well as many others who have provided insight into the deep dark details of the uEFI specification. Fedora 27 will ship with a quite heavily patched, well by Fedora’s standards anyway, u-boot 2017.09 which provides us the core of this functionality enabling us to use a vanilla upstream shim and grub2 to boot a standard Fedora. All this work is already upstream, or making it’s way there in 2017.11. In Fedora 28 there will be even more improvements that will enable us to do a bunch of other cool stuff (that’ll be a post for later!) and also enable much quicker upstream board enablement now all the core bits are in place.

So what do we actually support? Well all the usual bits that you would expect on a standard Fedora install, whether it be x86_64, ARMv7 or aarch64, like SELinux, containers, desktops and all the other bits. There’s a few bits and pieces that are a little rough around the edges but overall the feature is pretty robust. On a board by board feature set lets break the this down across the boards:

Raspberry Pi 3

The support for the Raspberry Pi3 is the equivalent to the ARMv7 support but with boot via uEFI/grub2. The memory isn’t quite as good as on 32-bit but that’s to be expected, overall it’s pretty reasonable for a device of the specs and cost. Like on 32 bit support we’re seeing regular improvements each release and throughout the releases. The aarch64 support for the RPi3 is just an evolution to this.

DragonBoard 410c

The support for the DragonBoard 410c is looking pretty decent. Qualcomm has been doing a pretty decent effort to get stuff upstream, we have firmwares for the GPU and for video decode/encode upstream as well, along with kernel drivers and the open freedreno 3D drivers, HDMI audio should work as well. The WiFi firmware isn’t yet upstream but I’ll document how/where to get that and hopefully that should be in linux-firmware soon as well. Overall I’m quite happy with the status of this device, although like all devices with 1Gb RAM it’s a little constrained, but that should make the newly announced 820c with 3Gb of RAM a decent device ;-). All the details for getting it running will soon be in the Fedora 96boards wiki page.

HiKey

Most features and functionality of the HiKey are supported, note this isn’t the HiKey960 (look to F-28 for support for that), except accelerated graphics due to the use of a MALI GPU. Other than that the functionality is pretty decent. You’ll likely want the latest tianocore firmware and the details for that can be found on the Fedora 96boards wiki page.

Pine64 (AllWinner A64 SoC)

We actually should have a number of devices based on the AllWinner A64 SoC working here but we’ve only tested the 3, 2Gb/1Gb/512Mb, Pine64 device sizes. The support for these devices is headless and you will need a serial console else you’re on your own as none of the display bits in the kernel have made it upstream, and of course the GPU is a MALI 400 series so when it does it won’t be fast. The support for the rest of the device is basic, it’s usable for a headless server style device, we support network, USB, KVM, RTC and a few other bits. Other than display we don’t yet support the SDIO attached wireless, sound, crypto offload or any of the other media interfaces. A lot of this is under review upstream so I think Fedora 28 should look much better for this series of devices and 4.15 might even bring very basic console output. Speaking of series of devices which ones should actually work other than the three Pine64 devices? Well the following A64 SoC devices have a Fedora built u-boot and kernel DT support so should work as well as the Pine64: BananaPi-m64, OrangePi Win, SoPine baseboard (PineBook boots if you’re happy with serial console), NanoPi-A64 and the A64-OLinuXino. We had some troubles with the AllWinner H5 SoC devices earlier in the cycle but I’ve had a couple of reports that it seems to be resolved so they should work too and that adds the Orange Pi PC2, Prime and Zero+ 2 as well as the NanoPi NEO2. So that’s around a dozen or so devices! 🙂

Other ARM64 SBCs

I’ve had reports that other aarch64 SBCs boot on Fedora just fine. I’ve not listed those where I can’t verify whether they boot with our uEFI enabled u-boot. Looking around on my desk I do have a number of devices that I expect us to be supporting in Fedora 28, or maybe even just enabling u-boot bits in a F-27 update.

Overall I’m pretty happy with the state of Aarch64 SBCs for Fedora 27 and what we’ve managed to achieve is such a short cycle!

Why I’m not backing the Purism Librem 5 phone

NOTE: This is a post about my opinion on the device, hence the title of “Why I’m not backing….”, people have been explicitly asking me why I’ve not backed it, this documents it so I don’t need to keep repeating myself!

Numerous people have come up to me and asked “So will you get Fedora to run on your Librem phone?” and when my response is “No, I’ve not backed it” I get weird looks with a question of “Why?” I had thought it was time to do document my concerns with this laudable venture. This was certainly further qualified when I had to inform someone from the EFF that they’re delirious about the “it doesn’t need closed source firmware” on the i.MX chips that are being proposed. While I applaud the general principals and ideas of a fully open rights protecting phone I can’t help but feel that the group doing it either are being false at worst or naive at best with some of their statements.

Firmware

Their site claims “The i.MX 6/8 CPU will be completely free software without any binaries whatsoever!” while this could be sort of true if you want a hobbled device it’s not really the case at all. There’s a number of firmwares that are needed to make a number of pieces of functionality of a mobile device useful. Firstly the SDMA driver needs a firmware to run at any level of reasonable speed. Secondly the accelerated media decode, which will be required if you actually want to consume meda on your phone and have more than moments of battery life, also needs firmware. You’ll note on the media decode I don’t reference the actual firmware! Why? Because it’s not actually distributed in the linux-firmware repository so you have to request it from NXP or somewhere with appropriate signups (while I’m not 100% sure I believe having a driver in the linux kernel without required firmware in linux-firmware is a breach of the requirements of said driver in the upstream kernel).

i.MX6 or i.MX8 SoC

The i.MX 6 or i.MX 8 option concerns me. Basically make a choice! The core issue I have is the i.MX6 SoC is ancient, being announced back in Jan 2011, and based on a Cortex-A9 SoC. It doesn’t support USB-C so to ship as promised they’ll need to add a PCI-e attached USB-3 controller, charging circuitry, and maybe even a LVDS to Display Port option plus a chip to MUX the three through the actual USB-C port if they want to be able to ‘dock’ which will make power consumption on the phone even worse! If I’d spent $600 and received a phone in January 2019 based on an eight year old 32-bit chip design I’d be seriously pissed off. To be quite honest the i.MX8 is no spring chicken either, being announced in September 2013, it’s based on the already quite long in the teeth Cortex-A53 SoC design which was the original aarch64 “little” low end design but it does at least support USB-C on the SoC. The i.MX SoCs are generally quite nice, and the i.MX6 is quite well supported upstream, but the whole line of SoCs are more targeted towards embedded applications like cars, so they do have a long support cycle. They’re not a mobile phone focused SoC though, they also tend to be quite slow to get moving in the market/availability and the i.MX8 has little upstream support in the kernel as yet. Sure the etnaviv driver enables an open 3D accelerated driver, one that isn’t supported by the vendor so doesn’t enable all the features upstream, but you can run this today on Fedora 26/27 on the i.MX6 SoCs now but a free GPU driver is not the only reason to choose a platform.

General concerns

I can’t help but feel that there’s going to be a lot of disappointed people that will end up receiving an expensive sub standard device, probably late, that ends up not taking us much further along the road, to a fully open rights protecting phone, than the days of the Nokia n9xx series phones running Maemo. The HW, if based on an i.MX6, will certainly not be much further along that route and still stuck in the past in terms of HW. I personally believe the project would be better off engaging with the Qualcomm community team to use the Qualcomm 820c/600c SoCs because there’s an open driver that Qualcomm are working to improve, and while there’s a need for firmware for GPU and media offload, in reality it’s no better/worse than the i.MX devices with the bonus that their devices that are more current and aimed at mobile workloads with the vendor actively working to upsteam the enhance the support support of their SoCs.

Ultimately I think a device that closely resembles the specs of a phone of recent history, than that of ancient smart phone history, is likely to get a better following and hence a better software ecosystem than one that’s the same era as the Motorola DROID 4.

Update: So just to clarify a few things that were bought up in a couple of threads:

First this reddit thread:

  • I don’t question the value of hardware isolation of the various wireless interfaces that they claim, or the ability of them being able to deliver that bit on what ever SoC they choose, that is why it wasn’t addressed above.
  • I bought up USB-C because it is explicitly claimed as a feature.
  • I bought up display port because given USB-C above and their “It can be a desktop computer and phone all-in-one” claim, including a nice picture, then DP over USB-C is the only real option for that functionality.
  • Qualcomm chips were an example of other SoCs I believe might be a better fit, yes they do provide options in their APQ line without built in radios, they were just an example of another option, there are other possibilities, it wasn’t meant to provide a guarantee, it was an alternative example.
  • Yes, all ARM processors have onboard boot firmware, it’s generically referred to as “PBL” (Primary Boot Loader), so do most other processors.

Second this purism thread:

  • Sure you can disable the media engine with an e-FUSE but media is kind of useful for a lot of use cases like video config, audio calls and music, not just watching videos. If they choose this route, I would hope they leave the fuses unblown and document how a end user can do it else the $600 is a whole lot less useful for a lot of people

Ultimately the main points is I have to make about all of the above is two fold, firstly it’s my opinions and why I didn’t back the device, and secondly there appears to be some large discrepancies in the statements they make about the device and SoCs which I would not have expected and that is the key problem for me because it causes concerns over the ability to deliver a working and useful device to me. $600 is not a small amount of money, for me at least, to hedge on what *might* be a i.MX8 or might be an old 32 bit i.MX6.

The state of open source accelerated graphics on ARM devices

I’ve been meaning to write about the state of accelerated open source graphics options for a while now to give an update on a blog post I wrote over 5 years ago in January 2012, before the Raspberry Pi even existed! Reading back through that post it was pretty dark times for any form of GUI on ARM devices but with the massive changes in ARM devices and the massive change in SBCs (Single Board Computers) heralded by things like the Raspberry Pi have things improved at all? The answer is generally yes!

The bad

Looking back at that post the MALI situation is still just as dire with ARM still steadfastly refusing to budge. The the LIMA reverse engineering effort started with promise, but went up in smoke with a fairly public community break down, I don’t envision that situation improving any time soon although just recently there appears to be some forward movement happening finally after a long silence. This only covers the MALI-400 series and any newer GPU is a completely different architecture/IP. Even with sessions recently at Linaro Connect titled What’s happening with ARM Mali drivers I don’t see fast change here.

The Imagination Technologies PowerVR is still just as dire as situation as it was five years ago. The company’s incompetent management recently managed to avoid being bought by Apple which in turn, because they’ve screwed the open source community while milking the Apple cash cow, essentially means they’re screwed. I suspect they’ll either open source to try and remain a relevant contender or die in a tire fire. Only time will tell there, in the mean time any ARM SoC that has this IP on board is useless for anything graphical so I’d tend to avoid it, thankfully there seems to be less of them these days.

The good

Despite the two bad examples above there’s actually been a lot of good change in the last five years. We now have a number of options for fully accelerated 2D/3D graphics on ARM SoCs and I run GNOME Shell on Wayland, yes the full open source shiny, on a number of different devices regularly.

NVIDIA true to the rumours did open up all the graphics on the Tegra series of hardware. The new Tegra K/X series have GPUs similar to their x86 offerings with Kepler/Maxwell/Pascal GPU cores but NVIDIA supports these devices by contributing to the nouveau open driver rather than the closed x86 driver. The performance on 32 bit TK1 devices has been decent for a number of releases of Fedora and improves all the time, we’ll be supporting the X series (X1/X2) with their Maxwell/Pascal GPUs in Fedora 27.

In the old post I brushed past Vivante with a mere mention of Marvell and Freescale (now NXP). The Vivante GPUs ship in NXP i.MX6 and i.MX4, some Marvell chips and some TI chips. There was a reverse engineering effort called etnaviv that must have started not long after I wrote that post and after a number of years of development support landed upstream in the kernel late 2015, and in mesa in the 7.1 release allowing us to support fully accelerated Wayland in Fedora 26! Did anyone notice? I didn’t really yell about it as much as I should have! It supports fully accelerated 3D in mesa/wayland, is pretty stable and is improving all the time, well done to all the contributors to that effort!

Another I brushed past in the old post was the Qualcomm Snapdragon SoC. They ship with a Adreno GPU. This was previously closed source, with the SoC primarily used by phone/tablet manufacturers I suspect they didn’t care… until Rob Clark (and no doubt there were other contributors) decided to reverse engineer the driver with the open freedreno driver. This is now the default driver with even Qualcomm contributing to it. We’ll support this in Fedora 27, initially with the 96boards Dragonboard 410c using the freedreno driver, but I doubt it’ll be the last Qualcomm based device we support. The Snapdragon 835 SoC, the device in all the high end Android phones this year and the ARM Windows 10 laptops, is really nice with decent performance, I’d love to be able to support a device with that SoC!

Raspberry Pi, as I mentioned in the introduction, wasn’t even out in when I wrote the original post. When it fist launched there wasn’t an open driver but 5 years later there is, sponsored by Broadcom no less. We introduced initial support for the Raspberry Pi with the open vc4 driver by Eric Anholt in Fedora 25 and it’s improving regularly. It supports fully accelerated 3D in mesa/wayland, and 2D via glamor in mesa.

So in conclusion we have improved by A LOT! We now have numerous different GPUs with open drivers to choose from in all price ranges that support fully accelerated 2D/3D desktops from four different vendors on both ARMv7 and aarch64. The media acceleration offload is also looking quite good, but that’s one for another post. The biggest holdout is MALI, and that would need two open drivers or ARM to come to the table, LIMA might work out for the 400 series, but that won’t work on the newer midguard series. With support in a number of drivers for the shiny new Wayland there’s an increasing number of devices people can use to enjoy the latest desktops fully accelerated!

Raspberry Pi improvements in Fedora 26

So since I landed support for the Raspberry Pi 2 and 3 just in time for Fedora 25 Beta it’s been a bit of a fun ride. The support for Raspberry Pi is mostly done in my spare time along side all the other responsibilities I have and it’s been interesting to see people’s feedback. Going into Fedora 25 I knew it wasn’t going to be perfect but the experience was going to be reasonable for newbies to get going without generally needing serial consoles and it met Fedora’s (and mine) exacting standards on free drivers. I think we achieved that quite well but I also learned a lot in the Fedora 25 cycle and what’s coming in Fedora 26 is quite a substantial jump forward.

Hardware for a good experience

So what have I learned about the first six months or so of Raspberry Pi in Fedora? Well there’s a couple of things that the user can do to ensure a decent starting experience themselves. The biggest FAQs I’ve dealt with on the various support forums are generally fixed by these three things:

  • A proper spec power supply. For the RPi2 this means at least 2 AMPs and for the RPi3 at least 2.5 AMPs. If you want to plug in USB WiFi dongle and a USB HDD you’ll likely want to add a little more! In most cases an old phone charger will not suffice.
  • A good quality Class 10 micro SD card. I generally use Samsung EVO or SanDisk Ultra cards.
  • A Raspberry Pi 2 or 3. Yes, it’s surprising how many people hope to run it on something else. SORRY (actually, I’m not!)!

What’s in Fedora 26 Final

So enough of what to do! Everyone wants to know what improvements arrived in the Fedora 26 Final with the 4.11.x kernels:

  • Pi3 WiFi: It’s been working in F-26 since Alpha and is surprisingly stable. There’s a file you need to grab to enable it. See details in the wiki here.
  • Performance: In the process of dealing with wifi I worked out one of the reasons we were seeing poor performance on the SD card. We’ve had some minor improvements in F-25 but this fix over doubles the performance for me on the SD card.
  • HDMI video: There’s been issues around certain monitors crashing the video (vc4) driver and people getting black screens during boot. While this isn’t perfect yet (ain’t hardware great!!) it’s greatly improved across numerous devices.
  • Composite video: We’ve had support for the composite video since 4.10 but I need people to help test this.
  • Sound: HDMI audio is supported, I’ve done minor testing with the one HDMI audio capable device I have. Analogue audio out isn’t upstream yet.
  • HAT support: We now have all the support needed to do overlays in the kernel/bootloader and dtc stack. I just need to test it some more, document it and work out how we can best distribute pre built overlays to ease consumption. There’s still no consensus on an Overlay Manager from upstream to auto load overlays based on EEPROM on the HATs. In a lot of cases you want to load the overlays from u-boot anyway for things like display. Look out for docs and blog posts on this soon!

What arrived with the 4.12 kernel rebase

  • Thermal support: so if the RPi runs too hot it’ll slow it down
  • More performance improvements and tweaks.

What’s coming in the 4.13 kernel rebase

  • Bluetooth support: upstream finally tracked down the issues here. It’s been a much requested features and I should have the bits in place soon!
  • More performance, stability and graphics improvements and tweaks.

What about Fedora 25?

Some of the above pieces will be coming to Fedora 25 with the 4.12 rebase. The focus of my spare time is Fedora 27 mostly now, with the above coming to F-26. Some components are a lot harder to back port without issues or a complex series of package updates to ensure smooth upgrade. The WiFi and performance improvements were the hardest as part of that change moves around the use of hardware blocks and drivers. I managed to stop both the RPi2 and RPi3 booting numerous times in testing before I properly realised the implications of the change. Getting these changes for users back into a stable release without issues is hard and time consuming to do across all the various use cases. I tried this with some fixes in 4.9 and ended up making the RPi3 very unstable. This cost me a lot of time to debug and fix and I don’t really want a repeat of that!!

Graphics device

One of the surprising side effects was the discovery of a device that is five years old is that Fedora suffered from early adopters issues. We were one of the first distributions to adopt a fully upstream open kernel and graphics stack and with that came a number of issues around monitor detection, especially older/cheaper models that aren’t 1920/1080 “Full HD” or via HDMI to VGA adapters. We’re still working through these with upstream and have improved the situation quite a bit in Fedora 26 overall but it takes time and reproducible use cases which with random hardware isn’t easy or quick! 🙁

Next up?

I’ll leave Fedora 27 features and functionality for another, this post has been sitting in my drafts folder since June so it’s time to get it out and like my development move on to Fedora 27!

Getting started with Zephyr on Fedora

So while Fedora is great for a lot of IoT use cases it can’t be used everywhere, such as on tiny micro controllers such as an ARM Cortex-M series or Intel Quark micro controllers, but that doesn’t mean that Fedora doesn’t make a fantastic developer platform for working with these devices.

I have a handful of Zephyr capable devices (BBC Micro:bit, NXP FRDM-K64F, 96Boards Carbon, TI CC3200 LaunchPad) so how can you get a build environment up and running quickly so you can start doing real development as quickly as possible.

In testing this I used a Digital Ocean cloud instance for a build host. Wherever you choose to build it make sure you have at least 2GB of RAM available as from my experience you need at least 2GB for building a Zephyr image.

From there we diverge a little from the upstream notes by installing the Fedora ARM cross compiler (only tested with ARM, not sure of state of other targets) and developer tools:

sudo dnf install git-core gcc gcc-arm-linux-gnu glibc-static libstdc++-static make dfu-util dtc python3-PyYAML

Next up we clone the upstream Zephyr git repository:

git clone https://gerrit.zephyrproject.org/r/zephyr zephyr-project

If we want to use a particular stable branch we now switch to the chosen branch. I’m using the latest stable release branch:

cd zephyr-project; git checkout v1.7-branch

Set up the cross compiler variables:

export GCCARMEMB_TOOLCHAIN_PATH="/usr"
source zephyr-env.sh
cd $ZEPHYR_BASE/samples/hello_world

Select and build our target:

make CROSS_COMPILE="/usr/bin/arm-linux-gnu-" DTC=/usr/bin/dtc BOARD=96b_carbon

If we’re developing this on our local machine we can now just directly flash the new build straight to the device. To do this we connect a micro USB cable to the USB OTG port on the Carbon and to your computer. The board should power on. Force the board into DFU mode by keeping the BOOT0 switch pressed while pressing and releasing the RST switch.

Confirm DFU can see the device:

$ sudo dfu-util -l
dfu-util 0.9

Copyright 2005-2009 Weston Schmidt, Harald Welte and OpenMoko Inc.
Copyright 2010-2016 Tormod Volden and Stefan Schmidt
This program is Free Software and has ABSOLUTELY NO WARRANTY
Please report bugs to http://sourceforge.net/p/dfu-util/tickets/

Found DFU: [0483:df11] ver=2200, devnum=8, cfg=1, intf=0, path="2-1", alt=3, name="@Device Feature/0xFFFF0000/01*004 e", serial="123456789"
Found DFU: [0483:df11] ver=2200, devnum=8, cfg=1, intf=0, path="2-1", alt=2, name="@OTP Memory /0x1FFF7800/01*512 e,01*016 e", serial="123456789"
Found DFU: [0483:df11] ver=2200, devnum=8, cfg=1, intf=0, path="2-1", alt=1, name="@Option Bytes  /0x1FFFC000/01*016 e", serial="123456789"
Found DFU: [0483:df11] ver=2200, devnum=8, cfg=1, intf=0, path="2-1", alt=0, name="@Internal Flash  /0x08000000/04*016Kg,01*064Kg,03*128Kg", serial="123456789"

Flash our build onto the device:

sudo dfu-util -d [0483:df11] -a 0 -D outdir/96b_carbon/zephyr.bin -s 0x08000000

Now connect another micro USB cable to the UART port and run a console:

sudo screen /dev/ttyUSB0 115200

Hit the reset button and you should see the following output:

***** BOOTING ZEPHYR OS v1.7.1 - BUILD: Jun  6 2017 14:07:24 *****
Hello World! arm

Now we have a basic development environment setup, know we can build, flash and run a release on the 96boards Carbon next time we can do something more advanced 😉

Update (2017-06-13): Minor updates to dependency installs and make command

Updating Raspberry Pi firmware on Fedora

The upstream Raspberry Pi firmware/bootloader gets regular updates and improvements. In Fedora we ship that firmware in a package called bcm283x-firmware. I regularly follow the git repo of the upstream firmware and on occasion when I believe there’s reasonable changes that benefit Fedora I’ll prepare a new version, do some brief testing on my devices to make sure it boots and basic functionality hasn’t regressed at which point I’ll update the package and send it out to supported releases as an update.

Once the new bcm283x-firmware lands on your Raspberry Pi it doesn’t automatically update the firmware though. Why is that you ask? I don’t like to spring surprises on people where they end up with a device that might not boot or it might regress things they care about.

So how do you upgrade the firmware for the Raspberry Pi on Fedora? It’s simple! You simply run the command rpi-firmware-update and it’ll update the firmware and the u-boot to the latest one that’s shipped as a Fedora package. Then you just need to reboot to make it active.

The easiest way to work out which firmware you’re currently running is “dmesg | grep raspberrypi-firmware”

I tend to try and push out a new firmware update every month or so but if I see something that’s of interest or that fixes known issues I do it as needed.

Connect to a wireless network using command line nmcli

I use a lot of minimal installs on various ARM devices. They’re good because they’re quick to download and you can test most of the functionality of the device to ensure it’s working or to quickly test specific functionality but of course it doesn’t have a GUI to use the nice graphical tools which are useful to quickly connect to a wifi network or other things.

This where nmcli comes in handy to quickly do anything you can do with the GUI. To connect to a wireless network I do:

Check you can see the wireless NIC and that the radio is enabled (basically “Airplane” mode):

# nmcli radio
WIFI-HW  WIFI     WWAN-HW  WWAN    
enabled  enabled  enabled  enabled 
# nmcli device
DEVICE  TYPE      STATE         CONNECTION 
wlan0   wifi      disconnected  --         
eth0    ethernet  unavailable   --         
lo      loopback  unmanaged     --         

Then to actually connect to a wireless AP:

# nmcli device wifi rescan
# nmcli device wifi list
# nmcli device wifi connect SSID-Name password wireless-password

And that should be enough to get you connected. You can list the connection with nmcli connection and various other options. It’s pretty straight forward. The only complaint I have is that it doesn’t prompt for a password if you leave it out so it means the AP password is stored on the command line history. Not a major given it’s quite easy to find where it’s stored anyway on the system but it would be a useful addition.

Getting started with MQTT using the Mosquitto broker on Fedora

MQTT is a lightweight publish/subscribe messaging transport designed for machine-to-machine “Internet of Things” connectivity. It’s been used in all sorts of industries from home automation and Facebook Messenger mobile app to health care and remote monitoring over satellite links. Its ideal for mobile apps and IoT because of it’s small footprint, low power usage and small data packets. You can connect to a MQTT Broker over standard TCP/IP ports with or without TLS or over Web Sockets. There are many cloud based MQTT services such as OpenSensors, CloudMQTT, HiveMQ, AWS IoT or Azure IoT Hub but where is the fun in that?

There’s a few message brokers that support MQTT in Fedora including RabbitMQ or ActiveMQ (the version in Fedora is old but it would be nice if someone did update it to the latest version!) but for this I’ll be using Mosquitto as it’s small and relatively straight forward for home and demo usage.

Initial Install
I’ve tested this on a Digital Ocean Fedora 23 VM but for this I’ll be using a Beagle Bone Black with a Fedora 24 Beta minimal image but of course you can run it on any recent Fedora release on any supported device, ARM or otherwise.

After installing Fedora and getting it running on the network install and configure mosquitto:

  dnf install -y mosquitto screen
  mv /etc/mosquitto/mosquitto.conf.example /etc/mosquitto/mosquitto.conf
  firewall-cmd --permanent --add-port=1883/tcp
  firewall-cmd --permanent --add-port=1883/udp
  firewall-cmd --permanent --add-port=8883/tcp
  firewall-cmd --permanent --add-port=8883/udp
  systemctl enable mosquitto.service
  systemctl start mosquitto.service

By default the default settings are relatively straight forward, it runs under the mosquitto user, listens on all interfaces, with config options in /etc/mosquitto. In there is a well documented example config file called mosquitto.conf.example which contains all the default settings. Above we rename it to mosquitto.conf to use as a base config for us to modify. There’s details on using the Let’s Encrypt service on the mosquitto site. If you’re using it on the public internet I’d strongly suggest you configure TLS.

Testing it works
Now that we have a basic install done lets make sure it works and get a basic idea of how MQTT works. Included in the mosquitto package is a couple of client utilities useful for testing the MQTT broker pub/sub functionality. Start up a screen session and in a window run the following commands which subscribes us to the “test/topic” topic to the MQTT broker running on localhost:

  mosquitto_sub -h localhost -t "test/topic" 

Now lets publish some data to that topic. In another screen tab run the following command which will publish a string of text to the topic:

  mosquitto_pub -h localhost -t "test/topic" -m "Hello World!"

You can run multiple subscribers either from the screen session or remote from a different host but remember to change the “-h localhost” to appropriate hostname/IP if you’re testing remotely.

MQTT can also use two types of wild cards when subscribing to topic names. So to be able to read all temperature sensors publishing to a root topic you could use “home/+/temp” which would provide “home/bedroom/temp” and “home/livingroom/temp”. The “+” will match one hierarchical level where as “#” stands for everything deeper so “home/garden/#” would provide all sensors from the garden. When using wild cards you’ll need to know which topic is reporting so use mosquitto_sub in verbose mode:

  mosquitto_sub -h localhost -v -t "test/#"

Quality of Service
One of the killer features of MQTT for low power IoT sensor networks is that it’s a “store and forward” protocol with a couple of options for quality of service (QoS) guarantees. Once the node has registered with the broker any time it sleeps the broker will queue messages and take care of delivering that message to the node as soon as it reconnects. This means the node can sleep as long as needed to preserve battery life and just wake up and connect when it has data to send or at a predetermined interval without ever missing a message.

There’s three things needed to support QoS: 1) The subscriber needs to be registered by ID with the broker 2) The subscriber needs to disable clean-slate behavior (the default is enabled) 3) Both the publisher and subscriber need to be using quality-of-service level one or two so the broker knows it needs to store the messages. More details on QoS levels are covered here.

Here’s a small demo to play with QoS basics. First connect a subscriber to the broker supplying an ID, disable clean slate and enabling QoS level 1:

  mosquitto_sub -h localhost -v -t "test/#" -c -q 1 -i "PotPlant"

Then send the following two messages, one each with and without QoS. You should see them both show up on the subscriber:

  mosquitto_pub -h localhost -t "test/topic" -m "foo" -q 1
  mosquitto_pub -h localhost -t "test/topic" -m "bar"

Now stop the subscriber (Ctrl+c) and while it’s stopped send both of the messages again before restarting the subscriber. When it reconnects you should just receive the “foo” message because it’s the only one specifying a QoS level.

Further Reading
There’s a lot of information about MQTT on the web. The MQTT Essentials page from HiveMQ is a good place to start for further information and Awesome-MQTT is a curated list of MQTT related stuff with everything from language bindings to gateways/plugins for integration with various projects whether they be home audio or enterprise products.

Fedora 24 Alpha for aarch64 and POWER

So Fedora 24 Alpha is out for aarch64 and POWER. Keen followers will note that we were a couple of days behind the primary architecture’s Alpha release, which hasn’t been the case for the last few Fedora cycles where we’ve generally released on the same day.

The primary reason for the delay was the Pungi Refactor. While the pungi 4 change has been massive for primary architectures for the secondary architectures it’s the single biggest change to our release process EVER! Basically we’ve thrown the lot out and started again. When I started in release engineering over 18 months ago the number one goal that was set for me can be summarised as “Be more like primary. Make the whole secondary architecture as close to primary as possible!” and we’ve been continuously moving, albeit not as fast as I would have liked, in that general direction. With the arrival of pungi 4 for Fedora 24 we’re almost at that end goal in terms of the current way we do secondary architectures.

With Fedora 24 we’re also adding a lot more release engineering focused features and functionality to the secondary architectures. We have now have full nightly composes on rawhide and branched whereas previously we’d just produce a “Everything” repo. This allows ongoing continual testing on things so it’s easier to know when things regress. On PowerPC we’ve produced qcow2 cloud images to some degree since Fedora 22 but it was a bit of a manual process. These are now fully integrated into the pungi/koji process and, like on primary produced nightly, similarly they’ll be coming to aarch64 very shortly too. In Fedora 24 we’ve added Docker base images, they’re produced nighly on branched and rawhide for PowerPC now, and will be nightly for aarch64 at the same time the qcow2 cloud images arrive. Finally aarch64 will also soon have disk images like ARMv7 on primary to enable us to easily support the new shiny aarch64 Single Board Computers (SBCs) that are _FINALLY_ becoming available for the architecture, for Fedora 24 it’ll be a bit of a hack, but with Fedora 25 both ARMv7 and aarch64 will be able to move to koji based live-media-creator image build process but I’ll outline more of that in another post.

So the pungi refactor has been big for the secondary architectures. It’s required big changes in our infrastructure which is now mostly complete, there’s a few infrastructure cleanups and final changes that are in process, these will be done in the next few weeks in the lead up to Beta. We have a single host left to migrate to ansible (YAY!!) and some final moving around of resources. We’ll be changing the way we sync content out to mirrors too which will close out one of the final deltas of the rel-eng secondary process. Overall the last few weeks have been challenging getting all the bits in place, but by the time we hit Beta it’ll all be complete! The new processes lay the foundations for the secondary architectures to add functionality quicker than ever before, and by being almost identical to primary the “onboarding” of new people to use that process, or end users be able to consume the output of the rel-eng process is easier than ever before and that makes me happy! 🙂

Lipstick on a Pig AKA the Raspberry Pi 3

So while waiting for local scratch kernel builds for much more interesting devices I started looking around to see if I could find details of the kernel sources for the new BCM2837 SoC that is centre stage in the Raspberry Pi 3.

The problem is I couldn’t. What I did find is the hack the Raspberry Pi Foundation uses to boot the RPi3 on github.

So there is no source code release for the new BCM2837 SoC, just a device tree file. Someone said to me “They’re violating the GPL” and before people get out their pitch forks… they’re NOT because this is the code they ship, they are meeting their obligations there.

So for the lay person (yes, I know there’s a lot of deep level tech details I’m glossing over deep ARM architecturey people!!) basically what they are doing is booting this device as a ARMv7 device, and because the code isn’t built for ARMv8 (32 or 64 bit) they really just get the speed bump of a ARMv7 device running a bit faster, and possibly some better memory speeds and other general improvements.

So what does this mean for other distributions that wish to actually to support the Raspberry Pi 3 as a aarch64 device? You currently can NOT do so!. Why? Basically it boils down to two things:

  • Source code release for the kernel: To be honest I don’t think this should be large. People with low level knowledge of ARMv7 and the BCM283x could probably hack this up
  • Firmware support: I suspect there will need to be a new firmware that supports booting this as a aarch64 device. I obviously don’t know for sure but I’m guessing the firmware will need changes to actually properly boot this as a aarch64 device. I’ve little doubt there’s a bunch of hackery going on in there!

Of the above two, if my theory is correct, the firmware is the problematic one because it relies on the Raspberry Pi Foundation to do the work. This work for something that they feel, at the moment, gives them no particular gain but only confusion about multiple OSes. They are of course correct for their use case, basically like old school enterprise where you buy a bigger server to scale vertically because your app won’t scale horizontally, but this is another kick in the guts of the Open Source community they so heavily rely on! Oh well, it’s about as much as I expected from the Raspberry Pi Foundation as after all their devices are only just now becoming usable with upstream kernels and open userspace GPU drivers…. after a mere four years.

So what does this mean for Fedora? Basically the only way we’ll be able to support it in the short, possibly medium, term is like it’s sibling the Raspberry Pi 2 as an ARMv7 device but with added shitty wifi. Really, this device isn’t a cheap aarch64 device, it’s just like lipstick on a pig! If a cheap aarch64 device is what you want one of those go and buy a PINE64.

On the plus side the work needed to support it as a ARMv7 device at the same time as it’s sibling should just be some minor u-boot and kernel device tree patches on top of what I previously documented . Note I’ve not looked closely at this as yet, I’m still waiting for mine to arrive (YAY day 3 of 1 day shipping)! Frankly I’d sooner support it this way, an aarch64 device with terrible USB2 IO and 1GB of RAM won’t provide much, if any, of a perf bump over ARMv7, and then have the Raspberry Pi Foundation spend their time working with Broadcom on fixing the wifi and enabling distribution of the wifi firmware in linux-firmware as proper opensource broadcom wifi support would have a wider impact on the Open Source community the Foundation relies upon!