Category: Fedora

WiFi on Raspberry Pi 3 for Fedora 26 Alpha

So I managed to land just about everything needed for the WiFi on the Raspberry Pi 3 for Fedora 26 Alpha (around 4.11 rc3). There’s one thing missing, because we can’t currently redistribute it, but it’s straight forward for the end user to do themselves once they’ve done the initial setup:

sudo curl https://raw.githubusercontent.com/RPi-Distro/firmware-nonfree/master/brcm80211/brcm/brcmfmac43430-sdio.txt -o /lib/firmware/brcm/brcmfmac43430-sdio.txt

Or you can also do it when you’re flashing the image if you mount the root filesystem but the above is likely easier. It’s been surprisingly stable in my testing.

Before you all ask, at the moment I don’t plan on pushing this to earlier Fedora releases, as the upgrade path is not trivial. I will also soon publish more details of some of the other new features coming for the Raspberry Pi to Fedora 26 but I thought you’d all like the WiFi details now. The wiki has also been updated to reflect the status of the WiFi.

PS: No this is not an April Fool’s joke (it’s well past midday in UK).

Updating Raspberry Pi firmware on Fedora

The upstream Raspberry Pi firmware/bootloader gets regular updates and improvements. In Fedora we ship that firmware in a package called bcm283x-firmware. I regularly follow the git repo of the upstream firmware and on occasion when I believe there’s reasonable changes that benefit Fedora I’ll prepare a new version, do some brief testing on my devices to make sure it boots and basic functionality hasn’t regressed at which point I’ll update the package and send it out to supported releases as an update.

Once the new bcm283x-firmware lands on your Raspberry Pi it doesn’t automatically update the firmware though. Why is that you ask? I don’t like to spring surprises on people where they end up with a device that might not boot or it might regress things they care about.

So how do you upgrade the firmware for the Raspberry Pi on Fedora? It’s simple! You simply run the command rpi-firmware-update and it’ll update the firmware and the u-boot to the latest one that’s shipped as a Fedora package. Then you just need to reboot to make it active.

The easiest way to work out which firmware you’re currently running is “dmesg | grep raspberrypi-firmware”

I tend to try and push out a new firmware update every month or so but if I see something that’s of interest or that fixes known issues I do it as needed.

Connect to a wireless network using command line nmcli

I use a lot of minimal installs on various ARM devices. They’re good because they’re quick to download and you can test most of the functionality of the device to ensure it’s working or to quickly test specific functionality but of course it doesn’t have a GUI to use the nice graphical tools which are useful to quickly connect to a wifi network or other things.

This where nmcli comes in handy to quickly do anything you can do with the GUI. To connect to a wireless network I do:

Check you can see the wireless NIC and that the radio is enabled (basically “Airplane” mode):

# nmcli radio
WIFI-HW  WIFI     WWAN-HW  WWAN    
enabled  enabled  enabled  enabled 
# nmcli device
DEVICE  TYPE      STATE         CONNECTION 
wlan0   wifi      disconnected  --         
eth0    ethernet  unavailable   --         
lo      loopback  unmanaged     --

Then to actually connect to a wireless AP:

# nmcli device wifi rescan
# nmcli device wifi list
# nmcli device wifi connect SSID-Name --ask

And that should be enough to get you connected. You can list the connection with nmcli connection and various other options. It’s pretty straight forward.

When creating updates remember to build for rawhide and Fedora 25 (devel)

When ever we branch for a new release of Fedora I, and others, end up spending a non trivial amount of time ensuring that there’s a clean upgrade path for packages. From the moment we branch you need to build new versions and bug fixes of packages for rawhide (currently what will become Fedora 26), for the current stabilising release (what will become Fedora 25) as well as what ever stable releases you need to push the fix for. For rawhide you don’t need to submit it as an update but for the current release that’s stabilising you do need to submit it as an update as it won’t just automagically get tagged into the release.

As a packager you should know this, it’s been like it for a VERY LONG TIME! Yet each cycle from the moment of branching right through to when a new release goes GA I still end up having to fix packages that “get downgraded” when people upgrade between releases!!

So far this cycle I’ve fixed about 20 odd with the latest being bash-completion (built but not submitted as an update for F-25) and certmonger (numerous fixes missing from F-25 and master branch).

The other silly packaging bug I end up having to fix quite a bit is NVR downgrades where even though it’s a newer package the way the NVR is handled makes rpm/dnf/yum think the newer package is a lesser version than the current version and hence you’re new shiny fix won’t actually make it to end users. I see this a lot where people push a beta/RC package to a devel (F-25/rawhide) release. Just something to be aware of, there’s lots of good docs around the way rpm/dnf/yum handles eNVR upgrades.

Fedora 24 released on all architectures simultaneously!!

So for the first time ever we’ve released Fedora 24 across both primary and alternate architectures pretty much simultaneously! That’s the three primary architectures, x86_64, ARMv7 and i686, plus the alternate architectures of aarch64, ppc64, ppc64le and s390x. This is the first time we’ve ever released SEVEN architectures on the same day!

Fedora 24 from a Release Engineering perspective has been fairly momentous, we’ve made the single biggest change to our tooling that composes the release since Fedora Core and Fedora Extras was merged in Fedora 7! The plans for this change had been long discussed and first started to move back in the Fedora 21 cycle with the hope it could be implemented in Fedora 22…. boy were we wrong! But on the flip slide, while it’s still not perfect, it has massively improved the process for the releases. We now FINALLY have a full compose every single night! No more Test Composes! It now means QA can automate tests against any bit of the compose output, it means the installer team can see the results of their changes the next day. For the average user this has no material visible impact, for those much closer into the process it has a hugely positive impact!

From the alternate “secondary” architecture perspective I’ve ticked of a massive amount of “goal check boxes” that I made for myself when I joined this role almost two years ago.

My big ticket item was “make the release process like primary”. Our release process was a lot more manual than “primary” but we had also a lot less “features”. Well with this release neither are now true. We’re not 100% of the primary process, but the difference is tiny. We have a full nightly compose now, like primary, where previously we basically had a “newRepo” process we now have a full installer stack with network and iso installs produced. This in itself is a massive improvement! To this we add docker to aarch64 and Power64, cloud to aarch64 (and more automation of cloud to Power64), and initial “tech preview” disk images to aarch64 for Single Board Computers (I’m sorry I really did want to nail down this feature but even I have to sleep!) like the Pine64.

Some of the “smaller” tick box improvements to non primary arches is that we’re now fully 100% ansible run for the aarch64 and Power64 infrastructure, and only have a few minor bits to work out for s390x. This means from the back end everything is 100% like primary and orchestrated as such, the hubs all looks the same and the experience is consistent. Changes are deployed simultaneously and hence consistently. YAY automation! We’ve also simplified the way the secondary content gets to the mirrors so it’s more consistent and faster!

On the not x86 architecture front we’ve got a whole bunch more exciting features planned, and improvements to make, for the Fedora 25 cycle and beyond, some of which will begin happening very soon. Watch this space, the second half of the year looks to me to be just as frantic as the first!!

Getting started with MQTT using the Mosquitto broker on Fedora

MQTT is a lightweight publish/subscribe messaging transport designed for machine-to-machine “Internet of Things” connectivity. It’s been used in all sorts of industries from home automation and Facebook Messenger mobile app to health care and remote monitoring over satellite links. Its ideal for mobile apps and IoT because of it’s small footprint, low power usage and small data packets. You can connect to a MQTT Broker over standard TCP/IP ports with or without TLS or over Web Sockets. There are many cloud based MQTT services such as OpenSensors, CloudMQTT, HiveMQ, AWS IoT or Azure IoT Hub but where is the fun in that?

There’s a few message brokers that support MQTT in Fedora including RabbitMQ or ActiveMQ (the version in Fedora is old but it would be nice if someone did update it to the latest version!) but for this I’ll be using Mosquitto as it’s small and relatively straight forward for home and demo usage.

Initial Install
I’ve tested this on a Digital Ocean Fedora 23 VM but for this I’ll be using a Beagle Bone Black with a Fedora 24 Beta minimal image but of course you can run it on any recent Fedora release on any supported device, ARM or otherwise.

After installing Fedora and getting it running on the network install and configure mosquitto:

  dnf install -y mosquitto screen
  mv /etc/mosquitto/mosquitto.conf.example /etc/mosquitto/mosquitto.conf
  firewall-cmd --permanent --add-port=1883/tcp
  firewall-cmd --permanent --add-port=1883/udp
  firewall-cmd --permanent --add-port=8883/tcp
  firewall-cmd --permanent --add-port=8883/udp
  systemctl enable mosquitto.service
  systemctl start mosquitto.service

By default the default settings are relatively straight forward, it runs under the mosquitto user, listens on all interfaces, with config options in /etc/mosquitto. In there is a well documented example config file called mosquitto.conf.example which contains all the default settings. Above we rename it to mosquitto.conf to use as a base config for us to modify. There’s details on using the Let’s Encrypt service on the mosquitto site. If you’re using it on the public internet I’d strongly suggest you configure TLS.

Testing it works
Now that we have a basic install done lets make sure it works and get a basic idea of how MQTT works. Included in the mosquitto package is a couple of client utilities useful for testing the MQTT broker pub/sub functionality. Start up a screen session and in a window run the following commands which subscribes us to the “test/topic” topic to the MQTT broker running on localhost:

  mosquitto_sub -h localhost -t "test/topic"

Now lets publish some data to that topic. In another screen tab run the following command which will publish a string of text to the topic:

  mosquitto_pub -h localhost -t "test/topic" -m "Hello World!"

You can run multiple subscribers either from the screen session or remote from a different host but remember to change the “-h localhost” to appropriate hostname/IP if you’re testing remotely.

MQTT can also use two types of wild cards when subscribing to topic names. So to be able to read all temperature sensors publishing to a root topic you could use “home/+/temp” which would provide “home/bedroom/temp” and “home/livingroom/temp”. The “+” will match one hierarchical level where as “#” stands for everything deeper so “home/garden/#” would provide all sensors from the garden. When using wild cards you’ll need to know which topic is reporting so use mosquitto_sub in verbose mode:

  mosquitto_sub -h localhost -v -t "test/#"

Quality of Service
One of the killer features of MQTT for low power IoT sensor networks is that it’s a “store and forward” protocol with a couple of options for quality of service (QoS) guarantees. Once the node has registered with the broker any time it sleeps the broker will queue messages and take care of delivering that message to the node as soon as it reconnects. This means the node can sleep as long as needed to preserve battery life and just wake up and connect when it has data to send or at a predetermined interval without ever missing a message.

There’s three things needed to support QoS: 1) The subscriber needs to be registered by ID with the broker 2) The subscriber needs to disable clean-slate behavior (the default is enabled) 3) Both the publisher and subscriber need to be using quality-of-service level one or two so the broker knows it needs to store the messages. More details on QoS levels are covered here.

Here’s a small demo to play with QoS basics. First connect a subscriber to the broker supplying an ID, disable clean slate and enabling QoS level 1:

  mosquitto_sub -h localhost -v -t "test/#" -c -q 1 -i "PotPlant"

Then send the following two messages, one each with and without QoS. You should see them both show up on the subscriber:

  mosquitto_pub -h localhost -t "test/topic" -m "foo" -q 1
  mosquitto_pub -h localhost -t "test/topic" -m "bar"

Now stop the subscriber (Ctrl+c) and while it’s stopped send both of the messages again before restarting the subscriber. When it reconnects you should just receive the “foo” message because it’s the only one specifying a QoS level.

Further Reading
There’s a lot of information about MQTT on the web. The MQTT Essentials page from HiveMQ is a good place to start for further information and Awesome-MQTT is a curated list of MQTT related stuff with everything from language bindings to gateways/plugins for integration with various projects whether they be home audio or enterprise products.

Fedora 24 Alpha for aarch64 and POWER

So Fedora 24 Alpha is out for aarch64 and POWER. Keen followers will note that we were a couple of days behind the primary architecture’s Alpha release, which hasn’t been the case for the last few Fedora cycles where we’ve generally released on the same day.

The primary reason for the delay was the Pungi Refactor. While the pungi 4 change has been massive for primary architectures for the secondary architectures it’s the single biggest change to our release process EVER! Basically we’ve thrown the lot out and started again. When I started in release engineering over 18 months ago the number one goal that was set for me can be summarised as “Be more like primary. Make the whole secondary architecture as close to primary as possible!” and we’ve been continuously moving, albeit not as fast as I would have liked, in that general direction. With the arrival of pungi 4 for Fedora 24 we’re almost at that end goal in terms of the current way we do secondary architectures.

With Fedora 24 we’re also adding a lot more release engineering focused features and functionality to the secondary architectures. We have now have full nightly composes on rawhide and branched whereas previously we’d just produce a “Everything” repo. This allows ongoing continual testing on things so it’s easier to know when things regress. On PowerPC we’ve produced qcow2 cloud images to some degree since Fedora 22 but it was a bit of a manual process. These are now fully integrated into the pungi/koji process and, like on primary produced nightly, similarly they’ll be coming to aarch64 very shortly too. In Fedora 24 we’ve added Docker base images, they’re produced nighly on branched and rawhide for PowerPC now, and will be nightly for aarch64 at the same time the qcow2 cloud images arrive. Finally aarch64 will also soon have disk images like ARMv7 on primary to enable us to easily support the new shiny aarch64 Single Board Computers (SBCs) that are _FINALLY_ becoming available for the architecture, for Fedora 24 it’ll be a bit of a hack, but with Fedora 25 both ARMv7 and aarch64 will be able to move to koji based live-media-creator image build process but I’ll outline more of that in another post.

So the pungi refactor has been big for the secondary architectures. It’s required big changes in our infrastructure which is now mostly complete, there’s a few infrastructure cleanups and final changes that are in process, these will be done in the next few weeks in the lead up to Beta. We have a single host left to migrate to ansible (YAY!!) and some final moving around of resources. We’ll be changing the way we sync content out to mirrors too which will close out one of the final deltas of the rel-eng secondary process. Overall the last few weeks have been challenging getting all the bits in place, but by the time we hit Beta it’ll all be complete! The new processes lay the foundations for the secondary architectures to add functionality quicker than ever before, and by being almost identical to primary the “onboarding” of new people to use that process, or end users be able to consume the output of the rel-eng process is easier than ever before and that makes me happy! 🙂

Lipstick on a Pig AKA the Raspberry Pi 3

So while waiting for local scratch kernel builds for much more interesting devices I started looking around to see if I could find details of the kernel sources for the new BCM2837 SoC that is centre stage in the Raspberry Pi 3.

The problem is I couldn’t. What I did find is the hack the Raspberry Pi Foundation uses to boot the RPi3 on github.

So there is no source code release for the new BCM2837 SoC, just a device tree file. Someone said to me “They’re violating the GPL” and before people get out their pitch forks… they’re NOT because this is the code they ship, they are meeting their obligations there.

So for the lay person (yes, I know there’s a lot of deep level tech details I’m glossing over deep ARM architecturey people!!) basically what they are doing is booting this device as a ARMv7 device, and because the code isn’t built for ARMv8 (32 or 64 bit) they really just get the speed bump of a ARMv7 device running a bit faster, and possibly some better memory speeds and other general improvements.

So what does this mean for other distributions that wish to actually to support the Raspberry Pi 3 as a aarch64 device? You currently can NOT do so!. Why? Basically it boils down to two things:

  • Source code release for the kernel: To be honest I don’t think this should be large. People with low level knowledge of ARMv7 and the BCM283x could probably hack this up
  • Firmware support: I suspect there will need to be a new firmware that supports booting this as a aarch64 device. I obviously don’t know for sure but I’m guessing the firmware will need changes to actually properly boot this as a aarch64 device. I’ve little doubt there’s a bunch of hackery going on in there!

Of the above two, if my theory is correct, the firmware is the problematic one because it relies on the Raspberry Pi Foundation to do the work. This work for something that they feel, at the moment, gives them no particular gain but only confusion about multiple OSes. They are of course correct for their use case, basically like old school enterprise where you buy a bigger server to scale vertically because your app won’t scale horizontally, but this is another kick in the guts of the Open Source community they so heavily rely on! Oh well, it’s about as much as I expected from the Raspberry Pi Foundation as after all their devices are only just now becoming usable with upstream kernels and open userspace GPU drivers…. after a mere four years.

So what does this mean for Fedora? Basically the only way we’ll be able to support it in the short, possibly medium, term is like it’s sibling the Raspberry Pi 2 as an ARMv7 device but with added shitty wifi. Really, this device isn’t a cheap aarch64 device, it’s just like lipstick on a pig! If a cheap aarch64 device is what you want one of those go and buy a PINE64.

On the plus side the work needed to support it as a ARMv7 device at the same time as it’s sibling should just be some minor u-boot and kernel device tree patches on top of what I previously documented . Note I’ve not looked closely at this as yet, I’m still waiting for mine to arrive (YAY day 3 of 1 day shipping)! Frankly I’d sooner support it this way, an aarch64 device with terrible USB2 IO and 1GB of RAM won’t provide much, if any, of a perf bump over ARMv7, and then have the Raspberry Pi Foundation spend their time working with Broadcom on fixing the wifi and enabling distribution of the wifi firmware in linux-firmware as proper opensource broadcom wifi support would have a wider impact on the Open Source community the Foundation relies upon!

Supporting Fedora 24 on the Raspberry Pi 2

So I get asked the question dozens of times a day so I thought I’d outline the answer to the question “When is Fedora going to support the Raspberry Pi 2?” and “The kernel support is upstream in the 4.5 kernel, why isn’t it enabled in Fedora 24?”

Ultimately support in the kernel is great, it’s obviously a core blocker, and the first steps to supporting a new piece of hardware in Fedora. The thing is that when people say kernel support is easy they are partially right but it’s only a very small part of what’s needed to support a complex device such has an ARMv7 Single Board Computer for the average user, especially one as popular as the Raspberry Pi! To make the device work with Fedora we could just enable the kernel bits but it doesn’t make for a good user experience OOTB (Out Of The Box).

With a lot of ARMv7 devices these days a new device comes out and it just works with Fedora. It’s awesome when I read a report, or someone tells me “I tried device X with Fedora and it just worked”. That’s because of a lot of work Fedora, and others, have done to ensure upstream boot loaders and boot process just works with new devices. It’s taken a long time to get us to this point. The Raspberry Pi is sadly not like all the other ARMv7 or aarch64 devices. It doesn’t have a standard boot process, doesn’t use u-boot or uEFI, needs vFAT partitions, firmware, text config files and other things that none of our other supported devices need. All of these differences need to be taken into account.

So in terms of the support being upstream in 4.5 the answer to that is it’s “mostly” upstream, there’s still a bunch of patches we’d need to pull in to ensure a nice OOTB experience. This isn’t a blocker in my opinion, it’s something that’s relatively straight foward with most of the bits already headed upstream into 4.6 so it’s a short term issue.

For the rest of the bits what do I consider a nice out of the box experience? This:

  • A single image to support the Raspberry Pi and all out devices (more on that below)
  • Graphics and USB support from boot to login
  • Most basic peripherals working, at a minimum USB (inc keyboard/mouse), HDMI, wired ethernet, a decent selection of wireless USB dongles, storage (MMC and USB) and preferably sound (analog, digital HDMI, mic)
  • A means of easily creating a bootable SD card from at least the Fedora command line, and probably Windows or MacOSX
  • Good documentation, FAQ etc

Single Image:
We currently produce a lot of different images for ARMv7 like Workstation, Server, Minimal and various desktops. If we had to double the amount of images we make to add vFAT that would double the work needed by QA, rel-eng and also lead to confusion by end users as to which image is needed. I have no intention of doubling anyone’s work, or adding confusion for end users, there needs to be a proper engineering solution to this problem!

Serial Console
A number of people have said to me “just enable it and tell them to use the serial console” but having been working on ARMv7 for over 6 years now I know from experience that this leads to vast amounts of confusion by end users as to why “it’s not working” and it leads to a lot of time “providing support” to end users. With a device such as the Raspberry Pi this will become an order of magnitude worse which won’t provide users a good Fedora experience, and likely drive the people who are trying to support the device a even more nuts than normal!

Basic peripheral support
I feel that network, storage, display, input and sound, although I’m still on the fence about sound, are the minimum viable supported peripherals needed for a good OOTB experience for users.

I know that initially we won’t support HATs very well, you can already copy around device tree overlays in the /sys filesystem for basic support, but upstream still hasn’t finalised what a good experience will look like in this regard. I think we can live with this. This won’t stop the use of I2C or SPI devices connected to the 40 pin header, most of these should work just fine.

Image to card creation
Closely related to the single image and basic peripheral support I think the ability to easily create an image to use is important. I’m not sure we’re going to be able to easily solve the Windows/OSX problem, although to be honest I’ve not looked at what’s out there and we might be able to extend LiveUSBCreator here.

Raspberry Pi 3
So when will we support this? Well the kernel support hasn’t been released yet, at least that I could find at the time of writing this. That being said I don’t think it’ll be a particularly evasive or large patch set, the hardware around the Cortex-A53 is the same, so it’ll be just some glue and a pinctrl driver to make all the bits work together, likely not too dissimilar to other recent SoCs that have gained ARMv8 support. This won’t land in 4.6 as the major changes are already queued to land in that, so likely 4.7 will be the earliest upstream kernel. The wireless on the other hand could be more interesting.

So basically the work needed for support of the Raspberry Pi 3, with a proper 64 bit OS 😉 , shouldn’t be too hard once the kernel bits are upstream.

Firmwares
The other issue we had was the legal ability to redistribute the “GPU boot loader firmware” thankfully that problem was resolved about a year ago, although it did take us over 2 years to do so!

The other firmware issue which will cause problems with Raspberry Pi 3 support is that the Broadcom wireless is notoriously terrible, as any Linux running Mac user will contest to, and their wireless firmware isn’t re-distributable in the standard process of being included in the upstream linux-firmware. With luck this is something that the Raspberry Pi Foundation could assist Broadcom in improving!

Summary
So we’re well on the way to adding support. Assistance in the issues above would be very welcome, whether for supporting a single image or adding support for writing images for the Pi, to help us get the bits done sooner rather than later. Feel free to reach out to me on IRC (pbrobinson on Libera.Chat) or some other means.

My ARM grab bag device list

They say the first step of coming to terms with addiction is admitting you have a problem… I have a problem with collecting ARM devices… there I said it! How big is this problem you ask? How about I list them out and let you decide!

I’ll break the list down into categories as I believe it’s big enough to do so :-/

The aarch64 set of devices currently stands at:

  • 2x Applied Mustang (different x-gene SoC revs)
  • AMD Seattle
  • 96boards HiKey (hi6220)

The ARMv7 boards list is currently:

  • Compulabs Trimslice (tegra-2)
  • Toshiba AC100 (tegra-2)
  • nVidia Jetson TK1 (tegra-124)
  • Acer Chromebook (tegra-124)
  • BeagleBoard xM (omap3)
  • Nokia n900 (omap3)
  • Nokia n950 prototype (omap3)
  • BeagleBone (am33xx)
  • BeagleBone Black (am33xx) x3
  • BeagleBone Green (am33xx)
  • PandaBoard ES Prototype (omap4)
  • PandaBoard ES B2 (omap4)
  • CubieBoard (allwinner-a10)
  • CubieTruck (allwinner-a20)
  • Banana Pi (allwinner-a20)
  • C.H.I.P. Alpha x2 (allwinner-r8)
  • Snowball (u8500)
  • Compulabs Utilite (imx6q)
  • WandBoard Quad revb (imx6q)
  • novena board (imx6q)
  • RIoTboard (imx6s)
  • UDOO Neo (imx6sx)
  • Origen (exynos-4)
  • OLPC XO 1.75 – a number of variants (mmp2) xNumerous
  • OLPC XO-4 including XO-Touch (mmp3) xNumerous
  • Linksys WRT1900AC (armada-xp)
  • Mirabox (armada-370)
  • ifc6410 (qcom)
  • Parallella Board (zynq7000)
  • Raspberry Pi 2 x3

The Cortex-M series for IoT sensors is currently:

  • TI SensorTag 2015
  • ARM mBed IoT starter kit
  • BeeWi SmartClim

Other random related bits:

  • BeagleBone Breadboard Prototyping Cape x2
  • BeagleBone CryptoCape
  • Original 256Mb Raspberry Pi model B
  • Grove starter kit for BeagleBone Green
  • Explorer HAT
  • PiGlo HAT
  • TI CC2531 802.15.4 USB dongle x3
  • numerous random sensors

So the list above is the devices that I use for hacking on. I count 41 without listing out the dozen or so ARM based XOs I have (various prototypes and models). I also don’t have in that list phones, tablets and two drones as I don’t really hack on those as it’s not like with the list above I don’t already have enough toys! So do I have a problem?