Getting IoT kick started on Fedora

So a number of people have been discussing the Internet of Things on Fedora for some time. We now have a Fedora IoT mailing list where these discussions can be more centralised and directed.

So where and how do we get started here? I’m going to kick start some ideas here and repost it as a mail to the list so we can use it as a basis to start the discussion.

As I outlined in my Using Fedora as a base for the IoT revolution talk at Flock there’s a lot of use cases and components that make up a complete IoT stack. I think initially we should focus on two initial goals rather than biting off too much:

  • A IoT internet gateway device
  • A IoT sensors endpoint device

The general idea here is that both of the above would be a very minimal shared build, likely using atomic images to enable easy update/rollback with some specific components for each use case. Initially I suggest we focus on a single, or maybe a couple, of specific devices to limit the scope to something more achievable and to add features as we go.

IoT internet gateway device specs and features

  • Wired and/or wireless ethernet to provide internet connectivity
  • Bluetooth Smart (AKA LE)
  • Thread Stack support (6LoWPAN and friends)
  • 802.15.4 support
  • MQTT Broker support (not standard for a IoT GW but enables easier localised testing)
  • MQTT Client
  • Atomic support: updates, rollback etc
  • Works with both our endpoint below and other IoT OSes such as Contiki

IoT internet sensors endpoint specs and features

  • Wired or wireless ethernet IP support
  • Bluetooth Smart (AKA LE)
  • Equivalent to Thread Stack support (6LoWPAN and friends)
  • MQTT Broker support (not standard for a IoT GW but enables easier testing
  • MQTT Client
  • CoAP client
  • Atomic support: updates, rollback etc
  • Support for various inputs and outputs and sensors

I have no doubt missed a lot of details in the above use cases, it’s somewhere to start. I think we also need to look at tools like Node-RED and tools for managing the devices. IoT is a big topic, the idea is we need to get the conversation start somewhere. I’ll look forward to seeing you all on the list to do that.

Flock Rochester

I’m not going to do a day by day outline of what I did at flock, if I did it would basically be “blah blah blah I talked a lot to a lot of people about a lot of tech topics” and anyone that’s ever met me would have guessed that! It was, as in the past, a great conference. A big shout out to the organisers for an excellent event with two excellent evening events! So I’m going to give a brief summary to my talks and link to slides and video recordings.

My first talk was an overview of the state of aarch64 and POWER as secondary architectures. The slides aren’t particularly interesting as they’re just words for discussion points. The video has all the interesting bits. A related talk was Dennis’s Standardising ARMv7 booting with a memorial quote by Jon Masters 😉

My second talk was about using Fedora as a base for IoT. Slides are here but the talk was quite a bit different to the slides and is more interesting so I suggest watching the video.

I also actively participated in Dennis’s Fedora Release Engineering going forward because well obviously I’m part of it 😉 and it was interesting for where we’re going, and even where we’ve come from in the last year or so 🙂

Finally I loved the Keynote Be an inspiration, not an impostor by Major Hayden. He’s published a follow up blog post with a FAQ too.

The least memorable bit was the terrible Amtrak ride back to New York City. On the plus side it makes the worst of the British National Rail service seem amazingly on time! NEVER AGAIN!

Fedora 22 on ARM and aarch64

So with Fedora 22 well and truly out for both ARMv7 and aarch64 lets have a look at the release in general and also at the 4.0 kernel it ships.

Firstly I’d like to shout out to the AMAZING job done by the web team on the new sites for Spins, ARM and Labs. They really do look awesome!

It’s all to easy to get bogged down in the actual technical components that make up the actual release and not forget that there’s work that goes on all over the place that contributes to making the release easy to use even before you begin the download.

Anyway! On to ARMv7. We shipped GA with the 4.0.4 kernel and u-boot 2015.01. This combination gives us improved support for numerous devices primarily through new DT support and improvements to drivers. The 4.0 kernel from an ARM HW support PoV really was a minor revision. With the fully packaged and updated fedora-arm-installer it’s even easier to get a device up and running.

We were hoping to get enough time to rebase this to 2015.04 but we just plain ran out of time, although we do have a plan to be able to update the u-boot when writing an image to card/stick without the need to regenerate the images. More details on the improvements we’re planning for fedora-arm-installer for another post!

From an aarch64 point of view the big change, although one an average user won’t notice, was we went from carrying a rather large (around 65K lines) enablement patch for the kernel to a small collection of 4 bug/problem specific patches! This is a massive change where in the F-21 cycle we had a gigantic architecture enablement patch! This makes it much more straight forward and less stressful for both myself as the architecture maintainer and the core Fedora kernel developers. In terms of the rest of aarch64 we still support VExpress, APM Storm platform primarily with the Mustang boards, and the AMD Seattle platform.

Overall the Fedora 22 release on the ARM platforms is a really nice release, there’s been some good changes there to enable easier and quicker updates in the future and easier means of adding decent support to new devices mid cycle. AArch64 is maturing and hopefully we’ll start to see some more platforms land and the architecture start to depart the niche status!

3.19 Fedora ARM kernel status

I’ve been a bit lazy on the ARM kernel status updates. There wasn’t one at all for 3.18 but the fact was, that while there was lots of under the hood improvements for ARM/aarch64, the new device support or improvements from a user’s point of view was positively boring so I never bothered!

That said the 3.19 kernel is now on it’s way to the stable Fedora releases and there’s some bit of interest there 🙂

Beginning with aarch64 there’s been a raft of code support landing upstream for the core platforms we support (VExpress, APM Storm, and AMD Seattle) which means the enablement patch set has shrunk massively. The core missing bit from this is primarily the ACPI patches for the server standards. There’s also been a lot of stability improvements for various device drivers particularly on the APM Storm SoC (which massively helps the high network and IO traffic we generate when doing composes in release engineering!). Other improvements include support for seccomp. The upstream support for aarch64 is really starting to settle down nicely which is good because there’s devices finally starting to get to the point where they’ll be more widely available and affordable 🙂

On to ARMv7 changes. In terms of new supported SoCs the support for AllWinner A-23 (aka sun8i) is the most interesting in terms of new devices. There was also a lot of general SoC improvements and cleanups. The largest here is probably Rockchips, QCom and ZYNQ with notable mentions to Tegra, OMAP and i.MX6 too. In terms of new devices we now solely support DeviceTree devices and the built .dtb files we ship that are possible to support with the kernel jumped from 250 to 265 devices. Of course it doesn’t mean we’re testing all of those devices but we’re testing devices across all main SoC groups to ensure at least the core support works. Of course feedback for what works and doesn’t is always welcome. In this cycle there was also significant driver work with special mention to Hans and his significant movement on the Allwinner devices.

I’ll do a longer post for 4.0 and the new u-boot we’ll be supporting in Fedora 22 soon.

Fedora aarch64, device tree and u-boot support

A question that I’ve had a few times in the last couple of weeks is whether Fedora supports Cute Embedded Nonsense Hacks, also known as u-boot and device tree, on aarch64 (ARM64) platforms?

The answer is YES!, of course, why wouldn’t we?

I know people are well aware of Red Hat’s involvement in the Server Base System Architecture (SBSA) which mandates the use of UEFI 2.4 and ACPI 5.1 bindings and that the Red Hat Partner Early Access Program uses that standard to enable easy booting and support of server platforms running on aarch64 platforms but the fact is that is not Fedora.

Fedora plans to support the SBSA to enable easy use of Fedora on aarch64 server platforms. But we also plan to support the current standard u-boot with device tree boot options. The fact of the matter is that a lot of non server based aarch64 platforms will continue to use these options and so we’ll continue to actively support them. Just like Fedora support Xen when the Fedora derived enterprise product does not. Basically it’s not hard for us to continue these options and with the improved generic distro support in u-boot, which we’ve actively participated in and driven, testing of Cute Embedded Nonsense Hacks on aarch64 should be easy and straight forward.

Of course the support of both SBSA based uEFI/ACPI or u-boot/DTB isn’t perfect on aarch64 yet so if you’ve got access to aarch64 systems on either platforms I would love testing and bug reports. If you’re a vendor that plans on using u-boot/DTB on aarch64 I would ask to ensure that you support the generic distro options because it’ll enable out of the box booting of at least Fedora, Debian and openSUSE to seamlessly just work on your devices.

3.17 Fedora ARM kernel status

With 3.17 due momentarily and Fedora 21 been delayed a little we’ll now be bumping the kernel that ships in F-21 GA. So lets have an overview of what improvements and changes are going to be there for ARM.

Overall 3.17 has been relatively boring in terms of shiny new hardware support for ARMv7. We’ve added support for a bunch of new devices through the addition of appropriate device tree bits. Some of the highlights there include a number of AllWinner devices such as the Banana Pi, a number of new FreeScale i.MX6 devices, some of RockChips devices, and the ZYNQ Parallella.

On the aarch64 side there’s been general improvements all over the place. Over all we don’t have any new platforms but there’s improvements to the three we do support (VExprees, APM X-Gene, AMD Seattle) but the VExpress Juno device should work and initial support for the ACPI 5.1 standard and improved uEFI both of which are part of ARM SBSA Server standard.

Along side 3.17, or at least very shortly there after, u-boot 2014.10 or at least a release candidate should land in F-21 as well. This release adds support for a lot of new devices, primarily AllWinner A-10/13/20 categories, as well as the Tegra Jetson K1, a few i.MX6 devices such as the RIoT Board and the newly upstream distro standards for booting. This makes it much easier for us to “just boot” Fedora ARM with a lot more devices making the experience of getting started a lot easier for most people with supported devices.

The combination of u-boot 2014.10 and the 3.17 kernel will be what we head towards Fedora 21 GA with and things are starting to come together nicely.

Flock 2014 revisited

So having almost recovered from the lack of sleep that is one of the guarantees of conferences in general, but definitely a Fedora one, I thought I would reflect on a few bits. I’m not going to cover all the talks as a lot of people have done that and all the talks are on the Flock 2014 YouTube channel for your viewing pleasure.

As others have mentioned the venue was great, easy to get to and from for transport and the hotel. Huge kudos to the organisers of the event! An event such as this takes a lot of time and energy, and with the dust barely settled the Flock 2015 Bid process is already under way so if you’re interested in hosting 2015 in North America…

My State of ARM and aarch64 in Fedora went well, I enjoyed it and the room was packed with as many people standing as there were sitting 🙂 There was lots of good questions and interest, both in the talk, and in the hallway in general.

I went to numerous excellent talks, too many to count or remember and I’m looking forward to catching up on a number of talks I missed due to schedule conflicts, via the videos, when I get some spare time. Of course the other major part of the conference is the hall way track. There I had too many conversations to recall and caught up with numerous old friends, met a number of people I’d been dealing with online and had never met in person, and of course met a whole bunch of new friends too! It’s amazing how much can be achieved when talking to someone on the walk between conference rooms!

One of the other major things I enjoyed about Flock this year was the overall positivity of everything about the conference, whether it be people’s general attitude, the presentation titles and the presentations themselves or people in the Litre Pub 😉 . And of course being one of our values I have to mention that catching up with so many good friends is always the sugar on top of the cake!

Fedora 21 and ARM device support

As we slowly meander our way towards the pointy end of the Fedora 21 release, with Alpha speeding up in the rear view mirror, the Fedora ARM team are starting to discuss the best way to deal with the blossoming amount of ARMv7 devices that can and do run out of the box on Fedora.

With our 3.16 kernel containing device tree blobs for 200+ devices, the Fedora 3.17 rawhide kernel already containing 230+, it’s truly impossible to actively test and support all of those devices. So much like previous releases we’ll be focusing on testing a group of “primary devices” with the remainder being considered as secondary. This doesn’t mean they won’t work, it just means they’re not necessarily a testing focus of the regular contributors or they might not be readily available to purchase.

So what makes a device primary? Well there’s a number of considerations we’ve put into the list. Firstly the device has to be widely available and well supported upstream. Some will notice that some of the devices are no longer widely available (yes Panda, Trimslice and Calxeda I’m looking at you!) and I did consider their removal from the list but given a lot of contributors have them I think it’s worthwhile keeping them around for the moment. The primary devices list won’t be release blocking, we don’t block x86 releases for specific single devices, so I don’t believe we should for ARMv7 either.

Astute readers will notice the proposed primary list of around two dozen devices is much larger than the core devices we supported in Fedora 20! YAY! is all I have to say about that 🙂

The list is not final, at the moment it’s a suggested list and one open for discussion to some degree and what we’ll be heading from Alpha to Beta with. I fully expect it to be tweaked as we go along, there might be cool new shiny Chromebooks 😉 that arrive on the scene and end up working nicely and are hence worth actively supporting (no EXYNOS Chromebooks I’m not looking at you!) and some devices on the list below that end up not making the grade. One thing is for sure the grade includes that they support Cute Embedded Nonsense Hacks ie DeviceTree… there’s no board support here.

Primary:

  • Wandboard (all models/revisions)
  • Utilite (all models)
  • Cubox-i (all models)
  • Hummingboard (all models)
  • RIoTBoard
  • BeagleBone Black
  • Tegra K1 Jetson
  • CubieBoard (all models)
  • Banana Pi
  • Trimslice
  • PandaBoard (all models)
  • Calxeda Highbank/Midway
  • VExpress (qemu)

Secondary:

  • BeagleBone White
  • Beagle xM
  • Novena
  • UDOO
  • AC100
  • Qualcomm (IFC6410, DragonBoard)
  • Various Marvell devices (Mirabox, AX3, CuBox)
  • Various Exynos devices
  • Other AllWinner devices (as per available u-boot/DT support)
  • Gumstix Overo series of devices
  • OMAP5 EVM
  • STE SnowBall

ARM64:

  • AMD Seattle
  • APM Mustang
  • VExpress (qemu)

So what can a user expect from the primary devices above? Will all the functionality of a device work? Well it depends on the specific device and the associated SoC. For example the AllWinner SoC GPU support is far from upstream so unfortunately there will be no graphical UX for those devices, the Tegra K1 support for the GPU isn’t quite there yet but we’re hoping by GA it will be. Some will be better than others in terms of certain features but for example the AllWinner devices would make good storage devices with their SoC attached SATA and Network, no ugly usb storage/network here, so they are useful to support as a primary device and can easily have feature enablement in the F-21 cycle with a “yum upgrade” to a newer kernel.

We’ll delve deeper into the specifics of each device and the final list closer to beta.

Flock 2014 Prague

So I’m at Flock in Prague. So far I’ve been to a bunch of interesting talks about Release Engineering, Secondary Architectures, Fedora Workstation, Docker and Infrastructure.

Of course then there’s the hallway track of which I always actively participate and it’s been always fabulous to meet a bunch of people in real life that I’ve been dealing with online on a regular basis, in some cases for years!

I’ll be around for the entire conference and if you’re interested in chatting about secondary architectures (not just ARM), Sugar, Cloud or just about anything else or just to say hi please come and find me!

Fedora 21 and ARM aarch64 status for alpha

With the Fedora 21 Alpha freeze looming in the rear view mirror, although the object wasn’t as close as it would appear, I thought it was high time that I gave a brief overview of the state of ARM aarch64 in Fedora. Some might assume the silence means not a lot has been happening but this is extremely far from the truth!

So lets start with a few statistics:

  • Builds the same with mainline: 14973 odd (yes, that’s nearly 15,000 Fedora source packages built on aarch64!)
  • Older builds: 217 (we have a built but it’s not the same NVR as mainline)
  • Missing builds: 352

So that’s looking pretty damn good! The main components that we’re missing that make up the missing builds comprise of two main groups.

The first is builds that are FTBFS on mainline and that’s basically, if it can’t be build on F-21 on mainline we have no chance of rebuilding the f21 tag.

The second reason is platforms that aren’t yet supported on the aarch64 architecture. The core group of these come down to mono (and anything that depends on mono), golang, v8 (mongodb/nodejs etc), pypy make up the majority of that list. We’re working with upstreams to hopefully fill those gaps before long.

There’s a few other minor stragglers that don’t really fit into either of the above. erlang just needs to be bootstrapped plus a few others like thunderbird, libreoffice and hadoop that need some attention which we’ll get to soon.

So the aarch64 userspace, while still not 100% there, is looking EXTREMELY good and there’s a number of people that are now putting it through it’s paces on a daily basis which in turn allows us to improve it as we go.

Hardware
As I indicated in my 3.16 kernel status we now have support for a number of hardware options to run the userspace. Some of them are emulated (qemu, ARM foundation model) and some actual physical (APM Mustang, AMD Seattle) if you’re lucky enough to have access. The support for these devices is improving all the time and support for kernel features are coming along pretty thick and fast.

So in summary the Fedora aarch64 is in very good shape for the Fedora 21 Alpha and will only improve as we apply polish along side x86 and ARMv7 in the lead up to Fedora 21 GA.