Using ZRAM as swap on Fedora

One of the changes I did for Fedora 29 adding using ZRAM as swap on ARM. The use of compressed RAM for swap on constrained single board computer devices has performance advantages because the RAM is an order of faster than most of the attached storage and in the case of SD/emmc and related flash storage it also saves on the wear and tear of the flash there extending the life of the storage device.

The use of ZRAM as swap isn’t limited to constrained SBCs though, I also use it on my x86 laptop to great effect. It’s also very simple to setup.

# dnf install zram
# systemctl enable zram-swap.service
# reboot

And that’s it! Simple right? To see how it’s being used there are three commands that are useful:

# systemctl status zram-swap.service
● zram-swap.service - Enable compressed swap in memory using zram
   Loaded: loaded (/usr/lib/systemd/system/zram-swap.service; enabled; vendor preset: disabled)
   Active: active (exited) since Tue 2018-10-09 22:13:24 BST; 3 days ago
 Main PID: 1177 (code=exited, status=0/SUCCESS)
    Tasks: 0 (limit: 4915)
   Memory: 0B
   CGroup: /system.slice/zram-swap.service

Oct 09 22:13:24 localhost zramstart[1177]: Setting up swapspace version 1, size = 7.4 GiB (7960997888 bytes)
Oct 09 22:13:24 localhost zramstart[1177]: no label, UUID=d79b7cf6-41e7-4065-90a9-000811c654b4
Oct 09 22:13:24 localhost zramstart[1177]: Activated ZRAM swap device of 7961 MB
Oct 09 22:13:24 localhost systemd[1]: Started Enable compressed swap in memory using zram.
# swapon
NAME       TYPE      SIZE   USED PRIO
/dev/zram0 partition 7.4G 851.8M   -2
# zramctl
NAME       ALGORITHM DISKSIZE   DATA  COMPR  TOTAL STREAMS MOUNTPOINT
/dev/zram0 lz4           7.4G 848.3M 378.4M 389.9M       8 [SWAP]
#

When I was researching the use of ZRAM there was a lot of information online. A lot of implementations sliced up the zram into multiple slices to enable the balancing of the slices across CPUs, but this is outdated information as the zram support in recent kernels is now multi threaded so there’s no performance advantage to having multiple smaller swap devices any longer, and having a single larger swap space allows the kernel to be more effective in using it.

In Fedora all the pieces of the Fedora implementation are stored in the package source repo. So those that are interested in using zram for other use cases are free to test it. Bugs and RFEs can be reported as issues in pagure or in RHBZ like any other package.

Fedora on the UP Squared

With the IoT Working Group and Edition moving forward I’ve been looking for an x86_64 device suitable for testing IoT related use cases. I was originally planning on using the MinnowBoard or Joule but given Intel has killed those product lines off it was back to the drawing board. I eventually settled on the UPΒ², in particular I chose the UPΒ² Pentium-4GB-32GB-PACK as it had everything I wanted in one box.

Hardware

The on paper hardware specs show a recent generation Intel Apollo Lake core, reasonable memory and storage options, an onboard FPGA, USB-3, dual ethernet and various other bits. The kit comes with options for active or passive cooling and the later the heatsink is massive, for the moment I’m running it on the passive cooling. The case is OK, I wouldn’t rave about it though. The power connector on the other hand is terrible, the PSU cable doesn’t seat well into the board and I’ve bumped it already and had it lose power, the power button is also tiny, so small in fact I mistook it for a reset button.

Fedora support

As you would expect the support in Fedora 27 is decent, accelerated 4K graphics, wired RealTek ethernet NICs, Intel m.2 PCI-e WiFI/Bluetooth, although the later is only 4.2 for IoT I would have appreciated Bluetooth 5, all work out of the box as expected. The firmware is uEFI and in theory supports secure-boot but I couldn’t work out how to turn it on in the firmware menus as it was greyed out, it also has a TPM2 module I’ve not had time to investigate. They eagle eyed would also note that I mention Fedora 27 even though Fedora 28 has been out a few days. Well for some reason F-28 doesn’t boot, I tried the network installer and the Workstation live image, they both get to the grub menu, then I get no output and nothing for a moment then it resets and we start again. I need to investigate this further but Fedora 27 Workstation livecd booted and installed fine so that’s what it’s got for the moment. Ultimately this is going to a host to test Fedora IoT so while I tested the general support this is fine.

IoT support

There’s a number of reasons I chose this particular device as an IoT test device:

  • Reasonably priced with a reasonable feature set.
  • Intel hasn’t killed it off yet like they’ve done with the Joule platform and Minnowboard 3 so I could actually buy it πŸ™‚
  • Multiple network interfaces, reasonable WiFi and Bluetooth support.
  • Industrial IO sensors via an onboard Intel Sensor Hub. lsiio is reporting 9 sensors of various types. I’ve not checked this further yet.
  • USB-3, a Raspberry Pi HAT compatible connector and other options to add IoT related functionality or interfaces.

FPGA support

I’ve not looked at the FPGA support at all. The upstream kernel now has a FPGA Manager Framework and there’s a bunch of Altera FPGA support there but I’m not sure how it maps to this device. I also have to investigate open source toolchains for FPGA bitstreams as a lot of them just aren’t, I’ll likely do the HW enablement side of things and leave the toolchain bits to people that understand them. I also have the 96boards Ultra96 board so FPGA investigation was already on my Fedora 29 To Do list, and a lot of other people seem quite interested in them of late, no idea why πŸ˜‰

The Raspberry Pi 3 B+ in Fedora

So I’m sure none of you are surprised to hear that I’ve been asked a bunch about support for the Raspberry Pi 3 B+ in Fedora. Well the good news is that it’ll be supported in Fedora 28. Most of the bits were there for the official Fedora 28 beta, it just needed a minor work around, but nightly images since Beta have had all the bits integrated so the upcoming Fedora 28 GA release will support the Raspberry Pi 3 B+ to the same levels as the original 3 B on both ARMv7 and aarch64. The Fedora Raspberry Pi FAQ has now been updated with all the details of both the RPi3+ and Fedora 28.

WiFi

As with the original 3 there’s files with the firmware we can’t redistribute. The details are documented in the Fedora Raspberry Pi FAQ.

You can grab the files like for the 3 although there’s now an extra one, which you don’t really need, but it gives you all the 802.11a frequencies:

$ sudo curl https://fedora.roving-it.com/brcmfmac43455-sdio.txt -o /lib/firmware/brcm/brcmfmac43455-sdio.txt
$ sudo curl https://fedora.roving-it.com/brcmfmac43455-sdio.clm_blob -o /lib/firmware/brcm/brcmfmac43455-sdio.clm_blob

I’ve also done a rpm of the files for both editions of the RPi3 (plus a few other brcm adapters used on ARM boards). You can either just grab the rpm or setup the repo if you want to get the latest if I update it:

$ sudo curl https://fedora.roving-it.com/wireless.repo -o /etc/yum.repos.d//wireless.repo
$ sudo dnf install brcm-firmware

Issues

Like all new things there’s often a few teething problems. Over all the support for the new RPi 3 B+ is relatively solid but like all new devices there’s some bits still bedding down, and the combination of a new device and new OS there’s been a few minor issues that have been seen in some circumstances. We pulled in the latest firmware just before GA to fix some issues but no doubt as it gets wider testing both in Fedora and in the wider Raspberry Pi community more issues may well come up.

The ones we’re aware of are:

  • The new USB hub and gigbit ethernet interface have seen a few issues in some cases. We’ve pulled in quite a few patches to help stabilise the NIC in Linux and it now mostly works in the vast majority of use cases.
  • The USB hub in u-boot, uEFI and grub on aarch64 can cause some issues. If there’s too many USB devices connected it sometimes won’t boot or will do so really slowly. The work around for the moment is to disconnect all the USB devices until Fedora has started to boot and then plug them in.

In Fedora we’ll deliver updates and fixes by the usual updates, in particular as fixes land upstream we’ll review and land them where useful into Fedora, more than likely via a kernel update. If you see a uboot-tools or a bcm283x-firmware update you’ll want to run the rpi-firmware-update command to update the firmware and then reboot for it to take effect.

Older releases

We won’t be supported the new device in the older releases. Why I hear you say? Well it’s possible but it needs update to the firmware, U-Boot and kernel to work. The Raspberry Pi foundation respun Raspbian to support it and basically it’s not straight forward. Much better to have a new shiny Fedora for a new shiny device!

Getting started with Fedora on the 96boards Dragonboard

Support for this board has been a long time coming, it was originally announced in March 2015 and shipped later that summer. Two years on we can finally add support for it to Fedora. The enablement here will also assist us with supporting the newly announced 600c and 820c boards more quickly. We’re not all the way there yet, there’s still some firmwares that needs to go upstream into linux-firmware, but the improvement is fantastic and it’s been a pleasure working with the 96boards and Qualcomm teams getting to where we are today.

At the moment we support running Fedora off either an micro SD card or a USB stick. We don’t currently support running off the eMMC and currently basically treat that as the location of the firmware. Anyway lets get started!

Updating the firmware

You’ll want to update to the latest firmwares, my board originally had an old firmware without support for PSCI and so it didn’t bring up all four cores or support reboot. OOPS! You’ll need the latest linux rescue images from the 96boards download site. As I write this the latest is the 17.09 release (version 88). Create a directory for this file before you unzip it because it’ll expand all into the current directory. While there we also need a u-boot build that’s prepared for flashing, the upstream support isn’t quite complete, we add a few patches to the Fedora build to get everything working nicely. You can grab a pre-built version here and also get LK firmware build which enables display output.

You’ll need a host with the fastboot utility, in Fedora this is found in the android-tools package, and a micro USB cable. This process is very similar to flashing a phone with a new image, not surprising given the chipset really. If you have a serial console on the board you can follow along on the console but it’s not required for this board.

To put the board into fastboot mode we hold down the volume down button, labeled as ‘(-)’ near the middle USB port and then power it on. Wait around 30 seconds to ensure it’s booted to fastboot. You can test this with the fastboot devices command. You’ll likely want to run the next commands as root, or use sudo, and be in the directory you created with the extracted firmware and u-boot build:

sudo ./flashall
sudo fastboot flash aboot emmc_appsboot.mbn
sudo fastboot flash boot u-boot.img
sudo fastboot oem select-display-panel adv7533_1080p

The flashall command runs a series of fastboot command to write out various early boot firmware to the eMMC, then we write u-boot out to the boot partition, and finally ensure that output is configured to appear on the HDMI port. Assuming you don’t get any errors from fastboot that should be all the firmware done and in place.

Fedora image and further setup

Next up is the Fedora image. I chose the Workstation image, but we also have a Minimal Image and a traditional Server image. GNOME not the fastest in the world as 1Gb of RAM isn’t really enough for GNOME-3 anymore, but it works well enough. On a USB stick or Micro-SD card (I’ve tried both). We need to write out the image, then expand the rootfs (Note: update XXX for the device you’re writing to):

xzcat Fedora-Workstation-27-1.6.aarch64.raw.xz | sudo dd status=progress bs=4M of=/dev/XXX
sudo gparted /dev/XXX (expand the last partition)
partprobe

Next up we need to adjust the kernel command line slightly, mount up the first partition and edit /EFI/fedora/grub.cfg and search for the string cma=256MB and delete it, then add in it’s place the following console=tty1 console=ttyMSM0,115200n8. Next mount the boot partition (partition 2) and create a sym link

ln -s dtb-4.13.9-300.fc27.aarch64 dtb

. Unmount the partitions and we should be good to go on the Dragonboard.

Plug in a keyboard, mouse (and/or a usb cable for the serial console if you’re going that route) and a HDMI cable, plug in the USB stick or SD card and power it up. If you’re following along on the serial console you should see output straight away, screen might take a little longer.

Once you’ve booted you should be able to complete initial-setup (text or the one from Workstation) and login. To get the WiFI and Bluetooth working you need to install a Radio (WiFi and friends) firmware package which I’ve made into a rpm you can grab from here until it lands into linux-firmware.

What next?

The DragonBoard 410c is pretty functional. I’ve not widely tested sound, the Venus media offload components (we have all the firmware and kernel bits for this), the GPS or some of the other more advanced components but I’ll have more details about those soon. I’ll be documenting the above plus other bits on the Fedora ARM wiki so keep an eye on that or get involved and help out πŸ˜›

The state of open source accelerated graphics on ARM devices

I’ve been meaning to write about the state of accelerated open source graphics options for a while now to give an update on a blog post I wrote over 5 years ago in January 2012, before the Raspberry Pi even existed! Reading back through that post it was pretty dark times for any form of GUI on ARM devices but with the massive changes in ARM devices and the massive change in SBCs (Single Board Computers) heralded by things like the Raspberry Pi have things improved at all? The answer is generally yes!

The bad

Looking back at that post the MALI situation is still just as dire with ARM still steadfastly refusing to budge. The the LIMA reverse engineering effort started with promise, but went up in smoke with a fairly public community break down, I don’t envision that situation improving any time soon although just recently there appears to be some forward movement happening finally after a long silence. This only covers the MALI-400 series and any newer GPU is a completely different architecture/IP. Even with sessions recently at Linaro Connect titled What’s happening with ARM Mali drivers I don’t see fast change here.

The Imagination Technologies PowerVR is still just as dire as situation as it was five years ago. The company’s incompetent management recently managed to avoid being bought by Apple which in turn, because they’ve screwed the open source community while milking the Apple cash cow, essentially means they’re screwed. I suspect they’ll either open source to try and remain a relevant contender or die in a tire fire. Only time will tell there, in the mean time any ARM SoC that has this IP on board is useless for anything graphical so I’d tend to avoid it, thankfully there seems to be less of them these days.

The good

Despite the two bad examples above there’s actually been a lot of good change in the last five years. We now have a number of options for fully accelerated 2D/3D graphics on ARM SoCs and I run GNOME Shell on Wayland, yes the full open source shiny, on a number of different devices regularly.

NVIDIA true to the rumours did open up all the graphics on the Tegra series of hardware. The new Tegra K/X series have GPUs similar to their x86 offerings with Kepler/Maxwell/Pascal GPU cores but NVIDIA supports these devices by contributing to the nouveau open driver rather than the closed x86 driver. The performance on 32 bit TK1 devices has been decent for a number of releases of Fedora and improves all the time, we’ll be supporting the X series (X1/X2) with their Maxwell/Pascal GPUs in Fedora 27.

In the old post I brushed past Vivante with a mere mention of Marvell and Freescale (now NXP). The Vivante GPUs ship in NXP i.MX6 and i.MX4, some Marvell chips and some TI chips. There was a reverse engineering effort called etnaviv that must have started not long after I wrote that post and after a number of years of development support landed upstream in the kernel late 2015, and in mesa in the 7.1 release allowing us to support fully accelerated Wayland in Fedora 26! Did anyone notice? I didn’t really yell about it as much as I should have! It supports fully accelerated 3D in mesa/wayland, is pretty stable and is improving all the time, well done to all the contributors to that effort!

Another I brushed past in the old post was the Qualcomm Snapdragon SoC. They ship with a Adreno GPU. This was previously closed source, with the SoC primarily used by phone/tablet manufacturers I suspect they didn’t care… until Rob Clark (and no doubt there were other contributors) decided to reverse engineer the driver with the open freedreno driver. This is now the default driver with even Qualcomm contributing to it. We’ll support this in Fedora 27, initially with the 96boards Dragonboard 410c using the freedreno driver, but I doubt it’ll be the last Qualcomm based device we support. The Snapdragon 835 SoC, the device in all the high end Android phones this year and the ARM Windows 10 laptops, is really nice with decent performance, I’d love to be able to support a device with that SoC!

Raspberry Pi, as I mentioned in the introduction, wasn’t even out in when I wrote the original post. When it fist launched there wasn’t an open driver but 5 years later there is, sponsored by Broadcom no less. We introduced initial support for the Raspberry Pi with the open vc4 driver by Eric Anholt in Fedora 25 and it’s improving regularly. It supports fully accelerated 3D in mesa/wayland, and 2D via glamor in mesa.

So in conclusion we have improved by A LOT! We now have numerous different GPUs with open drivers to choose from in all price ranges that support fully accelerated 2D/3D desktops from four different vendors on both ARMv7 and aarch64. The media acceleration offload is also looking quite good, but that’s one for another post. The biggest holdout is MALI, and that would need two open drivers or ARM to come to the table, LIMA might work out for the 400 series, but that won’t work on the newer midguard series. With support in a number of drivers for the shiny new Wayland there’s an increasing number of devices people can use to enjoy the latest desktops fully accelerated!

WiFi on Raspberry Pi 3 for Fedora 26 Alpha

So I managed to land just about everything needed for the WiFi on the Raspberry Pi 3 for Fedora 26 Alpha (around 4.11 rc3). There’s one thing missing, because we can’t currently redistribute it, but it’s straight forward for the end user to do themselves once they’ve done the initial setup:

sudo curl https://raw.githubusercontent.com/RPi-Distro/firmware-nonfree/master/brcm80211/brcm/brcmfmac43430-sdio.txt -o /lib/firmware/brcm/brcmfmac43430-sdio.txt

Or you can also do it when you’re flashing the image if you mount the root filesystem but the above is likely easier. It’s been surprisingly stable in my testing.

Before you all ask, at the moment I don’t plan on pushing this to earlier Fedora releases, as the upgrade path is not trivial. I will also soon publish more details of some of the other new features coming for the Raspberry Pi to Fedora 26 but I thought you’d all like the WiFi details now. The wiki has also been updated to reflect the status of the WiFi.

PS: No this is not an April Fool’s joke (it’s well past midday in UK).

Connect to a wireless network using command line nmcli

I use a lot of minimal installs on various ARM devices. They’re good because they’re quick to download and you can test most of the functionality of the device to ensure it’s working or to quickly test specific functionality but of course it doesn’t have a GUI to use the nice graphical tools which are useful to quickly connect to a wifi network or other things.

This where nmcli comes in handy to quickly do anything you can do with the GUI. To connect to a wireless network I do:

Check you can see the wireless NIC and that the radio is enabled (basically “Airplane” mode):

# nmcli radio
WIFI-HW  WIFI     WWAN-HW  WWAN    
enabled  enabled  enabled  enabled 
# nmcli device
DEVICE  TYPE      STATE         CONNECTION 
wlan0   wifi      disconnected  --         
eth0    ethernet  unavailable   --         
lo      loopback  unmanaged     --         

Then to actually connect to a wireless AP:

# nmcli device wifi rescan
# nmcli device wifi list
# nmcli device wifi connect SSID-Name password wireless-password

And that should be enough to get you connected. You can list the connection with nmcli connection and various other options. It’s pretty straight forward. The only complaint I have is that it doesn’t prompt for a password if you leave it out so it means the AP password is stored on the command line history. Not a major given it’s quite easy to find where it’s stored anyway on the system but it would be a useful addition.

When creating updates remember to build for rawhide and Fedora 25 (devel)

When ever we branch for a new release of Fedora I, and others, end up spending a non trivial amount of time ensuring that there’s a clean upgrade path for packages. From the moment we branch you need to build new versions and bug fixes of packages for rawhide (currently what will become Fedora 26), for the current stabilising release (what will become Fedora 25) as well as what ever stable releases you need to push the fix for. For rawhide you don’t need to submit it as an update but for the current release that’s stabilising you do need to submit it as an update as it won’t just automagically get tagged into the release.

As a packager you should know this, it’s been like it for a VERY LONG TIME! Yet each cycle from the moment of branching right through to when a new release goes GA I still end up having to fix packages that “get downgraded” when people upgrade between releases!!

So far this cycle I’ve fixed about 20 odd with the latest being bash-completion (built but not submitted as an update for F-25) and certmonger (numerous fixes missing from F-25 and master branch).

The other silly packaging bug I end up having to fix quite a bit is NVR downgrades where even though it’s a newer package the way the NVR is handled makes rpm/dnf/yum think the newer package is a lesser version than the current version and hence you’re new shiny fix won’t actually make it to end users. I see this a lot where people push a beta/RC package to a devel (F-25/rawhide) release. Just something to be aware of, there’s lots of good docs around the way rpm/dnf/yum handles eNVR upgrades.

Flock 2014 Prague

So I’m at Flock in Prague. So far I’ve been to a bunch of interesting talks about Release Engineering, Secondary Architectures, Fedora Workstation, Docker and Infrastructure.

Of course then there’s the hallway track of which I always actively participate and it’s been always fabulous to meet a bunch of people in real life that I’ve been dealing with online on a regular basis, in some cases for years!

I’ll be around for the entire conference and if you’re interested in chatting about secondary architectures (not just ARM), Sugar, Cloud or just about anything else or just to say hi please come and find me!

Verne in beta…

I have a number of devices that run Fedora and they all get upgraded at various times through out the release process. The eeePC 901 gets upgraded, reinstalled and uses the upcoming release regularly and constantly. My build test VM runs the release from shortly after beta of the previous release. My “stable” devices upgrade times vary depending on the release. These are devices I need to live my life and include my work laptop, my NAS box, my media centre box, my firewall, and two servers. So what releases are they all running πŸ™‚

Work laptop: this was upgraded this week to Verne. It didn’t get upgraded to F-15 for a couple of months after the release. Most of the time it gets upgraded around the beta. I have some issues with it, most are thankfully not major road blocks. I’ll cover this one in a separate post.

NAS: Running F-16. Its solely CIFS/NFSv4 and some other services like Squeeze Centre Server. No major problems as yet.

Build VM: Its a VMWare VM. Has been fine, but for some reason of late its suspending or locking up of late. This is somewhat concerning.

Media Centre: I need to upgrade it, i’ve just not had the time. I’m looking forward to trying out Media Explorer once I’ve finished the package review πŸ™‚

Firewall: Sometime very soon. I wish NetworkManager would add support for native IPv6 on PPPoE connections

Servers: My primary hosted server is still running Fedora 14. It runs a number of websites, mail servers and VMs. I want to upgrade this but it needs a proper backup process first 😐 My other server is running F-15 and I’ll likely push it to F-16 once I’ve worked out why the buildbox VM is suspending/crashing, it is after all my primary mail server.

Overall I’m looking forward to Verne. Its looking to be a very solid release with some great new features and the stabilisation of two of the major changes in F-15 in systemd and gnome-3 with a lot of thanks going to Adam Williamson and the QA team and their tireless work to keep the rest of us in check and on schedule. I believe Adam barely slept in the four days leading up to Verne beta being declared good to go so we didn’t slip another week. Thank you all! I’m looking forward to a great release πŸ™‚