ARM – Page 5 – nullr0ute's blog

Fedora 21 and ARM aarch64 status for alpha

With the Fedora 21 Alpha freeze looming in the rear view mirror, although the object wasn’t as close as it would appear, I thought it was high time that I gave a brief overview of the state of ARM aarch64 in Fedora. Some might assume the silence means not a lot has been happening but this is extremely far from the truth!

So lets start with a few statistics:

Builds the same with mainline: 14973 odd (yes, that’s nearly 15,000 Fedora source packages built on aarch64!)
Older builds: 217 (we have a built but it’s not the same NVR as mainline)
Missing builds: 352

So that’s looking pretty damn good! The main components that we’re missing that make up the missing builds comprise of two main groups.

The first is builds that are FTBFS on mainline and that’s basically, if it can’t be build on F-21 on mainline we have no chance of rebuilding the f21 tag.

The second reason is platforms that aren’t yet supported on the aarch64 architecture. The core group of these come down to mono (and anything that depends on mono), golang, v8 (mongodb/nodejs etc), pypy make up the majority of that list. We’re working with upstreams to hopefully fill those gaps before long.

There’s a few other minor stragglers that don’t really fit into either of the above. erlang just needs to be bootstrapped plus a few others like ~~thunderbird~~, libreoffice and hadoop that need some attention which we’ll get to soon.

So the aarch64 userspace, while still not 100% there, is looking EXTREMELY good and there’s a number of people that are now putting it through it’s paces on a daily basis which in turn allows us to improve it as we go.

Hardware
As I indicated in my 3.16 kernel status we now have support for a number of hardware options to run the userspace. Some of them are emulated (qemu, ARM foundation model) and some actual physical (APM Mustang, AMD Seattle) if you’re lucky enough to have access. The support for these devices is improving all the time and support for kernel features are coming along pretty thick and fast.

So in summary the Fedora aarch64 is in very good shape for the Fedora 21 Alpha and will only improve as we apply polish along side x86 and ARMv7 in the lead up to Fedora 21 GA.

3.16 Fedora ARM kernel status

So 3.16 is has quite a few new features in terms of newly supported devices, also some what surprisingly this blog post will be out before 3.16! In terms of new device support all the SoCs listed here are exciting for a number of reasons for Fedora ARM. Aarch64 (ARM64) makes it’s first debut with support of real hardware although we’ve actually had kernel support enable for it for some time in Fedora even if only usable on the glacial Foundation emulator.

The 3.16 release is also very likely to be the kernel that ships with Fedora 21 GA and with the Alpha due in about a month we’re starting to polish and test all the platforms and devices we want to support for GA.

Anyway without any further a do let’s get into the gritty details:

NVIDIA Jetson TK1 support: While we’ve had the basics of this for a while all of the bits are there now.
EXYNOS support: This SoC is probably the most asked about platform and finally after a long wait the multiplatform support has landed upstream. We currently ship around 20 dtb files for exynos4 and 5 (Chromebook support anyone?). Testing is sought and feedback and greatly appreciated.
Qualcomm MSM 8×60, 8960 and 8974 support: While the multiplatform support for these devices landed upstream a few releases a go they’re now to the point they should be relatively usable so it’s time to get wider testing. This should be the beginning of supporting the venerable ifc6410 and dragonboard devices.
APM X-GENE support: One of our first aarch64 supported pieces of hardware. Similar to the QCom SoC the initial support has been upstream for a while but with 3.16 it becomes usable with the vast majority of basic support upstream so minimal patches are needed. More on aarch64 soon.
AMD Seattle support: The other of our aarch64 supported pieces of hardware if you’re lucky enough to get your mits on a device.

The other feature we’re starting to see mature is GPU and Graphics support. I’m not exactly sure yet as to what the final state of this functionality will be for Fedora 21 GA but we potentially will have suppport for:

nouveau/mesa support on the NVIDIA Tegra K1
freedreno/mesa support on the Qualcomm boards
etnaviv/mesa support on i.MX6 devices
improved modesetting support for a number of other devices. Some of this has already landed and is usable in rawhide now.

What covered above is just a high level overview of what’s new in the upcoming release. There’s been numerous other improvements in existing supported SoCs and devices all over the place that would take too long to cover off here but in short with all the shiny that’s landed in 3.16 what Fedora ARM will look like as part of the Fedora 21 GA release is quickly starting to take shape.

3.15 Fedora ARM kernel status

There’s been quite a bit of water under the bridge since my post on the 3.14 kernel status. With 3.15.x due to land in Fedora 20 shortly I thought I’d give an overview of changes for 3.15 and what’s happened since the last post.

From a shiny new devices and features point of view the 3.15 kernel is relatively boring on the ARM devices front, the advantage of that was that from a development point of view things tended to just work on Fedora. Running a diff between out 3.14 and 3.15 ARM kernel configs and checking our shipped Device Tree Blobs I get the following main changes:

Enabling of Marvell Dove platform. This primarily will be useful for people with the Original Cubox
SunXi MMC suuport. The enables initial basic support (Serial/MMC/network) for a number of AllWinner platforms
Zynq 7xxx platform improvements
OMAP DRM driver conversion to Device Tree (more on that below)
Initial Utilite support. It’s pretty basic with support for serial, MMC/SATA, and one of the two NICS. I plan on improving this soon
Added Device Tree support for a number of OMAP Overo devices
Added Device Tree support for a number of i.MX6 based devices

So while it was boring form a new device support point of view a kernel cycle for ARM is never really that boring! There was a lot of nice improvements generally under the hood and the march toward Device Tree is basically complete. I’m not aware of any device now that is supported not through DT in Fedora.

I mentioned above OMAP DRM. In the 3.14 post I mentioned I was sure we’d get Panda working soon. And we did! The main issue remaining was actually display support with 3.14 and with 3.15 that problem is now mostly closed because all the connectors and their associated drivers now support DT which meeans all the modules now load in the right order and things mostly just work. There’s some further improvments here in Xorg userspace in rawhide so I’d sugggest trying a nightly or the not far off Fedora 21 Alpha.

I’ve also had a few cycles to test Marvell mvebu support on my Mirabox and fixed a few kernel issues here so it now works. Unfortunately Marvell’s support of uboot, and hence Device Tree, is from the last decade and hence fairly horrible! I’ll save details of that for another post.

aarch64 (arm64) on Fedora 21 update

I realise that there’s not been an update as to the state of ARM aarch64 on Fedora for sometime. We should really shout out our achievements a lot more as over the last few months or so there’s been fairly extreme progress made but we all just tend to apply the nose to the grindstone and get one with it!

So where are we? Well we’ve got the vast majoriity of Fedora built. As we’re only building for rawhide/F-21 there’s a number of missing F-21 builds but with the mass rebuild well and truly underway on mainline we’re all hyped up and building on aarch64 too (appart from a power outage in the Huntsville facility that hosts some of the builders!) and we’re hot on the tail of mainline builds.

So we have HW virtualisation support and a bunch of virt bits on top of that, the vast majority of other services and low level support like storage platforms, all the various desktops and most of the various development stacks (even Ada!). It’s actually looking pretty good. In recent time we’ve added R, ghc and a raft of different language support. The java team have done a really good job getting java-1.8.0-openjdk up to spec too!

So appart for the mainline builds that haven’t yet been built for F-21 what are we missing? Well not much really now. Go and nodejs, the later due to missing v8 js engine support (it’s upstream in a newer v8 release but Fedora is stil trailing), and the things that depend on them are the main ones now missing so no Docker as yet either. There’s no mono, and while the core erlang is built I need to work out the correct order to finishing the bootstrap.

The rest of the job now becomes somewhat boring in that once the mass rebuild is complete we just need to go through packages and fix individual packages that are FTBFS and I strongly suspect most of those will have issues on mainline too.

On the hardware and virtualisation front the current rawhide kernel should boot just fine on the ARM foundation model emulator and the next release of qemu will have full system emulation of aarch64 to enable the running of aarch64 on x86_64 similar to what we can do with ARMv7 now. The 3.16 kernel should allow us to run on the currently and soon to be available HW without too many problems either as the last core bits of support should land upstream in the current merge window.

In short aarch64 on Fedora is looking awesome! By the time Fedora 21 Alpha lands in two months time we’ll basically be feature complete, and even now it’s extremely usable for those lucky enough to have hardware or patient enough to use software emulators.

3.14 Fedora ARM kernel status

With 3.14.1 out Josh and Justin are preparing to land the 3.14 kernel into Fedora 20. So what does it mean in terms of ARM on Fedora. Well it’s an evolution. There’s of course the usual raft of new devices and some new SoCs, and best of all lots of improvements in support for existing devices. Even the return of some old favourites! Generally from the stash of devices Paul, I and others have that get regularly tested things are looking pretty good with 3.14.

New SoCs and device support:

Tegra 4/K1 support has been enabled. For 3.14 this doesn’t mean much as there’s not a wide level of devices out there that we ship device tree blobs for but it’s a good preparation for 3.15 as we should have pretty reasonable support for the nice new Tegra K1 dev board!
TI devices: The 3.14 finally brings working support for the OMAP5 EVM board, this will improve further in 3.15 but it boots and generally works. It also adds support for the original BeagleBone and the USB is now finally all back for the Beagle-XM devices so they go back into the fully working list! Also the DT bits for the Overo devices has started to land so if is interested in those I’d love feedback from people with those devices.
Freescale i.MX: To this lovely growing list we add initial support for the various Cubox-i devices and the hummingboard. Still no HDMI support but here’s hoping for 3.15
Xilinx Zynq 7000: The SD controller has finally landed for this which means they should be bootable to login but from there I’m not sure the status of the rest of the support. We ship DT for the zynq-zc702, zynq-zc706 and zynq-zed devices so if you’ve got one of those and have time to test feedback would be welcome.

Interesting bugs fixed:
Nothing particularly exciting comes to mind here. The fix for the Beagle-XM usb hub power up is nice, as was the final config option for the BeagleBone White. There’s obviously a deluge of other ARM driver fixes and improvements (PTP high precision time support for modular CPSW ethernet on the BeagleBone’s anyone?).

Outstanding bugs and issues:
So this is really the list of items I have outstanding for 3.14 so that I can spin some new images. Feedback on any of these and anything I might not be aware of would be very useful.

OMAP DRM display. In 3.11 a new display framework landed for the OMAP DRM driver and in 3.12 the old one was dropped. This broke X on devices like the Pandaboard for 3.12+ kernels. I know roughly the problematic area but I just need to get the cycles to debug this. Any help is welcome.
Serial over USB. While this isn’t a kernel bug but rather just needs me to hack together some scripts it’s a blocker for easy OOTB support for devices like the BeagleBone Black
Tegra DRM display. After a re-write it’s back and modular again in 3.14. Just need to ensure it’s working and ready to go
Testing… testing… testing 🙂

That’s mostly all I have on my list. If there’s anything I’m not aware of please do let me know and I’ll endevour to help out where possible. In particular I’m very interested in boot issues for devices that would would be supportable with new ARM images based on 3.14. From the 3.11 kernel that F-20 GA shipped with there’s been a lot of change and improvements, and while non boot enhancements are easy to do with a “yum update” issues with boot aren’t quite so easy to deal with!

Semi irregular Fedora ARM kernel status reports

I thought I’d start doing semi irregular ARM kernel status reports. I’ll do them as often as I think they might be useful and those who know me know I travel a lot and randomly and that ARM isn’t my $dayjob.

Few are bored or stupid enough to follow the 20 or so ARM kernel trees or have the regular insight as to what’s happening, what’s landed, what new devices might work and what bugs come and go that I do so I thought I’d try and dispense some of the more interesting bits of that information and how it relates to Fedora ARM to a wider audience by both the fedora-arm mailing list and my blog so those people that don’t sit on the IRC channel and those that like to lurk might have a better idea what’s going on.

The general format I plan to use is basically:

What’s new including SoCs, boards and new devices
Interesting bugs fixed
Outstanding bugs and issues
Random other insights

I don’t intend them to be long but rather short, sweet and to the point. They’ll probably come out when new major releases hit either rawhide or stable or something of particular interest lands. Feedback on both the format most other things is welcome as are questions and status of devices people might have had success or less so with.

I plan to have the first for 3.14 out later today and one for 3.15 RSN.

Booting PandaBoard with Fedora 20 GA

So I’ve always thought the fix for Fedora 20 booting on the PandaBoard devices would be a simple fix. I’ve always assumed it would be a minor kernel config change. I was wrong, it was even more simple once we worked it out with some help of our friends!

So to make it work you can do the following. First get a Fedora ARM Image, I used the Minimal Image, you’ll need the VFAT variant for which ever one you choose.

As root (or use sudo) run the following commands but remember to change the device path:

xzcat Fedora-Minimal-VFAT-armhfp-20-1-sda.raw.xz > /dev/mmcblk0; sync

Eject the card and put back it back in so it gets the new partition table etc and then copy the uboot over to the uboot partition. In my case I did (but change the path to your mount points):

cd /run/media/USERNAME/__/usr/share/uboot-panda/
cp MLO u-boot.img /run/media/USERNAME/uboot/
cp uEnv.txt.panda_es /run/media/USERNAME/uboot/uEnv.txt
umount /run/media/USERNAME/__/ /run/media/USERNAME/uboot/
sync

At this point you’ll likely want to use gparted or similar tools to expand the root filesystem. Once complete run screen or your favourite serial console application:

screen /dev/ttyUSB0 115200

Put the SD card into the panda, apply power, and boot through to u-boot but interrupt the boot process and get the Panda# u-boot prompt and enter the following set of commands:

setenv bootm_size 0x20000000
setenv bootargs console=${console} vram=${vram} root=LABEL=_/ ro rootwait
ext4load mmc 0:3 0x82000000 /boot/vmlinuz-3.11.10-301.fc20.armv7hl
ext4load mmc 0:3 0x88080000 /boot/uInitrd-3.11.10-301.fc20.armv7hl
ext4load mmc 0:3 0x88000000 /boot/dtb-3.11.10-301.fc20.armv7hl/omap4-panda-es.dtb
bootz 0x82000000 0x88080000 0x88000000

From there it should boot all the way through to the first boot configuration tool on the serial console, once you set a root password, create new user and set the timezone it will boot you thought to the login prompt.

That will get you to the point of being able to play. Networking works fine as does USB and a bunch of other core functionality, X doesn’t appear to with the first look and if you update the kernel (I’ve tested the shipping 3.11.10-301.fc20 and latest 3.113.13.6-200) you’ll just need to follow the process above with updated kernel details to reboot.

We’re working to get updated arm-boot-config to ensure the expected automated boot process plus some updates for X etc but I wanted to get this process out for those chomping at the bit to play while we review the changes! As always please report any issues (or success) to the Fedora ARM mailing list or IRC channel.

In terms of working devices I’ve personally tested my Panda-ES prototype (which never successfully booted anything later than 3.6.x) and Panda-ES B2, I’ve had reports of a number of variants of the vanilla Panda and ES working. I’m not sure about the B3 revisions with the Elpida DDR2 RAM but you might want to try the above with the Fedora 2014.01 uboot as it has the uboot components needed.

Lastly I’d like to explicitly call out Paul Whalen for his persistence on this one and Tom Rini who ultimately provided the tiny bit of glue we needed to solve this so simply!

I’m glad to be able to add this mainstay of the Fedora ARM project back into the supported list with fan fair, I just wish we’d found this prior to F-20 GA!

Update: It appears that X with at least the XFCE image works with the GA image.

ARM hardware support on Fedora 20

A number of people complain about our hardware support on ARM. I’m not sure people really understand the issues we have bringing the hardware that we try to support to ARM and the constraints we work in. I’m going to try and outline the process we’ve been through over the last couple of releases and the reasons why we stick to the Fedora processes as much as possible. The other thing that I think people quickly forget is that the hardware support is a very small percentage of the overall work that has gone into bringing ARMv7 up to the excellent state it is in. Unfortunately like so many other projects the last 10% is always the hardest.

Support for hardware primarily revolves around three core components. They are the boot-loader (bios in x86 parlance), the kernel and any user space components needed which generally focuses on X drivers, but is no way the only thing.

Going back to the near memory Fedora 18 shipped with the 3.6.10 kernel. The 3.6 kernel didn’t ship with what’s known as the “Unified Multi Platform ARM kernel” which now allows us to ship a single kernel to boot multiple different SOCs (System on a Chip) by numerous manufacturers. The initial multi platform kernel support landed in 3.7 and initially allowed us to merge the Versatile Express and Calxeda Highbank kernels together. It was only with the 3.10 kernel we actually started to ship a single unified kernel that supported all the SoCs we support. We currently enable around a dozen different SoCs in the mainline kernel and actually support HW running on around half of those.

Supporting the multi platform kernel hasn’t been an easy ride. As you would expect of Fedora we were the first to adopt it and a lot of other distros still haven’t. Debian is in the process of moving over to it for Jessie, I thought Open SuSE was using it but it appears they’re not and Ubuntu certainly isn’t. I’m not sure about Arch or any of the others. It’s been an interesting ride and while a single kernel is something we’ve wanted for some time the lack of testing by a wider audience has certainly made it more painful for us than it could of been, it would be particularly helpful if Linaro, the main developers, actually used multi platform kernels for all their builds.

An example of problems we’ve had with the multi platform kernel is the support for the OMAP4 based PandaBoard, since before Fedora 17 it’s been consistently our best supported device but with the 3.11 kernel and the move of OMAP4 devices to DeviceTree it doesn’t boot and there’s been a number of us try to get to the bottom of the problem to no avail. Similarly going back to 3.6/3.7 the i.MX series of SOCs caused us massive problems to the point we just stopped supporting them but with the i.MX6 it’s very quickly becoming out best supported class of devices with a collection of almost a dozen very nice and relatively cheap devices (starting at USD$45) like the WandBoard dev boards, Compulabs Utilite and Cubox-i. Due to the high level of churn in the upstream kernel it at times makes it really hard and we can, and do, spend a lot of time chasing down issues with a single SoC or even device.

So what hardware actually works in Fedora 20? In theory we support 100s of devices but in practice the testing in limited to a selection of devices that we actually have to be able to easily test. So what actually works:

Versatile Express via QEMU emulation using libvirt.
Calxeda Highbank and Midway servers
Compulabs Trimslice (tegra).
The three WandBoards (i.MX6) in particular the Quad. 3.12 will improve the support too.
The BeagleBones (am33xx). In 3.11 there’s issues with usb, 3.12 is looking better.
BeagleBoard xM (omap3). For network you need to use the usb OTG port.
Mirabox and other Marvell (mvebu) devices with appended DTB (due to old uboot shipped with device).

Added to the above we ship around 100 device tree files, in 3.13 that rises to over a 120. I’ve had reports of a number of people successfully using some of these devices without issues. So what doesn’t work so well out of the box?:

PandaBoards (omap4). As mentioned above these broke with 3.11, I’m hoping to get them working again soon but my testing to date hasn’t been fruitful though.
Compulabs Utilite (i.MX6), well not with the 3.11 shipping kernel, we have initial support in 3.12 but there’s also issues with their initial uboot release.
AllWinner devices, but there’ll be a high quality remix that supports a lot of these devices, we’re hoping to improve the mainline support a lot in the F-21 development cycle.
Any device that doesn’t support multi platform kernels
Any device which isn’t supported in the upstream kernel

I’ve had reports of a number of other devices that “just work” or mostly work and with time this will improve. One thing that we’ve tried very hard not to do is pull 100s of patches in. Fedora likes to be as close to upstream as possible with their kernels and having lots of patches or kernel variants just doesn’t work as we just don’t have the resources to deal with them. We do on occasion pull in patches to fix or improve things but we try to keep as close to mainline as possible so if you want a device supported the first thing you’ll be asked is “What’s it’s supported state on the upstream kernel?”.

So what’s on the ToDo list? In short.. LOTS! With 3.14 we should finally get usable AllWinner devices, 3.13 in theory should have a number of RockChip devices supported and both of those SoCs bring a huge amount of cheap devices along that we have the potential to support. As it makes sense to enable new support we will. I’m also planning on improving and testing the support we currently have. I like the BeagleBone expansion capes so I plan on testing and playing with those and in general improving the support of devices that I have (a post on those coming soon).

All in all while we’ve got a long way to go I believe our hardware support has improved fairly well over the last couple of years where in Fedora 17 we officially supported two devices.

Serial console options on the Beagle Bone Black

So unlike the original Beagle Bone, which had a built in USB serial adapter, the Beagle Bone Black only has a serial header and you have to buy a USB to serial adapter to get a real serial console.

There’s one other option for a pseudo serial console over the USB On the Go port but the problem with this is that it doesn’t work with u-boot so only works once the kernel has booted as we can setup the port. The enabling and use of the USB OtG in Fedora is still on my ToDo list to investigate for ARM but we can possibly enable it as both serial and usb network at some point in the future.

So for now we need to use the 6 pin header to connect a USB to serial adapter. The most important thing to note here is that it requires 3.3 volts for the data signals so don’t use some of the older 5 volt units. The best USB to Serial to use is the FTDI FT232RL but at $20 it’s almost half the price of the device. The advantage is that it just works and the 6 pin connect just plugs straight onto the board (black goes to PIN 1). I’ve also tried the Adafruit 4 Pin Cable (PL2303), which at $9.95, is less than half the price and as the 4 pins are on 4 separate 1 pin headers it can be used on a number of different devices as it doesn’t matter how the serial headers are pinned out. To connect to The BBBlack the black wire goes to PIN 1 (Ground), the green wire to PIN 4 (Receive) and the white wire goes to PIN 5 (Transmit). The red wire is power and isn’t needed. CircuitCo has a number of other Serial Cable Options listed and the appropriate configurations for them too.

Now for a serial console app on Fedora I usually use screen. So simply once you have your serial console connected to the BeagleBoneBlack just plug the usb port into you computer and run sudo screen /dev/ttyUSB0 115200 and then power up the Beagle Bone and you should soon see the output from u-boot and you’re on your way.

Nose back to the grindstone

Some may have noticed that I’ve not been around as much as usual in the last month, I’ve been on 4 weeks (originally meant to be 3) of holidays visiting my family back in Australia. It’d been 2.5 years since I’d last been home and well over a year since I’d had any form of break at all so I decided to disconnect myself as much as possible. I vaguely kept up to date on email replying to the urgent stuff and just deleting most stuff leaving the bits in between to deal with when I got back. So I’m now going through the large stack of outstanding stuff but if you’ve been expecting a response from me now you know why you’ve not had one and if you don’t get a response by the end of the week I suggest you poke me via IRC/Email again to follow me up. I have Fedora 20, Fedora ARM, Fedora in general and $dayjob on my list…. not necessarily in that order! I’m endeavouring to prioritise… we’ll see how that goes!

For those that don’t believe I can disconnect here’s a couple of photos for your viewing pleasure (click for full size) up on the family farm.