Playing with Bluetooth Smart devices on Fedora

So I have a few Bluetooth Smart (AKA BT Low Energy or BT-LE) devices including a BeeWi SmartClim Smart Temperature & Humidity Sensor, a TI SensorTag2 and a Runtasic Orbit activity tracker. I thought I’d see if I could connect to them with something other than their proprietary apps that run on my phone and get anything useful out of them.

To do this I installed a Fedora 23 minimal image on a micro SD card for BeagleBone Black, inserted a CSR USB Bluetooth 4.0 dongle and powered it up. Once I’d completed the first boot wizard and logged in I installed the bluetooth command line utility packages with a “sudo dnf install -y bluez bluez-libs”. You don’t need to use a BeagleBone, you can use any ARM device or even a standard laptop which supports Bluetooth 4+ with BT-LE support.

Now that we have a device running we’re ready to start to play.

First lets see if our system sees our bluetooth dongle:

$ sudo hcitool dev
	hci0	AA:BB:CC:DD:EE:33

Then lets scan for any Bluetooth Smart devices that are in range:

$ sudo hcitool lescan 
LE Scan ...
A1:B2:C3:D4:E5:F6 (unknown)
A1:B2:C3:D4:E5:F6 CC2650 SensorTag
A9:B8:C7:D6:E5:F4 (unknown)
A9:B8:C7:D6:E5:F4 BeeWi SmartClim

Now create a connection to the device we discovered:

$ sudo hcitool lecc A1:B2:C3:D4:E5:F6
Connection handle 3586

In the bluez package gatttool is a tool we can use to interact with Bluetooth Smart devices. We can use gatttool in interactive mode to send commands to out previously scanned device address:

sudo gatttool -b A1:B2:C3:D4:E5:F6 -I

Now we connect to the deivce (note in older version of gattool the prefix would change from “[ ]” to “[CON]” now it changes the colour of the BT MAC):

[A1:B2:C3:D4:E5:F6][LE]> connect
Attempting to connect to A1:B2:C3:D4:E5:F6
Connection successful

We can request the device characteristics:

[A1:B2:C3:D4:E5:F6][LE]> characteristics
handle: 0x0002, char properties: 0x02, char value handle: 0x0003, uuid: 00002a00-0000-1000-8000-00805f9b34fb
handle: 0x0004, char properties: 0x02, char value handle: 0x0005, uuid: 00002a01-0000-1000-8000-00805f9b34fb
handle: 0x0006, char properties: 0x02, char value handle: 0x0007, uuid: 00002a04-0000-1000-8000-00805f9b34fb
handle: 0x0009, char properties: 0x20, char value handle: 0x000a, uuid: 00002a05-0000-1000-8000-00805f9b34fb
handle: 0x000d, char properties: 0x02, char value handle: 0x000e, uuid: 00002a23-0000-1000-8000-00805f9b34fb
handle: 0x000f, char properties: 0x02, char value handle: 0x0010, uuid: 00002a24-0000-1000-8000-00805f9b34fb
handle: 0x0011, char properties: 0x02, char value handle: 0x0012, uuid: 00002a25-0000-1000-8000-00805f9b34fb
handle: 0x0013, char properties: 0x02, char value handle: 0x0014, uuid: 00002a26-0000-1000-8000-00805f9b34fb
handle: 0x0015, char properties: 0x02, char value handle: 0x0016, uuid: 00002a27-0000-1000-8000-00805f9b34fb
handle: 0x0017, char properties: 0x02, char value handle: 0x0018, uuid: 00002a28-0000-1000-8000-00805f9b34fb
handle: 0x0019, char properties: 0x02, char value handle: 0x001a, uuid: 00002a29-0000-1000-8000-00805f9b34fb
handle: 0x001b, char properties: 0x02, char value handle: 0x001c, uuid: 00002a2a-0000-1000-8000-00805f9b34fb
handle: 0x001d, char properties: 0x02, char value handle: 0x001e, uuid: 00002a50-0000-1000-8000-00805f9b34fb
handle: 0x0020, char properties: 0x12, char value handle: 0x0021, uuid: f000aa01-0451-4000-b000-000000000000
handle: 0x0023, char properties: 0x0a, char value handle: 0x0024, uuid: f000aa02-0451-4000-b000-000000000000
handle: 0x0025, char properties: 0x0a, char value handle: 0x0026, uuid: f000aa03-0451-4000-b000-000000000000
handle: 0x0028, char properties: 0x12, char value handle: 0x0029, uuid: f000aa21-0451-4000-b000-000000000000
handle: 0x002b, char properties: 0x0a, char value handle: 0x002c, uuid: f000aa22-0451-4000-b000-000000000000
handle: 0x002d, char properties: 0x0a, char value handle: 0x002e, uuid: f000aa23-0451-4000-b000-000000000000
handle: 0x0030, char properties: 0x12, char value handle: 0x0031, uuid: f000aa41-0451-4000-b000-000000000000
handle: 0x0033, char properties: 0x0a, char value handle: 0x0034, uuid: f000aa42-0451-4000-b000-000000000000
handle: 0x0035, char properties: 0x0a, char value handle: 0x0036, uuid: f000aa44-0451-4000-b000-000000000000
handle: 0x0038, char properties: 0x12, char value handle: 0x0039, uuid: f000aa81-0451-4000-b000-000000000000
handle: 0x003b, char properties: 0x0a, char value handle: 0x003c, uuid: f000aa82-0451-4000-b000-000000000000
handle: 0x003d, char properties: 0x0a, char value handle: 0x003e, uuid: f000aa83-0451-4000-b000-000000000000
handle: 0x0040, char properties: 0x12, char value handle: 0x0041, uuid: f000aa71-0451-4000-b000-000000000000
handle: 0x0043, char properties: 0x0a, char value handle: 0x0044, uuid: f000aa72-0451-4000-b000-000000000000
handle: 0x0045, char properties: 0x0a, char value handle: 0x0046, uuid: f000aa73-0451-4000-b000-000000000000
handle: 0x0048, char properties: 0x10, char value handle: 0x0049, uuid: 0000ffe1-0000-1000-8000-00805f9b34fb
handle: 0x004d, char properties: 0x0a, char value handle: 0x004e, uuid: f000aa65-0451-4000-b000-000000000000
handle: 0x004f, char properties: 0x0a, char value handle: 0x0050, uuid: f000aa66-0451-4000-b000-000000000000
handle: 0x0052, char properties: 0x1a, char value handle: 0x0053, uuid: f000ac01-0451-4000-b000-000000000000
handle: 0x0055, char properties: 0x0a, char value handle: 0x0056, uuid: f000ac02-0451-4000-b000-000000000000
handle: 0x0057, char properties: 0x0a, char value handle: 0x0058, uuid: f000ac03-0451-4000-b000-000000000000
handle: 0x005a, char properties: 0x12, char value handle: 0x005b, uuid: f000ccc1-0451-4000-b000-000000000000
handle: 0x005d, char properties: 0x08, char value handle: 0x005e, uuid: f000ccc2-0451-4000-b000-000000000000
handle: 0x005f, char properties: 0x08, char value handle: 0x0060, uuid: f000ccc3-0451-4000-b000-000000000000
handle: 0x0062, char properties: 0x1c, char value handle: 0x0063, uuid: f000ffc1-0451-4000-b000-000000000000
handle: 0x0066, char properties: 0x1c, char value handle: 0x0067, uuid: f000ffc2-0451-4000-b000-000000000000

Request the name of the device by querying the appropriate characteristic by hhandle:

[A1:B2:C3:D4:E5:F6][LE]> char-read-hnd 0x3
Characteristic value/descriptor: 53 65 6e 73 6f 72 54 61 67 20 32 2e 30 

The response is returned as a hexadecimal string. To make it readable we need to convert it to ASCII with some simple python:

Python 2.7.10 (default, Sep  8 2015, 17:20:17) 
[GCC 5.1.1 20150618 (Red Hat 5.1.1-4)] on linux2
Type "help", "copyright", "credits" or "license" for more information.
>>> "53656e736f7254616720322e30".decode("hex")
'SensorTag 2.0'

So that’s the basics of playing with Bluetooth Smart on the command line. I want to work out all of the characteristics and write a simple python daemon to poll the device and then write the output to different locations depending on the config like to a text file or to a MQTT message bus.

Getting IoT kick started on Fedora

So a number of people have been discussing the Internet of Things on Fedora for some time. We now have a Fedora IoT mailing list where these discussions can be more centralised and directed.

So where and how do we get started here? I’m going to kick start some ideas here and repost it as a mail to the list so we can use it as a basis to start the discussion.

As I outlined in my Using Fedora as a base for the IoT revolution talk at Flock there’s a lot of use cases and components that make up a complete IoT stack. I think initially we should focus on two initial goals rather than biting off too much:

  • A IoT internet gateway device
  • A IoT sensors endpoint device

The general idea here is that both of the above would be a very minimal shared build, likely using atomic images to enable easy update/rollback with some specific components for each use case. Initially I suggest we focus on a single, or maybe a couple, of specific devices to limit the scope to something more achievable and to add features as we go.

IoT internet gateway device specs and features

  • Wired and/or wireless ethernet to provide internet connectivity
  • Bluetooth Smart (AKA LE)
  • Thread Stack support (6LoWPAN and friends)
  • 802.15.4 support
  • MQTT Broker support (not standard for a IoT GW but enables easier localised testing)
  • MQTT Client
  • Atomic support: updates, rollback etc
  • Works with both our endpoint below and other IoT OSes such as Contiki

IoT internet sensors endpoint specs and features

  • Wired or wireless ethernet IP support
  • Bluetooth Smart (AKA LE)
  • Equivalent to Thread Stack support (6LoWPAN and friends)
  • MQTT Broker support (not standard for a IoT GW but enables easier testing
  • MQTT Client
  • CoAP client
  • Atomic support: updates, rollback etc
  • Support for various inputs and outputs and sensors

I have no doubt missed a lot of details in the above use cases, it’s somewhere to start. I think we also need to look at tools like Node-RED and tools for managing the devices. IoT is a big topic, the idea is we need to get the conversation start somewhere. I’ll look forward to seeing you all on the list to do that.

Flock Rochester

I’m not going to do a day by day outline of what I did at flock, if I did it would basically be “blah blah blah I talked a lot to a lot of people about a lot of tech topics” and anyone that’s ever met me would have guessed that! It was, as in the past, a great conference. A big shout out to the organisers for an excellent event with two excellent evening events! So I’m going to give a brief summary to my talks and link to slides and video recordings.

My first talk was an overview of the state of aarch64 and POWER as secondary architectures. The slides aren’t particularly interesting as they’re just words for discussion points. The video has all the interesting bits. A related talk was Dennis’s Standardising ARMv7 booting with a memorial quote by Jon Masters 😉

My second talk was about using Fedora as a base for IoT. Slides are here but the talk was quite a bit different to the slides and is more interesting so I suggest watching the video.

I also actively participated in Dennis’s Fedora Release Engineering going forward because well obviously I’m part of it 😉 and it was interesting for where we’re going, and even where we’ve come from in the last year or so 🙂

Finally I loved the Keynote Be an inspiration, not an impostor by Major Hayden. He’s published a follow up blog post with a FAQ too.

The least memorable bit was the terrible Amtrak ride back to New York City. On the plus side it makes the worst of the British National Rail service seem amazingly on time! NEVER AGAIN!

Fedora 23 on the Thinkpad X1 Carbon gen 3

So my corporate laptop for the last three years has been a Thinkpad x220, it’s getting a bit long in the teeth, when I got it the x230 was already out but the corporate standard hadn’t rebased. This time I happened to get a new device just after the new corporate refresh so I have the shiny X1 Carbon gen 3 with real track pad buttons.

Of course I grabbed the latest rawhide nightly boot.iso to install the latest shiny! How did I get on? Initially it was disaster, the kernel crashed before I even got to anaconda (will spend some time to recreate and log that soon). Oops! So I grabbed a couple of the earlier installers and with a bit of trial and error before long I was at the initial anaconda screen. From there the process was relatively boring! Up and running with Fedora Workstation installed, disk encrypted I started to play with it to see which bits were good, bad or ugly.

So first up with the good. Over all the vast majority of the hardware just worked out of the box. The GPU, USB3, wired ethernet, wireless ethernet (iwl 7265 AC), trackpad/thumb pointer, bluetooth mouse, camera, onboard speakers all work just fine, I even managed to enrol my finger in the finger print reader without issue. I must say I’m loving the 1920*1080 screen over the old 1366*768 of the x220 and I’m getting use to the chiclet keyboard layout. Overall F-23/rawhide is solid right out of the gate with pretty much everything working as you’d expect on a stable release. 🙂

It’s not all rosy though and some of the bad is that I’ve not managed to get the firmware upgraded to support 4K displays @ 60hz, thanks to Major and Sandro for the heads up on this, but it never seems to find the usb stick as a bootable image. Not a major issue in the short term as the OneLink Pro Dock I ordered is AWOL but I’ll want it soon as I’ve got a standing desk with a Dell 24 inch 4K monitor. The x220 and 4K monitor had never really worked overly well, the big issue was it only ran at 30hz refresh but I knew that when I got the monitor because I knew I was due for a refresh soon, it also had random glitches. I also need to workout how to adjust the acceleration of the thumb pointer thingy with libinput. Adjusting either the trackpad or mouse options in control panel doesn’t seem to have any effect, I never really used the trackpad on the x220 but I admit I’m getting use to the two finger scrolling for reading long pages. This should all be relatively easy to solve with a bit of poking!

The ugly seems to be the stability of the iwl 7265 wireless driver/firmware. It generally works but regularly shits itself. Some times minor by dropping SSH connections, sometimes majorly resulting in a need to unload/reload the modules or even to reboot! URGH! I’ve heard people complain about recent Intel Wireless stability but the “Advanced-N 6205 [Taylor Peak] (rev 34)” that was in the x220 was always solid. The solid lockups seems to be when pushing a reasonable amount of data via rsync/ssh. I do have a WRT1900AC router and I’ve connected to the 5 ghz 11ac so I’m wondering whether this combo is part of the issues. There is a newer firmware that hasn’t made it upstream yet. I need to do some more playing here testing the 11n 2.4ghz network as well as testing the newer firmware and possibly some patches that are on the wireless mailing list which I’m hoping will actually just land in 4.2 before long 🙂

There’s a few other things I need to play with some more. I’ve not tried external HDMI video/audio, the external display port, the DP to VGA converter (nice one was included in the box though) or the headphone/mic socket. None of the functionality of the dock has been tested yet simply because it’s yet to show up! It reports around 7.5 hours of battery life but I want to look at what the state of power consumption on these devices after mjg59’s post about it to see if I can’t get that well into the double digit hours.

Overall I’m pretty impressed with out of box experience of F-23/rawhide on the Thinkpad X1 Carbon gen 3 🙂

Fedora 22 on ARM and aarch64

So with Fedora 22 well and truly out for both ARMv7 and aarch64 lets have a look at the release in general and also at the 4.0 kernel it ships.

Firstly I’d like to shout out to the AMAZING job done by the web team on the new sites for Spins, ARM and Labs. They really do look awesome!

It’s all to easy to get bogged down in the actual technical components that make up the actual release and not forget that there’s work that goes on all over the place that contributes to making the release easy to use even before you begin the download.

Anyway! On to ARMv7. We shipped GA with the 4.0.4 kernel and u-boot 2015.01. This combination gives us improved support for numerous devices primarily through new DT support and improvements to drivers. The 4.0 kernel from an ARM HW support PoV really was a minor revision. With the fully packaged and updated fedora-arm-installer it’s even easier to get a device up and running.

We were hoping to get enough time to rebase this to 2015.04 but we just plain ran out of time, although we do have a plan to be able to update the u-boot when writing an image to card/stick without the need to regenerate the images. More details on the improvements we’re planning for fedora-arm-installer for another post!

From an aarch64 point of view the big change, although one an average user won’t notice, was we went from carrying a rather large (around 65K lines) enablement patch for the kernel to a small collection of 4 bug/problem specific patches! This is a massive change where in the F-21 cycle we had a gigantic architecture enablement patch! This makes it much more straight forward and less stressful for both myself as the architecture maintainer and the core Fedora kernel developers. In terms of the rest of aarch64 we still support VExpress, APM Storm platform primarily with the Mustang boards, and the AMD Seattle platform.

Overall the Fedora 22 release on the ARM platforms is a really nice release, there’s been some good changes there to enable easier and quicker updates in the future and easier means of adding decent support to new devices mid cycle. AArch64 is maturing and hopefully we’ll start to see some more platforms land and the architecture start to depart the niche status!

Cheap travel routers

In the past I’ve traveled for work, conferences and personal a lot. The first category has declined a little from the “travel all the time” I’ve done in the past since I’ve joined the release engineering team. When travelling regularly I have a collection of things I pack that make life easier and a little more pleasant.

On of these is a travel router, often hotels have wired but no wireless, or flaky wireless or you need to pay for more than one device (or even one device). I don’t have a high spec need but it must have the following:

  • Dual ethernet WAN/LAN (everyone knows I have lots of ARM devices!)
  • Wireless, doesn’t need to be massivly fast (hotel internet is the bottle neck!) but it does need to be stable
  • Standard power cables, either figure 8 plug with inbuilt switching PSU or micro USB
  • Small
  • Open source and hackable. Probably supportable with OpenWRT
  • Quick and easy to reconfigure

For a number of years the device I used was an Apple Airport Express as it was one of the few that met most of the above criteria. But then I stopped using a iPod touch which meant the last option was dead due to the lack of openness.

Time for a new one! I’ve been looking for a while, I almost went for the popular TP-Link WR703N router as it’s well supported in OpenWRT but it meant that I lost the extra wired network port.

Then I came across the NEXX WT3020. It comes in four official options all of which have 802.11n, dual 100Mb ethernet, and runs from a micro USB connector! All but the bottom end model have a USB port for 3G or storage. The top two models just seem to have SW options but no other HW. Perfect, it even has OpenWRT support! So I went with the WT3020H, all for around $20.

I ordered one, while I was awaiting for it to arrive the OpenWRT project released Chaos Calmer 15.05-rc1 which has a prebuilt image and literally in less than five minutes I’d reflashed it to OpenWRT via the standard web flash interface! I’ve not had time to test performance, throughput and features such as USB but it seems to work pretty well for a $20 router and all the core features I need are working and it’s in a form factor that is TINY and I don’t even need to take an extra power supply. I’d call that a WIN!

3.19 Fedora ARM kernel status

I’ve been a bit lazy on the ARM kernel status updates. There wasn’t one at all for 3.18 but the fact was, that while there was lots of under the hood improvements for ARM/aarch64, the new device support or improvements from a user’s point of view was positively boring so I never bothered!

That said the 3.19 kernel is now on it’s way to the stable Fedora releases and there’s some bit of interest there 🙂

Beginning with aarch64 there’s been a raft of code support landing upstream for the core platforms we support (VExpress, APM Storm, and AMD Seattle) which means the enablement patch set has shrunk massively. The core missing bit from this is primarily the ACPI patches for the server standards. There’s also been a lot of stability improvements for various device drivers particularly on the APM Storm SoC (which massively helps the high network and IO traffic we generate when doing composes in release engineering!). Other improvements include support for seccomp. The upstream support for aarch64 is really starting to settle down nicely which is good because there’s devices finally starting to get to the point where they’ll be more widely available and affordable 🙂

On to ARMv7 changes. In terms of new supported SoCs the support for AllWinner A-23 (aka sun8i) is the most interesting in terms of new devices. There was also a lot of general SoC improvements and cleanups. The largest here is probably Rockchips, QCom and ZYNQ with notable mentions to Tegra, OMAP and i.MX6 too. In terms of new devices we now solely support DeviceTree devices and the built .dtb files we ship that are possible to support with the kernel jumped from 250 to 265 devices. Of course it doesn’t mean we’re testing all of those devices but we’re testing devices across all main SoC groups to ensure at least the core support works. Of course feedback for what works and doesn’t is always welcome. In this cycle there was also significant driver work with special mention to Hans and his significant movement on the Allwinner devices.

I’ll do a longer post for 4.0 and the new u-boot we’ll be supporting in Fedora 22 soon.

Changing ssh ports on Fedora or RHEL

I always forget the exact commands to change the port ssh (or any default service in the case of the selinux bits) runs on. It’s nicely simple though!

Edit /etc/ssh/sshd_config to change the port number:

Port 2022

You can add a second line if you wish to initially leave it running on Port 22 too in case something goes wrong, obviously don’t forget to remove it once the new port is working!

To add port 2022 to port contexts, enter:
# semanage port -a -t ssh_port_t -p tcp 2022

You can verify new settings, enter:

# semanage port -l | grep ssh
Sample outputs:
ssh_port_t tcp 2022,22

Reload the sshd service to pick up the new config:
#systemctl reload sshd.service

And of course don’t forget to update your firewall to allow the new port through.

Standing desks

So around three months ago (yes, I must do a post on that too) I changed roles at Red Hat and moved from constantly travelling and being on customer sites to working from home. As a result I needed to setup a workspace that I could use day to day.

One thing I’ve always wanted to try is a standing desk. I have back problems, and generally not the best posture, so I thought that would be one way to be able to deal with at least the later, and potentially even the former. The main problem, until recently, is that decent standing desks tend to be very expensive and I didn’t want to needlessly go and spent a lot of extra money for something that would be used for a week and never again. So I decided I would start with a cheap height adjustable desk, which I needed to get anyway due to my height, and then use it as a the basis of a standing desk and then hack it from there. The initial combo I decided on after a lot of looking was the IKEA Galant Height Adjustable Desk at £49 and the IKEA Lack Side table at £8 plus delivery. I figured at less than £100 including delivery if it was terrible I wasn’t wasting a lot of money!

As it turns out it’s been much better than I ever expected it to be. I initially setup the desk to the height I would want when sitting. At a height of six foot three inches I’m not the shortest of people so when sitting I prefer a higher than average desk. Sitting the Lack table on top of the desk by chance also ended up also giving me the perfect standing height. Bonus! A few quid for some foam gym mats plus a decent height adjustable monitor (the most expensive bit by far!) and I was done! Well mostly, I still haven’t decided on a decent keyboard yet.

So how does it look? Well a little bit weird to be honest. How does it work? Better than I ever expected as I find I can happily stand at the desk for a full eight hour working day without too much issue and I’ve even done longer (hello Fedora beta release candidates!!) and my back feels better than it has in a long time! I was also trying to decide on a decent but reasonably priced office chair to buy but now I’m not going to bother. Interestingly IKEA has also just launched the BEKANT sit/stand desk which is reasonably priced and has electric motors for raise/lower. It’s likely I’ll end up getting one of these one day but for the moment my IKEA hack is working pretty well.

Cheap IKEA Standing Desk

network bridge for libvirt host with a single live IP

The default network config for libvirt is simple and works for most basic use cases but there’s a number of use cases where you need more complex config like Adam outlined for a local bridged config.

I run a number of VMs on a hosted server on the internet and I’ve had on my ToDo list for some time to add a IPSEC site to site VPN but the default network doesn’t make that easy because libvirtd deals with the iptables networking including the NAT automagically.

The network config looks like this:

                    *-----*  --| Hyp |-(eth0)- internet
 (br0) VM net       *-----*

Create non routed network bridge
Initially create a basic network bridge and disable STP (spanning tree protocol). Note we don’t bind it with eth0 which is a public internet facing interface.

nmcli con add type bridge ifname br0
nmcli con modify bridge-br0 bridge.stp no

I then edited the /etc/sysconfig/network-scripts/ifcfg-bridge-br0 file and to add IP network config and adjust a few bits to get the following:


Once we’ve done that we can bring the bridge online and check the config looks OK:

ifup br0
nmcli c show
nmcli -f bridge con show bridge-br0
ip addr

Now we have a network bridge with an IP address you can now edit any VM configuration and reassign the virtual NICs to the new bridge and adjust the VM network config to the new subnet and assign static IPs to each VM, or configure dhcpd to assign IPs on the br0 interface. Once that’s done you should be able to ping the gateway ( and have local network connectivity.

Once you’ve moved everything over you can delete the original libvirtd network config.

Outbound NATed networking
Using the traditional iptables.service (firewalld investigation is on my todo list) you can add a basic outbound NAT configuration which restores the last of the missing functionality with the following basic rule set which will NAT by masquerading the br0 network out through the public IP on eth0:

iptables -t nat -A POSTROUTING -o eth0 -j MASQUERADE
iptables -A INPUT -m state --state RELATED,ESTABLISHED -j ACCEPT
iptables -A INPUT -p icmp -j ACCEPT
iptables -A INPUT -i lo -j ACCEPT
iptables -A INPUT -m state --state NEW -p tcp -m tcp --dport 22 -j ACCEPT
iptables -A INPUT -j REJECT --reject-with icmp-host-prohibited
iptables -A FORWARD -i eth0 -o br0 -m state --state RELATED,ESTABLISHED -j ACCEPT
iptables -A FORWARD -i br0 -o eth0 -j ACCEPT
iptables -A FORWARD -j REJECT --reject-with icmp-host-prohibited
iptables-save > /etc/sysconfig/iptables
systemctl enable iptables.service
systemctl start iptables.service

With the network now under complete control of NetworkManager and a IP firewall/NAT configuration under control from a single point it now makes it easier to add things like IPSEC connections and IPv6 configuration both of which are next on the list.