Using nmcli to configure a static dual stack wired network interface

I recently managed to break the network on my VM that hosts this blog. Basically I removed the NetworkManager-initscripts-ifcfg-rh package because I don’t use the old style ifcfg configuration anywhere else and I had forgotten how long I’d had this VM. So I went into the web console, manually bought up the network with ip commands and reinstalled the package but it made no difference. Oh well! Time to just move it to the new config so I just worked out the nmcli options for all the bits in the old ifcfg. This VM network is nothing special, it’s basically dual IPv4/IPv6 interface with associated DNS.

Step 1: Show existing connections:

$ sudo nmcli c
NAME  UUID                                  TYPE      DEVICE 
eth0  a603bba7-fad8-3c71-9d4c-2cd5dc50e114  ethernet  eth0   

Step 2: Delete existing connection:

$ sudo nmcli c del a603bba7-fad8-3c71-9d4c-2cd5dc50e114

Step 3: Create a new connection (Note the IP addresses are random, the DNS servers are the Google public ones):

$ sudo nmcli c add type ethernet ifname eth0 con-name eth0 mac 80:00:00:ab:cd:ef ip4 192.168.10.6/24 gw4 192.168.10.1 ip6 fe80::b257:377c:e7b3:29ed/64 gw6 2A03:B0C0:0003:00D0:0000:0000:0000:0001 ipv4.dns "8.8.8.8 8.8.4.4" ipv6.dns "2001:4860:4860::8888 2001:4860:4860::8844"

Now the blog is back! The new connection is stored in /etc/NetworkManager/system-connections/eth0.nmconnection

Increasing a libvirt/KVM virtual machine disk capacity

There’s a bunch of howto’s on the internet for increasing the size of a virtual disk of a VM. Of course the best is to use the very useful libguestfs-tools options but there’s been some improvement in tools like sfdisk so I thought I’d document what I did for reference using tools I already had installed.

First shutdown the VM. Once it’s shutdown you need to work out where the disk is located. As this VM is running from my local machine and is just using a raw disk this is straight forward. You can get the details from the virt-manager GUI or virsh dumpxml VM-Name.

Next up we use qemu-img, it’s installed by default with the libvirt stack, to add the extra space we need, in theory this can be done with the VM online, this is a random test VM so online time doesn’t matter, and of course if the VM matters to you there should be a proper backup done first! The fdisk isn’t necessary, it just allows you to see that the extra space is there.

# qemu-img resize /var/lib/libvirt/images/VM-Name.raw +4G
# fdisk -l /var/lib/libvirt/images/VM-Name.raw
Disk /var/lib/libvirt/images/VM-Name.raw: 8 GiB, 8589934592 bytes, 16777216 sectors
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disklabel type: dos
Disk identifier: 0xe8b201aa

Device                                            Boot   Start     End Sectors Size Id Type
/var/lib/libvirt/images/VM-Name.raw1 *       2048 2099199 2097152   1G 83 Linux
/var/lib/libvirt/images/VM-Name.raw2      2099200 8388607 6289408   3G 83 Linux
#

Now power up the VM, login as root (or use sudo) for the next bits on the VM. The sfdisk tool has had a bunch of improvements over the last few years for partitioning. If you’ve not used it or looked at it recently I recommend checking the well written man page. Here I’m just expanding last partition (partition 2) on the disk to the maximum size the disk offers. For all the other possibilities “man sfdisk” is your friend!

# echo ", +" | sfdisk -N 2 /dev/vda --no-reread
# partprobe
# resize2fs /dev/vda2

And with that you should be good to go, df and friends will show you the new space, no reboot needed! The VM I have here is very basic partitions, no LVM etc so straight forward, if you have LVM there’s lots of docs on how to deal with that elsewhere.

Configuring HTTP/2 with Apache on Fedora

HTTP/2 is the new version of the well known HTTP protocol which has been at the venerable 1.1 since late last century. Version 2 was derived out of Google’s SPDY protocol and it’s a binary protocol over the text based 1.1. It introduces a bunch of improvements including reducing latency, multiplexing, and server push. There’s some useful improvements that will be great for things like apps that use WebSockets. The Apache httpd daemon has included complete support for HTTP/2 since the 2.4.17 release in the form of mod_http2.

First you should configure your site with SSL, I suggest using LetsEncrypt/certbot as documented in this Fedora Magazine article.

Then you need to make sure the module is loaded, at least in Fedora 25 this is enabled in /etc/httpd/conf.modules.d/00-base.conf by default:

LoadModule http2_module modules/mod_http2.so

Then you just need to enable the protocol in either the general configuration or in specific VirtualHost directives for specific sites:

# for a https server
Protocols h2 http/1.1

# for a http server
Protocols h2c http/1.1

Then it’s just a systemctl restart httpd to make the changes take effect.

To test whether you’re serving over HTTP/2 you can test using this HTTP/2 testing site or with the OpenSSL client (check for “ALPN protocol: h2” in the output) with the following command:

openssl s_client -alpn h2 -connect HOSTNAME:443

Note: HTTP/2 is not currently supported in the httpd shipped in RHEL.

Changing ssh ports on Fedora or RHEL

I always forget the exact commands to change the port ssh (or any default service in the case of the selinux bits) runs on. It’s nicely simple though!

Edit /etc/ssh/sshd_config to change the port number:

Port 2022

You can add a second line if you wish to initially leave it running on Port 22 too in case something goes wrong, obviously don’t forget to remove it once the new port is working!

To add port 2022 to port contexts, enter:
# semanage port -a -t ssh_port_t -p tcp 2022

You can verify new settings, enter:

# semanage port -l | grep ssh
Sample outputs:
ssh_port_t tcp 2022,22

Reload the sshd service to pick up the new config:
#systemctl reload sshd.service

And of course don’t forget to update your firewall to allow the new port through.

network bridge for libvirt host with a single live IP

The default network config for libvirt is simple and works for most basic use cases but there’s a number of use cases where you need more complex config like Adam outlined for a local bridged config.

I run a number of VMs on a hosted server on the internet and I’ve had on my ToDo list for some time to add a IPSEC site to site VPN but the default network doesn’t make that easy because libvirtd deals with the iptables networking including the NAT automagically.

The network config looks like this:

                    *-----*
192.168.100.0/24  --| Hyp |-(eth0)- internet
 (br0) VM net       *-----*

Create non routed network bridge
Initially create a basic network bridge and disable STP (spanning tree protocol). Note we don’t bind it with eth0 which is a public internet facing interface.

nmcli con add type bridge ifname br0
nmcli con modify bridge-br0 bridge.stp no

I then edited the /etc/sysconfig/network-scripts/ifcfg-bridge-br0 file and to add IP network config and adjust a few bits to get the following:

DEVICE=br0
STP=no
TYPE=Bridge
BOOTPROTO=static
DEFROUTE=yes
IPV4_FAILURE_FATAL=no
IPADDR=192.168.100.254
NETMASK=255.255.255.0
IPV6INIT=no
IPV6_FAILURE_FATAL=no
NAME=bridge-br0
UUID=
ONBOOT=yes

Once we’ve done that we can bring the bridge online and check the config looks OK:

ifup br0
nmcli c show
nmcli -f bridge con show bridge-br0
ip addr

Now we have a network bridge with an IP address you can now edit any VM configuration and reassign the virtual NICs to the new bridge and adjust the VM network config to the new subnet and assign static IPs to each VM, or configure dhcpd to assign IPs on the br0 interface. Once that’s done you should be able to ping the gateway (192.168.101.254) and have local network connectivity.

Once you’ve moved everything over you can delete the original libvirtd network config.

Outbound NATed networking
Using the traditional iptables.service (firewalld investigation is on my todo list) you can add a basic outbound NAT configuration which restores the last of the missing functionality with the following basic rule set which will NAT by masquerading the br0 network out through the public IP on eth0:

iptables -t nat -A POSTROUTING -o eth0 -j MASQUERADE
iptables -A INPUT -m state --state RELATED,ESTABLISHED -j ACCEPT
iptables -A INPUT -p icmp -j ACCEPT
iptables -A INPUT -i lo -j ACCEPT
iptables -A INPUT -m state --state NEW -p tcp -m tcp --dport 22 -j ACCEPT
iptables -A INPUT -j REJECT --reject-with icmp-host-prohibited
iptables -A FORWARD -i eth0 -o br0 -m state --state RELATED,ESTABLISHED -j ACCEPT
iptables -A FORWARD -i br0 -o eth0 -j ACCEPT
iptables -A FORWARD -j REJECT --reject-with icmp-host-prohibited
iptables-save > /etc/sysconfig/iptables
systemctl enable iptables.service
systemctl start iptables.service

With the network now under complete control of NetworkManager and a IP firewall/NAT configuration under control from a single point it now makes it easier to add things like IPSEC connections and IPv6 configuration both of which are next on the list.

using ssh keys with screen

It always annoyed me I couldn’t use my ssh key in a screen session. Every now and again I would try and work it out with google and some trial and error. Eventually with the help of a couple of good bits off the net I worked out what I thought to be the easiest way to achieve it consistently.

Firstly the ssh config bits:

Add the following to your ~/.ssh/config file, creating it if you don’t already have one:

host *
  ControlMaster auto
  ControlPath ~/.ssh/master-%r@%h:p

And create the ~/.ssh/rc file:

#!/bin/bash
if test "$SSH_AUTH_SOCK" ; then
    ln -sfv $SSH_AUTH_SOCK ~/.ssh/ssh_auth_sock
fi

And make sure they have the correct permissions for ssh:

chmod 600 ~/.ssh/config ~/.ssh/rc

Finally add the following to your ~/.screenrc file:

setenv SSH_AUTH_SOCK $HOME/.ssh/ssh_auth_sock

I’m not sure it’s the best and most effective way but it’s nice and simple and to date it’s been working well for me, I’ve not had issues with it. Any suggestions for improvement feel free to comment.

Disable SSLv3 in Dovecot

Disabling SSLv3 in Dovecot is nice and straight forward.

In the /etc/dovecot/conf.d/10-ssl.conf file edit the ssl_cipher_list line to look as below (or adjust to suit your specific requirements):

ssl_cipher_list = ALL:!ADH:!LOW:!SSLv2:!SSLv3:!EXP:!aNULL:+HIGH:+MEDIUM

To test the option to ensure it’ll work you can run the following command before you restart dovecot and the output should look something like below:

$ openssl ciphers -v 'ALL:!ADH:!LOW:!SSLv2:!SSLv3:!EXP:!aNULL:+HIGH:+MEDIUM'
ECDHE-RSA-AES256-GCM-SHA384 TLSv1.2 Kx=ECDH     Au=RSA  Enc=AESGCM(256) Mac=AEAD
ECDHE-ECDSA-AES256-GCM-SHA384 TLSv1.2 Kx=ECDH     Au=ECDSA Enc=AESGCM(256) Mac=AEAD
ECDHE-RSA-AES256-SHA384 TLSv1.2 Kx=ECDH     Au=RSA  Enc=AES(256)  Mac=SHA384
ECDHE-ECDSA-AES256-SHA384 TLSv1.2 Kx=ECDH     Au=ECDSA Enc=AES(256)  Mac=SHA384
DHE-DSS-AES256-GCM-SHA384 TLSv1.2 Kx=DH       Au=DSS  Enc=AESGCM(256) Mac=AEAD
DHE-RSA-AES256-GCM-SHA384 TLSv1.2 Kx=DH       Au=RSA  Enc=AESGCM(256) Mac=AEAD
DHE-RSA-AES256-SHA256   TLSv1.2 Kx=DH       Au=RSA  Enc=AES(256)  Mac=SHA256
DHE-DSS-AES256-SHA256   TLSv1.2 Kx=DH       Au=DSS  Enc=AES(256)  Mac=SHA256
ECDH-RSA-AES256-GCM-SHA384 TLSv1.2 Kx=ECDH/RSA Au=ECDH Enc=AESGCM(256) Mac=AEAD
ECDH-ECDSA-AES256-GCM-SHA384 TLSv1.2 Kx=ECDH/ECDSA Au=ECDH Enc=AESGCM(256) Mac=AEAD
ECDH-RSA-AES256-SHA384  TLSv1.2 Kx=ECDH/RSA Au=ECDH Enc=AES(256)  Mac=SHA384
ECDH-ECDSA-AES256-SHA384 TLSv1.2 Kx=ECDH/ECDSA Au=ECDH Enc=AES(256)  Mac=SHA384
AES256-GCM-SHA384       TLSv1.2 Kx=RSA      Au=RSA  Enc=AESGCM(256) Mac=AEAD
AES256-SHA256           TLSv1.2 Kx=RSA      Au=RSA  Enc=AES(256)  Mac=SHA256
ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH     Au=RSA  Enc=AESGCM(128) Mac=AEAD
ECDHE-ECDSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH     Au=ECDSA Enc=AESGCM(128) Mac=AEAD
ECDHE-RSA-AES128-SHA256 TLSv1.2 Kx=ECDH     Au=RSA  Enc=AES(128)  Mac=SHA256
ECDHE-ECDSA-AES128-SHA256 TLSv1.2 Kx=ECDH     Au=ECDSA Enc=AES(128)  Mac=SHA256
DHE-DSS-AES128-GCM-SHA256 TLSv1.2 Kx=DH       Au=DSS  Enc=AESGCM(128) Mac=AEAD
DHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=DH       Au=RSA  Enc=AESGCM(128) Mac=AEAD
DHE-RSA-AES128-SHA256   TLSv1.2 Kx=DH       Au=RSA  Enc=AES(128)  Mac=SHA256
DHE-DSS-AES128-SHA256   TLSv1.2 Kx=DH       Au=DSS  Enc=AES(128)  Mac=SHA256
ECDH-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH/RSA Au=ECDH Enc=AESGCM(128) Mac=AEAD
ECDH-ECDSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH/ECDSA Au=ECDH Enc=AESGCM(128) Mac=AEAD
ECDH-RSA-AES128-SHA256  TLSv1.2 Kx=ECDH/RSA Au=ECDH Enc=AES(128)  Mac=SHA256
ECDH-ECDSA-AES128-SHA256 TLSv1.2 Kx=ECDH/ECDSA Au=ECDH Enc=AES(128)  Mac=SHA256
AES128-GCM-SHA256       TLSv1.2 Kx=RSA      Au=RSA  Enc=AESGCM(128) Mac=AEAD
AES128-SHA256           TLSv1.2 Kx=RSA      Au=RSA  Enc=AES(128)  Mac=SHA256

Finally to test it from client side you can run the following command to ensure it’s not enabled. If it’s configured as expected the negotiation should fail and it’ll return you straight to the command prompt:

openssl s_client -connect mail.example.com:993 -ssl3

puppetmaster on apache with passenger in 5 mins

Config management, DevOPs and all those sorts of time saving things are all well and good for readily repeatable tasks that happen on the same systems on a regular basis. But as a consultant my tasks are rarely the same and change from day to day depending on the customer, their mood or the phase of the moon. So while I’ve used puppet a lot I’ve only deployed a puppet platform once or twice and every time I do I end up digging around trying to work out the latest best practice.

So primarily to remind myself so I have a quick guide for the next time here’s my quick puppetmaster in five minutes using just rpm packages (so they’re easily upgraded and checked for security issues etc) on RHEL-6 or Fedora.

First ensure you have proper forward and reverse DNS records for the puppet master server and all puppet clients.

  1. Set SELinux to Permissive. This is a FIXME as there appears to be an issue running puppetmaster on rails on apache. A quick check it seems to be related to comms between httpd and rails. RHBZ 730837 has more info but I need to investigate but it should be straight forward .
    sed 's/SELINUX=enforcing/SELINUX=permissive/' /etc/sysconfig/selinux
    setenforce permissve
    
  2. On Fedora you already have all the repos required out of the box. On RHEL I’m using the upstream PuppetLabs repos so I used the following:
    rhn-channel -a -c rhel-x86_64-server-optional-6
    rpm -ivh http://yum.puppetlabs.com/el/6/products/x86_64/puppetlabs-release-6-10.noarch.rpm
    rpm -ivh http://dl.fedoraproject.org/pub/epel/6/x86_64/epel-release-6-8.noarch.rpm
    yum-config-manager --enable epel puppetlabs-products puppetlabs-deps
    
  3. Install the needed packages
    yum install puppet-server httpd mod_ssl rubygem-rack mod_passenger
    
  4. Start the puppetmaster and set it to run on startup. While we won’t be using WEBrick we start it initially so it will initialise the default puppet config and setup things like certificates.
    service puppetmaster start
    puppet resource service puppetmaster ensure=running enable=true
    
  5. Configure the puppetmaster Rails app
    mkdir -p /usr/share/puppet/rack/puppetmasterd
    mkdir /usr/share/puppet/rack/puppetmasterd/public /usr/share/puppet/rack/puppetmasterd/tmp
    cp /usr/share/puppet/ext/rack/config.ru /usr/share/puppet/rack/puppetmasterd/
    chown -R puppet.puppet /usr/share/puppet/rack
    restorecon -Rv /usr/share/puppet
    mkdir /var/run/passenger
    restorecon -Rv /var/run/passenger
    
  6. Configure apache by adding the config below and updating FQDN.pem to your local certificate name.
    vim /etc/httpd/conf.d/puppetmaster.conf

    # RHEL/CentOS:
    # And the passenger performance tuning settings:
    PassengerHighPerformance On
    PassengerUseGlobalQueue On
    # Set this to about 1.5 times the number of CPU cores in your master:
    PassengerMaxPoolSize 6
    # Recycle master processes after they service 1000 requests
    PassengerMaxRequests 1000
    # Stop processes if they sit idle for 10 minutes
    PassengerPoolIdleTime 600
    PassengerTempDir /var/run/passenger
    
    Listen 8140
    <VirtualHost *:8140>
        SSLEngine On
    
        # Only allow high security cryptography. Alter if needed for compatibility.
        SSLProtocol             All -SSLv2
        SSLCipherSuite          HIGH:!ADH:RC4+RSA:-MEDIUM:-LOW:-EXP
        SSLCertificateFile      /var/lib/puppet/ssl/certs/FQDN.pem
        SSLCertificateKeyFile   /var/lib/puppet/ssl/private_keys/FQDN.pem
        SSLCertificateChainFile /var/lib/puppet/ssl/ca/ca_crt.pem
        SSLCACertificateFile    /var/lib/puppet/ssl/ca/ca_crt.pem
        SSLCARevocationFile     /var/lib/puppet/ssl/ca/ca_crl.pem
        SSLVerifyClient         optional
        SSLVerifyDepth          1
        SSLOptions              +StdEnvVars +ExportCertData
    
        # These request headers are used to pass the client certificate
        # authentication information on to the puppet master process
        RequestHeader set X-SSL-Subject %{SSL_CLIENT_S_DN}e
        RequestHeader set X-Client-DN %{SSL_CLIENT_S_DN}e
        RequestHeader set X-Client-Verify %{SSL_CLIENT_VERIFY}e
    
        RackAutoDetect On
        DocumentRoot /usr/share/puppet/rack/puppetmasterd/public/
        <Directory /usr/share/puppet/rack/puppetmasterd/>
            Options None
            AllowOverride None
            Order Allow,Deny
            Allow from All
        </Directory>
    </VirtualHost>
    

    restorecon -Rv /etc/httpd/conf.d

  7. Disable the puppetmaster so it runs via Rails on Apache
    chkconfig puppetmaster off; service puppetmaster stop
    chkconfig httpd on; service httpd restart
    
  8. Setup a basic puppet config files:
    • autosign.conf

      vim /etc/puppet/autosign.conf

      *.infra.example.com
      
    • site.pp
      vim /etc/puppet/manifests/site.pp

      # /etc/puppet/manifests/site.pp
      
      import "nodes"
      
      # The filebucket option allows for file backups to the server
      filebucket { main: server => 'puppetmaster.infra.example.com' }
      
      # Set global defaults - including backing up all files to the main filebucket and adds a global path
      File { backup => main }
      Exec { path => "/usr/bin:/usr/sbin:/bin:/sbin" }
      
    • modules.pp
      vim /etc/puppet/manifests/modules.pp

      # /etc/puppet/manifests/modules.pp
      
      import "dns-client"
      import "ntp-client"
      
    • nodes.pp
      vim /etc/puppet/manifests/nodes.pp

      # /etc/puppet/manifests/nodes.pp
      
      node 'common' {
              include dns-client
              include ntp-client
      }
      
      node default inherits common {
      }
      
      node 'puppetmaster.infra.example.com' inherits common {
      }
      

    With the above you obviously need to have the dns-client and ntp-client modules in place in /etc/puppet/modules/ or update the config files to reflect the modules you have installed!

  9. Relabel all the various puppet directories to ensure we’re good with selinux:
    restorecon -Rv /etc/puppet
  10. Reboot to ensure it all works when it comes back online 😉

P.S. I’ll update this post once I’ve had time to work out the SELinux issues, pointers welcome in the comments below 🙂

RoundCube mail on RHEL/Fedora

I run my own mail server and for years I’ve used squirrelmail as it was a simple interface that just worked and I never really use many of the advanced web mail features anyway. The squirrelmail project hasn’t really advanced a whole lot though and while they do keep up the security fixes and make it work with the latest releases of php there’s been no real development in quite some time so with the move to a new hosted server I decided it was time for a change. I decided to go with RoundCube Mail. The instructions are identical for Fedora as well.

Initial Install
RoundCube is packaged in EPEL and while 0.8.6 isn’t the latest release the ability to “yum install” works for me.

Install the roundcubemail package, mysql and mod_nss for HTTPS (or mod_ssl if you prefer), I’m assuming here you already have a working imap/smtp server. So just a:

yum install roundcubemail mysql-server mod_nss

I plan to use a MySQL DB so to create that I did the following to create the DB and db user:

# mysql -u root -p
mysql> create database roundcubemail;
mysql> create user roundcube;
mysql> GRANT ALL PRIVILEGES ON roundcubemail.* TO roundcube@localhost IDENTIFIED BY 'changeme';
mysql> FLUSH PRIVILEGES;
mysql> quit
# mysql -u root -p roundcubemail < /usr/share/doc/roundcubemail-0.8.6/SQL/mysql.initial.sql

To configure RoundCube to access the database edit /etc/roundcubemail/db.inc.php:

$rcmail_config['db_dsnw'] = 'mysql://roundcube:changeme@localhost/roundcubemail';

To configure RoundCube for mail server settings edit /etc/roundcubemail/main.inc.php:

$rcmail_config['default_host'] = 'localhost';

The only config changes I made for the mod_nss was to change the default port from 8443 to the standard HTTPS port of 443 by editing /etc/httpd/conf.d/nss.conf

Configure RoundCube URL and various other apache config like enforcing HTTP edit /etc/httpd/conf.d/roundcubemail.conf:

Alias /webmail /usr/share/roundcubemail

    RewriteEngine on
    RewriteCond %{HTTPS} !=on
    RewriteRule (.*) https://%{HTTP_HOST}%{REQUEST_URI}

I'll likely tweak the apache config a little more but that ensures its running over SSL for now.

Finally restart apache to make it work 🙂 with a "service httpd restart"

Fixes to enable running SELinux in Enforcing mode
Initial on testing I wad getting an error in the apache logs about writing to logs. I figured this might be a SELinux error so I did a quick setenforce 0 to test my theory and I was right on.

[error] [client 192.168.100.1] PHP Warning:  Error writing to log file /var/log/roundcubemail/errors; Please check permissions in /usr/share/roundcubemail/program/include/main.inc on line 1965

To fix running in enforcing mode I needed to change two SELinux settings. The first was to set the log directory with http_log and the second was to allow httpd to connect to the network. Fixed easily with these two commands:

chcon -R system_u:object_r:httpd_log_t:s0 /var/log/roundcubemail
setsebool -P httpd_can_network_connect=1

Now I don't need to upset Major Hayden or make Dan weep 😉

Fixing the date display
The date column in the mail initially didn't display. Looking at the apache logs I needed to set the php date.timezone setting in /etc/php.ini. I have users in a number of timezones so I was a little concerned at first of chosing one in particular but it doesn't seem to make much difference. Just search for date.timezone in /etc/php.ini and your good to go.

Conclusion
I like my new mail setup. The migration has enabled me to clean up a number of things I've wanted to for some time and just never got around to it. All the commands are basically identical on Fedora or any other EL6 clones. Hopefully it will be useful for others, and of course feedback is welcome.