Hey, where did my VM LAN connectivity go?!

I was over-due my monthly patching on my KVM/QEMU host based on Ubuntu 16.10, so decided to diligently update packages and reboot last night… at least that was the plan.

After the host came back up I noted that all of my KVM-based workload lost network connectivity. Host <> VM was working, but VM<>VM and LAN<>VM were failing.

I confirmed the bridge config (br0) had survived the upgrade, that the bridge was up, and in fact showed the VMs being attached…  so I started to dig deeper.

I then came across this guide on bridge setup: https://wiki.libvirt.org/page/Networking#Debian.2FUbuntu_Bridging – the main stand-out was this section:

Finally add the ‘/etc/sysctl.conf’ settings

net.bridge.bridge-nf-call-ip6tables = 0
net.bridge.bridge-nf-call-iptables = 0
net.bridge.bridge-nf-call-arptables = 0

Closely followed by:

To ensure that the bridge sysctl settings get loaded on boot, add this line to ‘/etc/rc.local’ just before the ‘exit 0’ line. This is a work around for Ubuntu bug #50093.

Also to stop Circumventing Path MTU Discovery issues with MSS Clamping

 *** Sample rc.local file ***
 /sbin/sysctl -p /etc/sysctl.conf
 iptables -A FORWARD -p tcp --tcp-flags SYN,RST SYN -j TCPMSS  --clamp-mss-to-pmtu
 exit 0

I had done neither of these steps previously, and the host had been running for a few months/ various updates without issue.

Now, getting rc.local working on Ubuntu 16.10 was… “interesting” to say the least, and even when working, rc.local executes too early for this to be a “clean” fix. I’ll share my steps, and eventual workaround, but it isn’t pretty!

The Workaround

Create service file for rc-local :

 sudo vi /etc/systemd/system/rc-local.service

Contents as below:

[Unit]
Description=/etc/rc.local Compatibility
ConditionPathExists=/etc/rc.local
#After=network.target

[Service]
type=forking
ExecStart=/etc/rc.local start
TimeoutSec=0
StandardOutput=tty
RemainAfterExit=yes
SysVStartPriority=99

[Install]
WantedBy=multi-user.target

Now create the rc.local file:

vi /etc/rc.local

Contents as below (keep the header and the “sleep 10” line, without this the script will fail):

#!/bin/sh -e
#
# rc.local
#
# This script is executed at the end of each multiuser runlevel.
# Make sure that the script will "exit 0" on success or any other
# value on error.
#
# In order to enable or disable this script just change the execution
# bits.
#
# By default this script does nothing.

sleep 10
/sbin/sysctl -p /etc/sysctl.conf
#echo "0" > /proc/sys/net/bridge/bridge-nf-call-iptables
#echo "0" > /proc/sys/net/bridge/bridge-nf-call-ip6tables
#echo "0" > /proc/sys/net/bridge/bridge-nf-call-arptables

iptables -A FORWARD -p tcp --tcp-flags SYN,RST SYN -j TCPMSS --clamp-mss-to-pmtu
exit 0

Enable execution on the rc.local file:

chmod +x /etc/rc.local

Now reload-daemons, enable rc-local and then, when finished, reboot:

systemctl daemon-reload
systemctl enable rc-local