I was over-due my monthly patching on my KVM/QEMU host based on Ubuntu 16.10, so decided to diligently update packages and reboot last night… at least that was the plan.
After the host came back up I noted that all of my KVM-based workload lost network connectivity. Host <> VM was working, but VM<>VM and LAN<>VM were failing.
I confirmed the bridge config (br0) had survived the upgrade, that the bridge was up, and in fact showed the VMs being attached… so I started to dig deeper.
I then came across this guide on bridge setup: https://wiki.libvirt.org/page/Networking#Debian.2FUbuntu_Bridging – the main stand-out was this section:
Finally add the ‘/etc/sysctl.conf’ settings
net.bridge.bridge-nf-call-ip6tables = 0 net.bridge.bridge-nf-call-iptables = 0 net.bridge.bridge-nf-call-arptables = 0
Closely followed by:
To ensure that the bridge sysctl settings get loaded on boot, add this line to ‘/etc/rc.local’ just before the ‘exit 0’ line. This is a work around for Ubuntu bug #50093.
Also to stop Circumventing Path MTU Discovery issues with MSS Clamping
*** Sample rc.local file *** /sbin/sysctl -p /etc/sysctl.conf iptables -A FORWARD -p tcp --tcp-flags SYN,RST SYN -j TCPMSS --clamp-mss-to-pmtu exit 0
I had done neither of these steps previously, and the host had been running for a few months/ various updates without issue.
Now, getting rc.local working on Ubuntu 16.10 was… “interesting” to say the least, and even when working, rc.local executes too early for this to be a “clean” fix. I’ll share my steps, and eventual workaround, but it isn’t pretty!
The Workaround
Create service file for rc-local :
sudo vi /etc/systemd/system/rc-local.service
Contents as below:
[Unit] Description=/etc/rc.local Compatibility ConditionPathExists=/etc/rc.local #After=network.target [Service] type=forking ExecStart=/etc/rc.local start TimeoutSec=0 StandardOutput=tty RemainAfterExit=yes SysVStartPriority=99 [Install] WantedBy=multi-user.target
Now create the rc.local file:
vi /etc/rc.local
Contents as below (keep the header and the “sleep 10” line, without this the script will fail):
#!/bin/sh -e # # rc.local # # This script is executed at the end of each multiuser runlevel. # Make sure that the script will "exit 0" on success or any other # value on error. # # In order to enable or disable this script just change the execution # bits. # # By default this script does nothing. sleep 10 /sbin/sysctl -p /etc/sysctl.conf #echo "0" > /proc/sys/net/bridge/bridge-nf-call-iptables #echo "0" > /proc/sys/net/bridge/bridge-nf-call-ip6tables #echo "0" > /proc/sys/net/bridge/bridge-nf-call-arptables iptables -A FORWARD -p tcp --tcp-flags SYN,RST SYN -j TCPMSS --clamp-mss-to-pmtu exit 0
Enable execution on the rc.local file:
chmod +x /etc/rc.local
Now reload-daemons, enable rc-local and then, when finished, reboot:
systemctl daemon-reload systemctl enable rc-local