TL:DR: when testing Docker with “–icc=false” on Ubuntu Server 16.04.3 I found that br_netfilter was required but not configured by default. Even when enabled, I found that the Docker Host physical network was not protected against container breakout. Testing with IP Masquerade disabled addressed Docker Host physical network security, however, with ICC and IP Masquerade disabled it was just as “easy” to manage the environment with “–iptables=false” and a firewall script.
I recently ran through an exercise where I was testing Docker Container Network Isolation in a lab environment – this involved reviewing the impact of disabling ICC, IP Masquerade and Docker’s interaction with IP tables itself. The scenarios I was trying to provide isolation for are as follows:
- Docker Container access to Docker Containers within the same Docker network
- Docker Container access to other Docker Containers in different Docker network
- Docker Container access to Docker host
- Docker Container access to Docker host physical network/ other hosts
Disabling Inter-Container Connectivity (ICC)
By default there is limited Docker Container Network Isolation in that docker containers have external network access, can talk to each other and the Docker host itself – i.e. there is limited container network isolation in the event of a container being compromised. Docker has a built in capability called ICC that can be disabled in order to provide Docker Container Network Isolation. The caveat being that to enable cross-container communication you have to use the “–link” argument when launching containers in order for the docker daemon to automatically create the required iptables rules to enable communication between containers on exposed ports.
When testing this on Ubuntu 16.04.3, unfortunately, despite documentation indicating that “–link” will automatically create iptables rules to enable cross docker network container access, this proved not to be the case when running Ubuntu 16.04.3. I also found that inter container communication still worked, despite ICC being disabled.
If the Docker daemon is running with both –icc=false and –iptables=true then, when it sees docker run invoked with the –link= option, the Docker server will insert a pair of iptables ACCEPT rules so that the new container can connect to the ports exposed by the other container – the ports that it mentioned in the EXPOSE lines of its Dockerfile
Source
ICC was disabled on the Docker host using a daemon.json file under /etc/docker – once set, ensure you restart the docker service:
{
icc: "false"
}
On further investigation as to why ICC was not behaving as expected I found that the bridge netfilter (br_netfilter) driver is not enabled by default on Ubuntu 16.4.3. To enable this you need to configure the docker host as below:
# Enable iptables filtering on bridge interfaces sudo vi /etc/modules-load.d/bridge.conf: br_netfilter sudo vi /etc/sysctl.d/bridge.conf: net.bridge.bridge-nf-call-ip6tables = 1 net.bridge.bridge-nf-call-iptables = 1 net.bridge.bridge-nf-call-arptables = 1
Now load the kernel module loaded and set the required kernel parameters:
sudo modprobe br_netfilter sudo sysctl net.bridge.bridge-nf-call-iptables=1 sudo sysctl net.bridge.bridge-nf-call-ip6tables=1 sudo sysctl net.bridge.bridge-nf-call-arptables=1
With the br_netfilter module loaded, and required parameters set, ICC behaved as expected, in that:
- Containers on the same network could not communicate with one-another
- Containers could not communicate with containers on other Docker networks
- Containers could not communicate with the Docker host itself
It is important to note that any user/ docker-compose defined networks will have ICC enabled, despite the daemon.json file configuration. You will need to configure/ modify any network in docker so that ICC is disabled – the Docker compose file below shows and example of this:
[...]
networks:
default:
driver_opts:
com.docker.network.bridge.enable_icc: "false"
This protected the host, and other containers on any Docker Compose-created network.
The IP Masquerade “Gotcha”
The final issue I ran into when testing ICC was that I wanted to protect the Docker host subnet: in the event of a container becoming compromised – I wanted to protect my management network. By default IP masquerade is enabled on docker0 and any newly created Docker compose/ user defined networks (unless specifically disabled at time of creation). This meant that whilst the Docker host / Docker networks were protected, my physical network was not.
It is possible to disable IP masquerade in a Docker Compose file using the below driver option in docker compose:
[...]
networks:
default:
driver_opts:
com.docker.network.bridge.enable_icc: "false"
com.docker.network.bridge.enable_ip_masquerade: "false"
However, at this point you have to manually define masquerade rules for any outbound traffic from the container (i.e. for WordPress, the plugin store). This becomes, in my opinion, as complex to manage as running Docker itself with the “–iptables=fase” configuration flag, where the administrator has to manually configure all iptables rules for the entire system. A key consideration here is the masquerade rules themselves, where the interface name has to be specified but could change in the event of using docker compose – example rule below.
sudo iptables -t nat -A POSTROUTING -s $DOCKER_NETWORK_SUBNET ! -d $PROTECTED_LAN ! -o $DOCKER_NETWORK_INTERFACE -j MASQUERADE
YMMV: it is worth testing as ICC being disabled certainly has its advantages, for direct exposure of services comprised of multiple containers via Docker Expose for example) with ICC set to false you can avoid IPAM/ IP address assignment in defining services as it will enable hostname/ connectivity on exposed ports automatically, irrespective of whether IP address changes etc.
To Hell with It… disable Docker’s iptables Integration
So… ICC didn’t quite fit the bill due to management network isolation/ protection. As a result I opted to go down the route of disabling Docker’s hook in to iptables, thus requiring manual iptables configuration. This solution provided full Docker Container Network Isolation.
Target network topology is as illustrated below (click image for full size):
To disable the Docker hook into iptables the following daemon.json file is needed under /etc/docker/daemon.json – once created, restart the Docker service:
{
"iptables":false
}
I created an iptables script in order to provide like-for-like DOCKER and DOCKER-ISOLATION chain capabilities, without Docker itself creating or modifying the Docker host firewall itself. An example script is included below.
This script, in summary meets all of the requirements set out at the beginning of this post:
- Docker Container access to Docker Containers within the same Docker network: whilst allowed in the script below, this is easy to achieve and highlighted in script
- Docker Container access to other Docker Containers in different Docker network: blocked
- Docker Container access to Docker host: blocked
- Docker Container access to Docker host physical network/ other hosts: blocked, unless by exception
I also opted to use IPAM for containers within this environment – this meant defining Docker networks with IP subnets/ containers with static IP addresses:
# Network for use by APP1 sudo docker network create \ --driver=bridge \ --subnet=10.10.101.0/24 \ --ip-range=10.10.101.0/24 \ --gateway=10.10.101.1 \ int-app1 # Network for use by APP2 sudo docker network create \ --driver=bridge \ --subnet=10.10.102.0/24 \ --ip-range=10.10.102.0/24 \ --gateway=10.10.102.1 \ ext-app2
When creating containers I specified the relevant Docker network and required IP address for each container, as below:
# APP1 MySQL and WordPress IP Addresses APP1_MYSQLIP="10.10.101.2" APP1_WORDPRESSIP="10.10.101.3" # Create APP1 Docker Volumes docker volume create app1-mysql docker volume create app1-data # Create APP1 MySQL Container sudo docker run --net int-app1 --ip=$APP1_MYSQLIP --name app1-mysql \ --mount source=app1-mysql,target=/var/lib/mysql -v /tmp/scripts:/tmp/scripts \ -e MYSQL_ROOT_PASSWORD='<passw0rd>' \ --restart=always -d mysql:latest # Create APP1 WordPress Container sudo docker run --net int-app1 --ip=$APP1_WORDPRESSIP --name app1-wordpress \ --mount source=app1-data,target=/var/www/html/wp-content \ -e WORDPRESS_DB_HOST="app1-mysql" \ -e WORDPRESS_DB_PASSWORD="<passw0rd>" \ --restart=always -d wordpress # APP2 NGINX Container IP Address APP2_IP="192.168.100.2" # Create APP2 Docker Volumes docker volume create app2-conf docker volume create app2-www # Create APP2 NGINX Container sudo docker run --net ext-app2 --ip=$APP2_IP --dns=8.8.8.8 --name app2-nginx -p 80:80 -p 443:443 \ --mount source=app2-conf,target=/etc/nginx/conf.d/ \ --mount source=app2-www,target=/var/www/ \ --restart always \ -d nginx
Firewall one-time configuration commands as below – **note requires replacing the docker network ID’s**:
# For rule persistence this requires "iptables-persistent netfilter-persistent" # sudo apt-get install iptables-persistent netfilter-persistent # Physical Networks PROTECTED_LAN="10.10.10.0/24" # Docker Host PROTECTED_HOST="10.10.10.100" # Docker Networks APP1_SUBNET="10.10.101.0/24" APP2_SUBNET="10.10.102.0/24" # Physical Interfaces LAN_INT="ens3" # Docker Interfaces - obtain these IDs from the command: sudo docker network ls APP1_INT="br-xxxxxxxx1" APP2_INT="br-xxxxxxxx2" # Container IPs APP1_MYSQLIP="10.10.101.2" APP1_IP="10.10.101.3" APP2_IP="10.10.102.2" # Physical Servers WEB_SERVER="10.10.10.55" ################## # Flush existing INPUT Rules ################## sudo iptables -P INPUT ACCEPT sudo iptables -F sudo iptables -t nat -F ################## # Enable SSH from protected LAN to avoid lockout! ################## sudo iptables -I INPUT -s $PROTECTED_LAN -d $PROTECTED_HOST -p tcp --dport 22 -j ACCEPT ################## # Enable ICMP echo request from $PROTECTED_LAN ################## sudo iptables -A INPUT -s $PROTECTED_LAN -p ICMP --icmp-type 8 -j ACCEPT ################## # Enable related traffic ################## sudo iptables -A INPUT -m state --state ESTABLISHED,RELATED -j ACCEPT ################## # Set default action INPUT chain to protect host/ host gateway addresses ################## sudo iptables -P INPUT DROP ################## # Set Docker Network FORWARD Rules ################## sudo iptables -N DOCKER-ISOLATION sudo iptables -N DOCKER sudo iptables -A FORWARD -j DOCKER-ISOLATION sudo iptables -A FORWARD -j DOCKER ################## # Docker0 Network sudo iptables -A FORWARD -o docker0 -m conntrack --ctstate RELATED,ESTABLISHED -j ACCEPT # Enable Docker network > outside of host access sudo iptables -A FORWARD -i docker0 -o $LAN_INT -j ACCEPT # Enable same-Docker network access (i.e. containers on same network, ***** disable to block container>container access on same network) sudo iptables -A FORWARD -i docker0 -o docker0 -j ACCEPT ## Enable this rule to facilitate Docker0 network access to **anywhere** # sudo iptables -A FORWARD -i docker0 ! -o docker0 -j ACCEPT # APP1 Network sudo iptables -A FORWARD -o $APP1_INT -m conntrack --ctstate RELATED,ESTABLISHED -j ACCEPT # Enable Docker network > outside of host access sudo iptables -A FORWARD -i $APP1_INT -o $LAN_INT -j ACCEPT # Enable same-Docker network access (i.e. containers on same network, ***** disable to block container>container access on same network) sudo iptables -A FORWARD -i $APP1_INT -o $APP1_INT -j ACCEPT ## Enable this rule to facilitate Docker APP1 network access to **anywhere** # sudo iptables -A FORWARD -i $APP1_INT ! -o $APP1_INT -j ACCEPT # APP2 Network sudo iptables -A FORWARD -o $APP2_INT -m conntrack --ctstate RELATED,ESTABLISHED -j ACCEPT # Enable Docker network > outside of host access sudo iptables -A FORWARD -i $APP2_INT -o $LAN_INT -j ACCEPT # Enable same-Docker network access (i.e. containers on same network, ***** disable to block container>container access on same network) sudo iptables -A FORWARD -i $APP2_INT -o $APP2_INT -j ACCEPT ## Enable this rule to facilitate Docker APP2 network access to **anywhere** # sudo iptables -A FORWARD -i $APP2_INT ! -o $APP2_INT -j ACCEPT ################## # Set default action FORWARD chain to DROP in order to isolate containers unless rules specified above to facilitate access ################## sudo iptables -P FORWARD DROP ################## # DOCKER and DOCKER-ISOLATION rules ################## # APP2 NGINX reverse access on port 80/ 443 from LAN/ elsewhere sudo iptables -A DOCKER-ISOLATION -d $APP2_IP/32 ! -i $APP2_INT -o $APP2_INT -p tcp -m tcp --dport 443 -j ACCEPT sudo iptables -A DOCKER-ISOLATION -d $APP2_IP/32 ! -i $APP2_INT -o $APP2_INT -p tcp -m tcp --dport 80 -j ACCEPT # APP2 NGINX reverse proxy access to APP1 WordPress Instance on port 80 - note rules are two way when both containers on networks on same host sudo iptables -A DOCKER-ISOLATION -s $APP2_IP/32 -d $APP1_IP/32 -p tcp -m tcp --dport 80 -j ACCEPT sudo iptables -A DOCKER-ISOLATION -s $APP1_IP/32 -d $APP2_IP/32 -p tcp -m tcp --sport 80 -j ACCEPT # Same-host, same-Docker network access should not be an issue for containers ################## # All other traffic disabled/ dropped by default action DROP ################## # Chain Return Rules sudo iptables -A DOCKER-ISOLATION -j RETURN sudo iptables -A DOCKER -j RETURN ################## # DNAT Rules for APP2 NGINX Container ################## sudo iptables -t nat -A PREROUTING -i ens3 -p tcp -m tcp --dport 80 -j DNAT --to-destination $APP2_IP:80 sudo iptables -t nat -A PREROUTING -i ens3 -p tcp -m tcp --dport 443 -j DNAT --to-destination $APP2_IP:443 ################## # APP1 Access to Web Server on Physical / Host Network ################## sudo iptables -t nat -A POSTROUTING -s $APP1_IP -d $WEB_SERVER ! -o $APP1_INT -p tcp -m tcp --dport 80 -j MASQUERADE ################## # MASQUERADE/ NAT rules to enable external access etc. ################## sudo iptables -t nat -N DOCKER sudo iptables -t nat -A PREROUTING -m addrtype --dst-type LOCAL -j DOCKER sudo iptables -t nat -A OUTPUT ! -d 127.0.0.0/8 -m addrtype --dst-type LOCAL -j DOCKER sudo iptables -t nat -A POSTROUTING -s 172.17.0.0/16 ! -o docker0 -j MASQUERADE ################## # Docker Network External Access, excluding host LAN, unless specific rules above, such as APP1>Web Server ################## sudo iptables -t nat -A POSTROUTING -s $APP1_SUBNET ! -d $PROTECTED_LAN ! -o $APP1_INT -j MASQUERADE sudo iptables -t nat -A POSTROUTING -s $APP2_SUBNET ! -d $PROTECTED_LAN ! -o $APP2_INT -j MASQUERADE ################## # Chain return rules ################## sudo iptables -t nat -A DOCKER -i docker0 -j RETURN sudo iptables -t nat -A DOCKER -i $GITLAB_INT -j RETURN sudo iptables -t nat -A DOCKER -i $GUAC_INT -j RETURN sudo iptables -t nat -A DOCKER -i $WP_INT -j RETURN sudo iptables -t nat -A DOCKER -i $WP_TBA_INT -j RETURN sudo iptables -t nat -A DOCKER -i $EXT_INT -j RETURN ################## # Save rules, making them persistent (disabled for testing!) ################## # sudo sh -c "iptables-save > /etc/iptables/rules.v4"
I’d be interested to see how other have catered for Docker Container Network Isolation in their environments – feel free to leave comments/ examples below.