iptables_docker
Ansible Role: iptables for Docker (and Docker Swarm)
Add firewall rules to server via iptables, for Docker, and Docker Swarm. This will actually protect your Docker containers!
This Ansible Role exists because firewalld and Docker (and Docker Swarm) do not get along.
Problem being solved: When starting a container in Docker with a "published" port, you have no control and the port is exposed through your server's firewall. Even if you were using iptables, or another firewall on your server. Docker opens that "published" port to everyone, and bypasses your firewall.
Use case for this solution: Allow trusted IPs to connect to Docker containers (and Docker Swarm containers), along with other open OS ports. With an option to expose specified ports publicly (Docker/Docker Swarm and OS). The trusted IPs might not be in the same network IP range, or even the same network subnet.
This was suppose to be simple. Secure Docker with a firewall. But unfortuanately it is not. I've tried to keep this as simple as possible.
There could be unknown problems with this.. use at your own risk!
See also: https://ryandaniels.ca/blog/secure-docker-with-iptables-firewall-and-ansible/
And about Docker's use of the INPUT chain: https://ryandaniels.ca/blog/docker-iptables-input-chain/
Currently tested and working on:
- CentOS/RHEL 7
- Ubuntu 18.04
- Ubuntu 20.04
Features
- Works with Docker, and Docker Swarm (aka Docker SwarmKit).
- Secure by default. Once configured, only Docker IPs can access all containers, and all other OS processes that have open ports on the server.
- Simple as possible. The less iptables rules, the faster performance will be (in theory at least).
- Automatic. No manually adding ports to the firewall config (if you use a trusted set of IPs)
- Add "trusted" IPs that are allowed to communicate with all Docker containers, and all other OS processes that have open ports on the server.
- Open specified Docker container ports, or the server's OS ports to the public (everyone) through the firewall, like SSH.
- Interfaces can also be specified. By default all interfaces are filtered. You could filter specific network interface(s) and allow all other interfaces (only specify an untrusted interface).
- Everything done in "offline" mode. So there should be no issues with Docker when iptables rules are activated.
- You don't need to be an expert with iptables to use this.
- Works with Docker Swarm's undocumented use of iptables and encrypted overlay networks. (iptables rules are appeneded to the INPUT chain).
This solution is using iptables as the firewall, and ipset to allow iptables to have a list of IPs that are allowed. ipset also allows you to use a non-continueous range of IPs.
iptables chains used, and how:
INPUT, not flushed. Rule inserted at top to jump to custom chain for OS related rules.
DOCKER-USER, flushed. All Docker (and Docker Swarm) related rules are here to block containers from being exposed to everyone by default. By default only the Docker server IPs are allowed. Other IPs and container ports can be added by user.
FILTERS, flushed. Custom chain for server's processes (that aren't Docker). By default only the Docker server IPs are allowed. Other IPs and container ports can be added by user.
iptables manual: http://ipset.netfilter.org/iptables.man.html
Warnings
Don't lock yourself out of your server. This is modifying your firewall. Always have another way to get in, like a "console".
Note about IPs: This is for IPv4 only. IPv6 has not been tested. It is safer if you disable IPv6 on your servers.
Other security consideration:
If using non-Swarm (normal Docker), consider also binding a port to an internal IP for better security.
If using Swarm, consider using specific IPs for Docker Swarm communication.
Eg. docker swarm init --advertise-addr 192.168.100.100 --listen-addr=192.168.100.100 --data-path-addr=192.168.100.100
Important Note: Docker and firewalld do not get along. This Ansible Role has a check enabled to fail this role if the firewalld service is running or enabled.
For more information about firewalld and Docker:
https://success.docker.com/article/why-am-i-having-network-problems-after-firewalld-is-restarted
https://www.tripwire.com/state-of-security/devops/psa-beware-exposing-ports-docker/
https://docs.docker.com/network/iptables/
SELinux Bug:
Currently there's a bug with SELinux that prevents saving the iptables rules to the iptables.save file.
Impact: Saving the iptables rules a 2nd time will silently fail.
Workaround has been added so SELinux allows chmod to interact with the iptables.save file.
Alternatively you could disable SELinux, but that's not recommended.
Bug report: https://bugs.centos.org/view.php?id=12648
See below for more details about manually performing the workaround.
WARNING:
Make sure you test in non-production first, I cannot make any guarantees or held responsible.
Be careful, this will remove and add iptables rules on the OS. Use with caution.
Existing iptables rules could be removed! Confirm what you have setup before running this.
There could be unknown problems with this.. use at your own risk!
Docker versions tested
Docker Engine - Community Edition version:
- 19.03.8
- 19.03.9
- 19.03.12
Tested in normal Docker mode, and with a 3 node Docker Swarm cluster.
Distros tested
- CentOS: 7.7, 7.8
- Ubuntu 18.04
- Ubuntu 20.04
Dependencies
- iptables & iptables-services
Tested with v1.4.21 (Latest available in CentOS 7)
- ipset & ipset-service
Tested with v7.1 (Latest available in CentOS 7)
Default Settings
- Enable debug
debug_enabled_default: false
- Proxy (Needed when installing required packages if behind a proxy)
proxy_env: []
- Role disabled by default. Change to true in group_vars or playbook etc
iptables_docker_managed: false
- Check if (Docker) service is running or enabled, and fail the role
iptables_docker_check_problem_service_managed: true
- Services to check, and fail the role
iptables_docker_check_problem_service:
- docker.service
- Show configuration from variables
iptables_docker_show_config: true
- Start iptables service
iptables_docker_start: true
- Install iptables package
iptables_docker_managed_pkg: true
iptables_docker_packages:
- iptables
- iptables-services
- ipset
- ipset-service
- policycoreutils-python #required for semodule
- Force copy of ipset file to trigger ipset reload
iptables_docker_copy_ipset_force: false
- Force copy of iptables file to trigger iptables reload
iptables_docker_copy_iptables_force: false
- iptables saved configuration location
iptables_docker_iptables_config_save: /etc/sysconfig/iptables
- ipset saved configuration location
iptables_docker_ipset_config_dir: /etc/sysconfig/ipset.d
ipset maximum elements (IPs in the allow list)
If changed after first creation, must be deleted and re-created manually. 64k IPs should be enough.
iptables_docker_ipset_maxelem: 65536
User Settings
Override Docker server IPs (Optional)
Optionally specify the Docker server IPs. If not set, IPs will be determined from docker_hosts group in Ansible inventory.
# iptables_docker_server_ip_allow_set:
# - 192.168.100.100
# - 192.168.100.101
# - 192.168.100.102
- IPs allowed to use all Docker container's exposed ports and all server's processes' exposed ports.
# iptables_docker_ip_allow_set: []
iptables_docker_ip_allow_set:
- 192.168.100.1
- 192.168.101.0/24
- 192.168.102.0/24
Network adapter to restrict for OS rules
Only listed adapters will be blocked. Others will be allowed through. Defaults to block all (with '+').
If you want to restrict only specific network interface use exact name.
If you want to restrict all interfaces of the same type, use "interface+" to match every interface, since + is the wildcard for iptables.
Eg. To restrict the ethX interfaces, use "eth+". "eth+" is a wildcard for anything starting with eth.
DO NOT use "*". This is not a wildcard and matches nothing!
The less here the better. Safer to block all ('+') but if cannot, add network adapters with high traffic first.
local (lo) is not needed here.
iptables_docker_external_network_adapter:
- "+" #Wildcard for everything
# - "eth+"
# - "enp0s+"
# - "wlp1s+"
OS tcp ports open to public
Ports to allow everyone to connect (will be publicly accessible). Ports here will allow all tcp traffic to these ports from iptables level.
Only for ports on OS, not for Docker containers.
iptables_docker_global_ports_allow_tcp:
- 22 # SSH
OS udp ports open to public
Ports to allow everyone to connect (will be publicly accessible). Ports here will allow all udp traffic to these ports from iptables level.
Only for ports on OS, not for Docker containers.
iptables_docker_global_ports_allow_udp: []
Network adapter to restrict for Docker rules
Defaults to use the same setup as the network adapter for the OS.
iptables_docker_swarm_network_adapter: "{{ iptables_docker_external_network_adapter }}"
# iptables_docker_swarm_network_adapter:
# - "+" #Wildcard for everything
# # - "eth+"
Docker tcp ports open to public
Add Docker container tcp ports you want open to everyone. For Docker and Docker Swarm. Docker Swarm ports aren't needed here.
iptables_docker_swarm_ports_allow_tcp: []
# iptables_docker_swarm_ports_allow_tcp:
# - 9000
Docker udp ports open to public
Add Docker container udp ports you want open to everyone. For Docker and Docker Swarm. Docker Swarm ports aren't needed here.
iptables_docker_swarm_ports_allow_udp: []
- Docker bridge network name (docker0), and IP range (for DOCKER-USER iptables source allow)
iptables_docker_bridge_name: docker0
iptables_docker_bridge_ips: 172.17.0.0/16
- Docker Swarm bridge network IP range (docker_gwbridge) (for DOCKER-USER iptables source allow)
iptables_docker_swarm_bridge_name: docker_gwbridge
iptables_docker_swarm_bridge_ips: 172.18.0.0/16
Example config file (inventories/dev-env/group_vars/all.yml)
From the example below:
IPs will be added to the trusted list:
192.168.100.1192.168.101.0/24
All network interfaces will be restricted since using wildcard '+' for iptables_docker_external_network_adapter.
Port 22 will be open publicly.
---
iptables_docker_ip_allow_set:
- 192.168.100.1
- 192.168.101.0/24
iptables_docker_external_network_adapter:
- "+" #Wildcard for everything
iptables_docker_global_ports_allow_tcp:
- 22 # SSH
Example inventory file
[docker_hosts]
centoslead1 ansible_host=192.168.100.100
centoswork1 ansible_host=192.168.100.101
centoswork2 ansible_host=192.168.100.102
Example Playbook iptables_docker.yml
---
- hosts: '{{ inventory }}'
become: yes
vars:
# Use this role
iptables_docker_managed: true
roles:
- ryandaniels.iptables_docker
Usage
Before running make sure you check if you are already using iptables! Nothing should be overwritten/removed, unless you are using the same iptables chains as this.
By default no tasks will run unless you set iptables_docker_managed=true. This is by design to prevent accidents by people who don't RTFM.
ansible-playbook iptables_docker.yml --extra-vars "inventory=centos7 iptables_docker_managed=true" -i hosts-dev
Skip installing packages (if known already there - speeds up task)
ansible-playbook iptables_docker.yml --extra-vars "inventory=centos7 iptables_docker_managed=true" -i hosts --skip-tags=iptables_docker_pkg_install
Show more verbose output (debug info)
ansible-playbook iptables_docker.yml --extra-vars "inventory=centos7 iptables_docker_managed=true debug_enabled_default=true" -i hosts-dev
Do not start iptables service or add rules for iptables
ansible-playbook iptables_docker.yml --extra-vars "inventory=centos7 iptables_docker_managed=true iptables_docker_start=false" -i hosts-dev
Force ipset and iptables to update
ansible-playbook iptables_docker.yml --extra-vars "inventory=centos7 iptables_docker_managed=true iptables_docker_copy_ipset_force=true iptables_docker_copy_iptables_force=true" -i hosts-dev
Only show configuration (from variables)
ansible-playbook iptables_docker.yml --extra-vars "inventory=centos7 iptables_docker_managed=true iptables_docker_show_config=true" -i hosts --tags "iptables_docker_show_config"
Note about ipset size limit
Important: Make note of the size of "Number of entries". If that number is close to the maxelem size (65536), then you need to delete the ipset "ip_allow" and re-create it with a larger max size.
64K ought to be enough for anyone.
File is in: templates/ip_allow.set.j2
create -exist ip_allow hash:ip family inet hashsize 1024 maxelem 65536
Check size of ipset list:
ipset list |grep "Number of entries"
Important output:
Number of entries: 3
SELinux manual workaround for iptables and chmod
Bug details: https://bugs.centos.org/view.php?id=12648
The problem is when saving iptables a 2nd time, SELinux blocks it since chmod has a problem with the iptables.save file.
Use below as workaround to allow chmod to modify iptables.save file if not using the Ansible role.
To reproduce, restart iptables service after setting iptables config to save after restart/stop,
yum install audit policycoreutils policycoreutils-python
ausearch -m AVC,USER_AVC,SELINUX_ERR,USER_SELINUX_ERR -i|tail -55
ausearch -c 'chmod' --raw | audit2allow -M iptables_save_chmod
#or grep "iptables.save" /var/log/audit/audit.log|tail | audit2allow -M iptables_save_chmod
semodule -i iptables_save_chmod.pp
iptables Command Reference
More commands can be found in iptables documentation: http://ipset.netfilter.org/iptables.man.html
List iptables that are active:
iptables -nvL --line-numbers
Misc CentOS/RHEL useful commands:
cat /etc/sysconfig/ipset.d/ip_allow.set
systemctl restart ipset
ipset list | head
iptables -F DOCKER-USER
iptables -F FILTERS
iptables-restore -n < ansible_iptables_docker-iptables
grep -v "^#" ansible_iptables_docker-iptables
iptables -S INPUT
iptables -S DOCKER-USER
iptables -S FILTERS
Misc Ubuntu useful commands:
vi /etc/iptables/ipsets
#Manually add 'flush' before add, if removing IPs manually.
/usr/sbin/netfilter-persistent reload
cat /etc/iptables/ipsets
cat /etc/iptables/rules.v4
Manual Commands (CentOS/RHEL)
Check what iptables rules you already have. Make note in case they are lost!
iptables -nvL --line-numbers
Install required packages:
yum install iptables iptables-services ipset ipset-service
If using SELinux, also install:
yum install policycoreutils-python
Configure ipset with your server IPs and other trusted IPs:
mkdir -p /etc/sysconfig/ipset.d
cat > /etc/sysconfig/ipset.d/ip_allow.set << 'EOF'
create -exist ip_allow hash:ip family inet hashsize 1024 maxelem 65536
add ip_allow 192.168.1.123
add ip_allow 192.168.101.0/24
add ip_allow 192.168.102.0/24
EOF
Start, and Enable the ipset service:
systemctl status ipset
systemctl start ipset
systemctl enable ipset
See what ipset has in it's loaded configuration:
ipset list | head
iptables rules being added (by default port 22 is open to everyone):
cat > ansible_iptables_docker-iptables << 'EOF'
*filter
:DOCKER-USER - [0:0]
:FILTERS - [0:0]
#Can't flush INPUT. wipes out docker swarm encrypted overlay rules
#-F INPUT
#Use ansible or run manually once instead to add -I INPUT -j FILTERS
#-I INPUT -j FILTERS
-A DOCKER-USER -m state --state RELATED,ESTABLISHED -j RETURN
-A DOCKER-USER -i docker_gwbridge -j RETURN
-A DOCKER-USER -s 172.18.0.0/16 -j RETURN
-A DOCKER-USER -i docker0 -j RETURN
-A DOCKER-USER -s 172.17.0.0/16 -j RETURN
#Below Docker ports open to everyone if uncommented
#-A DOCKER-USER -p tcp -m tcp -m multiport --dports 8000,8001 -j RETURN
#-A DOCKER-USER -p udp -m udp -m multiport --dports 9000,9001 -j RETURN
-A DOCKER-USER -m set ! --match-set ip_allow src -j DROP
-A DOCKER-USER -j RETURN
-F FILTERS
#Because Docker Swarm encrypted overlay network just appends rules to INPUT. Has to be at top unfortunately
-A FILTERS -p udp -m policy --dir in --pol ipsec -m udp --dport 4789 -m set --match-set ip_allow src -j RETURN
-A FILTERS -m state --state RELATED,ESTABLISHED -j ACCEPT
-A FILTERS -p icmp -j ACCEPT
-A FILTERS -i lo -j ACCEPT
#Below OS ports open to everyone if uncommented
-A FILTERS -p tcp -m state --state NEW -m tcp -m multiport --dports 22 -j ACCEPT
#-A FILTERS -p udp -m udp -m multiport --dports 53,123 -j ACCEPT
-A FILTERS -m set ! --match-set ip_allow src -j DROP
-A FILTERS -j RETURN
COMMIT
EOF
Use iptables-restore to add the above rules into iptables. The very important flag is -n. That makes sure we don't flush the iptables rules if we have rules already in Docker (or Docker Swarm).
iptables-restore -n < ansible_iptables_docker-iptables
Next, add a rule to the INPUT chain, so we start using the new rules in FILTERS. It has to be at the top, and only needs to be added once:
iptables -I INPUT 1 -j FILTERS
Save the iptables rules:
/usr/libexec/iptables/iptables.init save
Start and Enable the iptables service:
systemctl status iptables
systemctl start iptables
systemctl enable iptables
If you want to customize the iptables rules to allow more ports to be open to everyone, just add the port to the appropriate rule in the iptables file (tcp or udp), then re-run the same commands from above:
iptables-restore -n < ansible_iptables_docker-iptables
/usr/libexec/iptables/iptables.init save
Don't miss the Warnings from above! Especially about SELinux.
Manual Commands (Ubuntu 20.04)
Check what iptables rules you already have. Make note in case they are lost!
Ubuntu 18.04 is almost the same. Except the ipset-persistent package doesn't exist in Ubuntu 18.04, so omit that package and copy the files from files/ubuntu/iptables-persistent*/plugins/*-ipset to /usr/share/netfilter-persistent/plugins.d/.
iptables -nvL --line-numbers
Install required packages:
apt install iptables iptables-persistent netfilter-persistent ipset ipset-persistent
Configure ipset with your server IPs and other trusted IPs:
mkdir -p /etc/iptables
cat > /etc/iptables/ipsets << 'EOF'
create -exist ip_allow hash:ip family inet hashsize 1024 maxelem 65536
flush
add ip_allow 192.168.1.123
add ip_allow 192.168.101.0/24
add ip_allow 192.168.102.0/24
EOF
Reload ipset:
/usr/sbin/netfilter-persistent reload
See what ipset has in it's loaded configuration:
ipset list | head
iptables rules being added (by default port 22 is open to everyone):
Use same command as above for CentOS/RHEL.
Use iptables-restore to add the above rules into iptables. The very important flag is -n. That makes sure we don't flush the iptables rules if we have rules already in Docker (or Docker Swarm).
iptables-restore -n < ansible_iptables_docker-iptables
Next, add a rule to the INPUT chain, so we start using the new rules in FILTERS. It has to be at the top, and only needs to be added once:
iptables -I INPUT 1 -j FILTERS
Save the iptables rules:
/usr/sbin/netfilter-persistent save
Start and Enable the iptables service:
systemctl status netfilter-persistent
systemctl start netfilter-persistent
systemctl enable netfilter-persistent
If you want to customize the iptables rules to allow more ports to be open to everyone, just add the port to the appropriate rule in the iptables file (tcp or udp), then re-run the same commands from above:
iptables-restore -n < ansible_iptables_docker-iptables
/usr/sbin/netfilter-persistent save
Don't miss the Warnings from above!
TODO
- Check for firewalld and fail if running or enabled
- Problem with iptables saving Docker rules in iptables rules? Should be fine.
- iptables_docker_ip_allow_set can't be empty. If it is, there's no point to this since nothing is blocked!
- add check in network adapters for * and error
- add automatic list of docker IPs in allowed list (uses IPs from inventory group docker_hosts)
- Change auto Docker server trusted IPs so can override
- confirm "when" and "tags" are ok
- Ubuntu? Ubuntu doesn't have iptables-services or ipset-service. has iptables-persistent and ipset-? No ufw support
- ipv6?? This is for ipv4 only
- test TCP, UDP Docker container and OS port work
- test outound traffic from Docker containers work
- add test? Molecule? Single node swarm mode only? how to test connection doesn't work from "untrusted" ip?
Author
Ryan Daniels
Manage iptables configuration to secure Docker (including Docker Swarm).
ansible-galaxy install ryandaniels/ansible-role-iptables-docker