This blog is part of my ongoing series on Docker containers. In this blog, I will cover Flannel which is a CoreOS networking solution to connect containers across multiple hosts. Please refer to my first blog on CoreOS to understand CoreOS basics as well as how to setup CoreOS cluster.
Flannel Overview:
Flannel creates an Overlay network using either udp or vxlan encapsulation. Flannel links itself to the Docker bridge to which the containers are attached and creates the overlay. Following picture illustrates this.
Demo:
Lets create 3 containers in 3 different CoreOS hosts that are part of same cluster in 11.0.0.0/16 subnet, connect them using Flannel and try to ping them. Create the 3 node cluster using the procedure in my previous CoreOS blog.
First step is to install flannel using the procedure described here. Flannel needs to be installed in all hosts that are part of the cluster. Following commands builds flannel binary downloading the source.
git clone https://github.com/coreos/flannel.git cd flannel sudo docker run -v `pwd`:/opt/flannel -i -t google/golang /bin/bash -c "cd /opt/flannel && ./build"
Next, lets create etcd key, value pair that contains the IP subnet that the cluster needs to use.
etcdctl rm /coreos.com/network/ --recursive
etcdctl mk /coreos.com/network/config '{"Network":"11.0.0.0/16"}'
We can go to any host and make sure that the key, value pair is updated.
core@core-02 ~ $ etcdctl get /coreos.com/network/config
{"Network":"11.0.0.0/16"}
Now, lets start the flannel service. Make sure that “-iface” argument is supplied and the interface chosen has a unique address across hosts. Flannel needs to have a unique host identifier for the overlay and by default, it uses the eth0 ip address. eth0 is a nat interface and all hosts of the cluster have the same ip address. This causes all hosts to get same ip address. Thanks to Flannel team, they helped me in identifying this issue, the issue and the solution are described here.
sudo /home/core/flannel/bin/flanneld -iface=eth1 &
Following command shows status of flannel service in host 1. It shows 11.0.13.0/24 subnet is allocated to this host.
core@core-01 ~ $ systemctl status flannel
● flannel.service
Loaded: loaded (/run/fleet/units/flannel.service; linked-runtime; vendor preset: disabled)
Active: active (running) since Sun 2015-01-18 13:23:52 UTC; 15min ago
Process: 677 ExecStartPre=/usr/bin/etcdctl mk /coreos.com/network/config {"Network":"10.0.0.0/16"} (code=exited, statu
s=4)
Main PID: 682 (flanneld)
CGroup: /system.slice/flannel.service
└─682 /home/core/flannel/bin/flanneld
Jan 18 13:23:52 core-01 etcdctl[677]: Error: 105: Key already exists (/coreos.com/network/config) [172807]
Jan 18 13:23:52 core-01 systemd[1]: Started flannel.service.
Jan 18 13:23:52 core-01 flanneld[682]: I0118 13:23:52.209542 00682 main.go:250] Installing signal handlers
Jan 18 13:23:52 core-01 flanneld[682]: I0118 13:23:52.211359 00682 main.go:119] Determining IP address of defau...erface
Jan 18 13:23:52 core-01 flanneld[682]: I0118 13:23:52.212317 00682 main.go:208] Using 10.0.2.15 as external interface
Jan 18 13:23:52 core-01 flanneld[682]: I0118 13:23:52.264848 00682 subnet.go:83] Subnet lease acquired: 10.11.32.0/20
Jan 18 13:23:52 core-01 flanneld[682]: I0118 13:23:52.279451 00682 main.go:218] UDP mode initialized
Jan 18 13:23:52 core-01 flanneld[682]: I0118 13:23:52.280220 00682 udp.go:239] Watching for new subnet leases
Jan 18 13:32:28 core-01 flanneld[682]: E0118 13:32:28.900556 00682 subnet.go:372] Error parsing subnet IP: /cor...etwork
Jan 18 13:36:30 core-01 flanneld[682]: I0118 13:36:30.878485 00682 udp.go:264] Subnet added: 11.0.13.0/24
Following is my ifconfig output in the host 1. Flannel1 is the interface created now, Flannel0 was created earlier.
core@core-01 ~ $ ifconfig
eth0: flags=4163<UP,BROADCAST,RUNNING,MULTICAST> mtu 1500
inet 10.0.2.15 netmask 255.255.255.0 broadcast 10.0.2.255
inet6 fe80::a00:27ff:fe8b:9c8b prefixlen 64 scopeid 0x20
ether 08:00:27:8b:9c:8b txqueuelen 1000 (Ethernet)
RX packets 1385 bytes 117063 (114.3 KiB)
RX errors 0 dropped 0 overruns 0 frame 0
TX packets 930 bytes 96237 (93.9 KiB)
TX errors 0 dropped 0 overruns 0 carrier 0 collisions 0
eth1: flags=4163<UP,BROADCAST,RUNNING,MULTICAST> mtu 1500
inet 172.17.8.101 netmask 255.255.255.0 broadcast 172.17.8.255
inet6 fe80::a00:27ff:fe3f:1cea prefixlen 64 scopeid 0x20
ether 08:00:27:3f:1c:ea txqueuelen 1000 (Ethernet)
RX packets 96931 bytes 21400016 (20.4 MiB)
RX errors 0 dropped 2 overruns 0 frame 0
TX packets 96037 bytes 13489117 (12.8 MiB)
TX errors 0 dropped 0 overruns 0 carrier 0 collisions 0
flannel0: flags=81<UP,POINTOPOINT,RUNNING> mtu 1472
inet 10.11.32.0 netmask 255.0.0.0 destination 10.11.32.0
unspec 00-00-00-00-00-00-00-00-00-00-00-00-00-00-00-00 txqueuelen 500 (UNSPEC)
RX packets 34 bytes 1904 (1.8 KiB)
RX errors 0 dropped 0 overruns 0 frame 0
TX packets 34 bytes 10370 (10.1 KiB)
TX errors 0 dropped 0 overruns 0 carrier 0 collisions 0
flannel1: flags=81<UP,POINTOPOINT,RUNNING> mtu 1472
inet 11.0.13.0 netmask 255.255.0.0 destination 11.0.13.0
unspec 00-00-00-00-00-00-00-00-00-00-00-00-00-00-00-00 txqueuelen 500 (UNSPEC)
RX packets 22 bytes 1232 (1.2 KiB)
RX errors 0 dropped 0 overruns 0 frame 0
TX packets 22 bytes 6710 (6.5 KiB)
TX errors 0 dropped 0 overruns 0 carrier 0 collisions 0
lo: flags=73<UP,LOOPBACK,RUNNING> mtu 65536
inet 127.0.0.1 netmask 255.0.0.0
inet6 ::1 prefixlen 128 scopeid 0x10
loop txqueuelen 0 (Local Loopback)
RX packets 39281 bytes 7234072 (6.8 MiB)
RX errors 0 dropped 0 overruns 0 frame 0
TX packets 39281 bytes 7234072 (6.8 MiB)
TX errors 0 dropped 0 overruns 0 carrier 0 collisions 0
Lets start the flannel service in host 2 and host 3. 11.0.45.0/24 gets allocated to host 2 and 11.0.48/24 gets allocated to host 3.
Lets look at Flannel environment variables in host 1.
core@core-01 ~ $ cat /run/flannel/subnet.env FLANNEL_SUBNET=11.0.13.1/24 FLANNEL_MTU=1472
Next step is to restart the docker service with the new subnet that flannel has allocated. Following set of commands will do that.
source /run/flannel/subnet.env
sudo rm /var/run/docker.pid
sudo ifconfig docker0 ${FLANNEL_SUBNET}
sudo docker -d --bip=${FLANNEL_SUBNET} --mtu=${FLANNEL_MTU} &
Lets create 1 container in all hosts.
sudo docker run -ti ubuntu /bin/bash
Following is the ifconfig output of the 3 containers created. The containers on the respective hosts uses the FLANNEL_SUBNET that was created by Flannel when the service was started.
host1:
root@90f252b18220:/# ifconfig
eth0 Link encap:Ethernet HWaddr 02:42:0b:00:0d:02
inet addr:11.0.13.2 Bcast:0.0.0.0 Mask:255.255.255.0
inet6 addr: fe80::42:bff:fe00:d02/64 Scope:Link
UP BROADCAST RUNNING MTU:1472 Metric:1
RX packets:20 errors:0 dropped:0 overruns:0 frame:0
TX packets:6 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:0
RX bytes:1609 (1.6 KB) TX bytes:508 (508.0 B)
lo Link encap:Local Loopback
inet addr:127.0.0.1 Mask:255.0.0.0
inet6 addr: ::1/128 Scope:Host
UP LOOPBACK RUNNING MTU:65536 Metric:1
RX packets:0 errors:0 dropped:0 overruns:0 frame:0
TX packets:0 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:0
RX bytes:0 (0.0 B) TX bytes:0 (0.0 B)
host2:
root@4120b2ff44c8:/# ifconfig
eth0 Link encap:Ethernet HWaddr 02:42:0b:00:30:02
inet addr:11.0.48.2 Bcast:0.0.0.0 Mask:255.255.255.0
inet6 addr: fe80::42:bff:fe00:3002/64 Scope:Link
UP BROADCAST RUNNING MTU:1472 Metric:1
RX packets:20 errors:0 dropped:0 overruns:0 frame:0
TX packets:6 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:0
RX bytes:1642 (1.6 KB) TX bytes:508 (508.0 B)
lo Link encap:Local Loopback
inet addr:127.0.0.1 Mask:255.0.0.0
inet6 addr: ::1/128 Scope:Host
UP LOOPBACK RUNNING MTU:65536 Metric:1
RX packets:0 errors:0 dropped:0 overruns:0 frame:0
TX packets:0 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:0
RX bytes:0 (0.0 B) TX bytes:0 (0.0 B)
host3:
root@ecf8d38dc93d:/# ifconfig
eth0 Link encap:Ethernet HWaddr 02:42:0b:00:2d:02
inet addr:11.0.45.2 Bcast:0.0.0.0 Mask:255.255.255.0
inet6 addr: fe80::42:bff:fe00:2d02/64 Scope:Link
UP BROADCAST RUNNING MTU:1472 Metric:1
RX packets:22 errors:0 dropped:0 overruns:0 frame:0
TX packets:7 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:0
RX bytes:1782 (1.7 KB) TX bytes:578 (578.0 B)
lo Link encap:Local Loopback
inet addr:127.0.0.1 Mask:255.0.0.0
inet6 addr: ::1/128 Scope:Host
UP LOOPBACK RUNNING MTU:65536 Metric:1
RX packets:0 errors:0 dropped:0 overruns:0 frame:0
TX packets:0 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:0
RX bytes:0 (0.0 B) TX bytes:0 (0.0 B)
Lets make sure that container 1 is able to reach container 2 and container 3:
root@90f252b18220:/# ping -c1 11.0.48.2 PING 11.0.48.2 (11.0.48.2) 56(84) bytes of data. 64 bytes from 11.0.48.2: icmp_seq=1 ttl=60 time=1.43 ms --- 11.0.48.2 ping statistics --- 1 packets transmitted, 1 received, 0% packet loss, time 0ms rtt min/avg/max/mdev = 1.435/1.435/1.435/0.000 ms root@90f252b18220:/# ping -c1 11.0.45.2 PING 11.0.45.2 (11.0.45.2) 56(84) bytes of data. 64 bytes from 11.0.45.2: icmp_seq=1 ttl=60 time=1.77 ms --- 11.0.45.2 ping statistics --- 1 packets transmitted, 1 received, 0% packet loss, time 0ms rtt min/avg/max/mdev = 1.774/1.774/1.774/0.000 ms
Following is an example of network config where we have specified a range of subnets and also we have changed the encapsulation from udp to vxlan.
curl -L http://127.0.0.1:4001/v2/keys/coreos.com/network/config -XPUT -d value='{
"Network": "11.0.0.0/8",
"SubnetLen": 20,
"SubnetMin": "11.10.0.0",
"SubnetMax": "11.99.0.0",
"Backend": {
"Type": "vxlan",
"Port": 7890}}'
Thanks to the Flannel team for the nice solution. I am not sure about the following points:
- As of now, Flannel seems to be specific to CoreOS cluster. I think at some point, the solution will extend beyond that.
- I did not find a way to do multi-tenant container networking.
Sreenivas Makam's Blog 