Docker Networking – CoreOS Flannel

This blog is part of my ongoing series on Docker containers. In this blog, I will cover Flannel which is a CoreOS networking solution to connect containers across multiple hosts. Please refer to my first blog on CoreOS to understand CoreOS basics as well as how to setup CoreOS cluster.

Flannel Overview:

Flannel creates an Overlay network using either udp or vxlan encapsulation. Flannel links itself to the Docker bridge to which the containers are attached and creates the overlay. Following picture illustrates this.

coreos3

Demo:

Lets create 3 containers in 3 different CoreOS hosts that are part of same cluster in 11.0.0.0/16 subnet, connect them using Flannel and try to ping them. Create the 3 node cluster using the procedure in my previous CoreOS blog.

First step is to install flannel using the procedure described here. Flannel needs to be installed in all hosts that are part of the cluster. Following commands builds flannel binary downloading the source.

git clone https://github.com/coreos/flannel.git
cd flannel
sudo docker run -v `pwd`:/opt/flannel -i -t google/golang /bin/bash -c "cd /opt/flannel && ./build"

Next, lets create etcd key, value pair that contains the IP subnet that the cluster needs to use.

etcdctl rm /coreos.com/network/ --recursive
etcdctl mk /coreos.com/network/config '{"Network":"11.0.0.0/16"}'

We can go to any host and make sure that the key, value pair is updated.

core@core-02 ~ $ etcdctl get /coreos.com/network/config
{"Network":"11.0.0.0/16"}

Now, lets start the flannel service. Make sure that “-iface” argument is supplied and the interface chosen has a unique address across hosts. Flannel needs to have a unique host identifier for the overlay and by default, it uses the eth0 ip address. eth0 is a nat interface and all hosts of the cluster have the same ip address. This causes all hosts to get same ip address. Thanks to Flannel team, they helped me in identifying this issue, the issue and the solution are described here.

sudo /home/core/flannel/bin/flanneld -iface=eth1 &

Following command shows status of flannel service in host 1. It shows 11.0.13.0/24 subnet is allocated to this host.

core@core-01 ~ $ systemctl status flannel
● flannel.service
   Loaded: loaded (/run/fleet/units/flannel.service; linked-runtime; vendor preset: disabled)
   Active: active (running) since Sun 2015-01-18 13:23:52 UTC; 15min ago
  Process: 677 ExecStartPre=/usr/bin/etcdctl mk /coreos.com/network/config {"Network":"10.0.0.0/16"} (code=exited, statu
s=4)
 Main PID: 682 (flanneld)
   CGroup: /system.slice/flannel.service
           └─682 /home/core/flannel/bin/flanneld
Jan 18 13:23:52 core-01 etcdctl[677]: Error:  105: Key already exists (/coreos.com/network/config) [172807]
Jan 18 13:23:52 core-01 systemd[1]: Started flannel.service.
Jan 18 13:23:52 core-01 flanneld[682]: I0118 13:23:52.209542 00682 main.go:250] Installing signal handlers
Jan 18 13:23:52 core-01 flanneld[682]: I0118 13:23:52.211359 00682 main.go:119] Determining IP address of defau...erface
Jan 18 13:23:52 core-01 flanneld[682]: I0118 13:23:52.212317 00682 main.go:208] Using 10.0.2.15 as external interface
Jan 18 13:23:52 core-01 flanneld[682]: I0118 13:23:52.264848 00682 subnet.go:83] Subnet lease acquired: 10.11.32.0/20
Jan 18 13:23:52 core-01 flanneld[682]: I0118 13:23:52.279451 00682 main.go:218] UDP mode initialized
Jan 18 13:23:52 core-01 flanneld[682]: I0118 13:23:52.280220 00682 udp.go:239] Watching for new subnet leases
Jan 18 13:32:28 core-01 flanneld[682]: E0118 13:32:28.900556 00682 subnet.go:372] Error parsing subnet IP: /cor...etwork
Jan 18 13:36:30 core-01 flanneld[682]: I0118 13:36:30.878485 00682 udp.go:264] Subnet added: 11.0.13.0/24

Following is my ifconfig output in the host 1. Flannel1 is the interface created now, Flannel0 was created earlier.

core@core-01 ~ $ ifconfig
eth0: flags=4163<UP,BROADCAST,RUNNING,MULTICAST>  mtu 1500
        inet 10.0.2.15  netmask 255.255.255.0  broadcast 10.0.2.255
        inet6 fe80::a00:27ff:fe8b:9c8b  prefixlen 64  scopeid 0x20
        ether 08:00:27:8b:9c:8b  txqueuelen 1000  (Ethernet)
        RX packets 1385  bytes 117063 (114.3 KiB)
        RX errors 0  dropped 0  overruns 0  frame 0
        TX packets 930  bytes 96237 (93.9 KiB)
        TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0

eth1: flags=4163<UP,BROADCAST,RUNNING,MULTICAST>  mtu 1500
        inet 172.17.8.101  netmask 255.255.255.0  broadcast 172.17.8.255
        inet6 fe80::a00:27ff:fe3f:1cea  prefixlen 64  scopeid 0x20
        ether 08:00:27:3f:1c:ea  txqueuelen 1000  (Ethernet)
        RX packets 96931  bytes 21400016 (20.4 MiB)
        RX errors 0  dropped 2  overruns 0  frame 0
        TX packets 96037  bytes 13489117 (12.8 MiB)
        TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0

flannel0: flags=81<UP,POINTOPOINT,RUNNING>  mtu 1472
        inet 10.11.32.0  netmask 255.0.0.0  destination 10.11.32.0
        unspec 00-00-00-00-00-00-00-00-00-00-00-00-00-00-00-00  txqueuelen 500  (UNSPEC)
        RX packets 34  bytes 1904 (1.8 KiB)
        RX errors 0  dropped 0  overruns 0  frame 0
        TX packets 34  bytes 10370 (10.1 KiB)
        TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0

flannel1: flags=81<UP,POINTOPOINT,RUNNING>  mtu 1472
        inet 11.0.13.0  netmask 255.255.0.0  destination 11.0.13.0
        unspec 00-00-00-00-00-00-00-00-00-00-00-00-00-00-00-00  txqueuelen 500  (UNSPEC)
        RX packets 22  bytes 1232 (1.2 KiB)
        RX errors 0  dropped 0  overruns 0  frame 0
        TX packets 22  bytes 6710 (6.5 KiB)
        TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0

lo: flags=73<UP,LOOPBACK,RUNNING>  mtu 65536
        inet 127.0.0.1  netmask 255.0.0.0
        inet6 ::1  prefixlen 128  scopeid 0x10
        loop  txqueuelen 0  (Local Loopback)
        RX packets 39281  bytes 7234072 (6.8 MiB)
        RX errors 0  dropped 0  overruns 0  frame 0
        TX packets 39281  bytes 7234072 (6.8 MiB)
        TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0

Lets start the flannel service in host 2 and host 3. 11.0.45.0/24 gets allocated to host 2 and 11.0.48/24 gets allocated to host 3.

Lets look at Flannel environment variables in host 1.

core@core-01 ~ $ cat /run/flannel/subnet.env
FLANNEL_SUBNET=11.0.13.1/24
FLANNEL_MTU=1472

Next step is to restart the docker service with the new subnet that flannel has allocated. Following set of commands will do that.

source /run/flannel/subnet.env
sudo rm /var/run/docker.pid
sudo ifconfig docker0 ${FLANNEL_SUBNET}
sudo docker -d --bip=${FLANNEL_SUBNET} --mtu=${FLANNEL_MTU} &

Lets create 1 container in all hosts.

sudo docker run -ti ubuntu /bin/bash

Following is the ifconfig output of the 3 containers created. The containers on the respective hosts uses the FLANNEL_SUBNET that was created by Flannel when the service was started.

host1:
root@90f252b18220:/# ifconfig
eth0      Link encap:Ethernet  HWaddr 02:42:0b:00:0d:02
          inet addr:11.0.13.2  Bcast:0.0.0.0  Mask:255.255.255.0
          inet6 addr: fe80::42:bff:fe00:d02/64 Scope:Link
          UP BROADCAST RUNNING  MTU:1472  Metric:1
          RX packets:20 errors:0 dropped:0 overruns:0 frame:0
          TX packets:6 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:0
          RX bytes:1609 (1.6 KB)  TX bytes:508 (508.0 B)

lo        Link encap:Local Loopback
          inet addr:127.0.0.1  Mask:255.0.0.0
          inet6 addr: ::1/128 Scope:Host
          UP LOOPBACK RUNNING  MTU:65536  Metric:1
          RX packets:0 errors:0 dropped:0 overruns:0 frame:0
          TX packets:0 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:0
          RX bytes:0 (0.0 B)  TX bytes:0 (0.0 B)
host2:
root@4120b2ff44c8:/# ifconfig
eth0      Link encap:Ethernet  HWaddr 02:42:0b:00:30:02
          inet addr:11.0.48.2  Bcast:0.0.0.0  Mask:255.255.255.0
          inet6 addr: fe80::42:bff:fe00:3002/64 Scope:Link
          UP BROADCAST RUNNING  MTU:1472  Metric:1
          RX packets:20 errors:0 dropped:0 overruns:0 frame:0
          TX packets:6 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:0
          RX bytes:1642 (1.6 KB)  TX bytes:508 (508.0 B)

lo        Link encap:Local Loopback
          inet addr:127.0.0.1  Mask:255.0.0.0
          inet6 addr: ::1/128 Scope:Host
          UP LOOPBACK RUNNING  MTU:65536  Metric:1
          RX packets:0 errors:0 dropped:0 overruns:0 frame:0
          TX packets:0 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:0
          RX bytes:0 (0.0 B)  TX bytes:0 (0.0 B)
host3:
root@ecf8d38dc93d:/# ifconfig
eth0      Link encap:Ethernet  HWaddr 02:42:0b:00:2d:02
          inet addr:11.0.45.2  Bcast:0.0.0.0  Mask:255.255.255.0
          inet6 addr: fe80::42:bff:fe00:2d02/64 Scope:Link
          UP BROADCAST RUNNING  MTU:1472  Metric:1
          RX packets:22 errors:0 dropped:0 overruns:0 frame:0
          TX packets:7 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:0
          RX bytes:1782 (1.7 KB)  TX bytes:578 (578.0 B)

lo        Link encap:Local Loopback
          inet addr:127.0.0.1  Mask:255.0.0.0
          inet6 addr: ::1/128 Scope:Host
          UP LOOPBACK RUNNING  MTU:65536  Metric:1
          RX packets:0 errors:0 dropped:0 overruns:0 frame:0
          TX packets:0 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:0
          RX bytes:0 (0.0 B)  TX bytes:0 (0.0 B)

Lets make sure that container 1 is able to reach container 2 and container 3:

root@90f252b18220:/# ping -c1 11.0.48.2
PING 11.0.48.2 (11.0.48.2) 56(84) bytes of data.
64 bytes from 11.0.48.2: icmp_seq=1 ttl=60 time=1.43 ms

--- 11.0.48.2 ping statistics ---
1 packets transmitted, 1 received, 0% packet loss, time 0ms
rtt min/avg/max/mdev = 1.435/1.435/1.435/0.000 ms
root@90f252b18220:/# ping -c1 11.0.45.2
PING 11.0.45.2 (11.0.45.2) 56(84) bytes of data.
64 bytes from 11.0.45.2: icmp_seq=1 ttl=60 time=1.77 ms

--- 11.0.45.2 ping statistics ---
1 packets transmitted, 1 received, 0% packet loss, time 0ms
rtt min/avg/max/mdev = 1.774/1.774/1.774/0.000 ms

Following is an example of network config where we have specified a range of subnets and also we have changed the encapsulation from udp to vxlan.

curl -L http://127.0.0.1:4001/v2/keys/coreos.com/network/config -XPUT -d value='{
    "Network": "11.0.0.0/8",
    "SubnetLen": 20,
    "SubnetMin": "11.10.0.0",
    "SubnetMax": "11.99.0.0",
    "Backend": {
        "Type": "vxlan",
        "Port": 7890}}'

Thanks to the Flannel team for the nice solution. I am not sure about the following points:

  • As of now, Flannel seems to be specific to CoreOS cluster. I think at some point, the solution will extend beyond that.
  • I did not find a way to do multi-tenant container networking.

References:

Advertisements

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s