Macvlan and ipvlan are Linux network drivers that exposes underlay or host interfaces directly to VMs or Containers running in the host. In this blog, I will cover basics of macvlan and ipvlan, compare macvlan and ipvlan to Linux bridge and sub-interfaces and also show how to create these interfaces in Linux system. In the next set of blogs, I will cover how macvlan and ipvlan interfaces are used in Docker and CoreOS.
VM and Container networking
When running a baremetal server, host networking can be straightforward with few ethernet interfaces and a default gateway providing external connectivity. When we run multiple VMs in a host, it is needed to provide connectivity between VMs within the host and across hosts. On an average, the number of VMs in a single host does not exceed 15-20. When running Containers in a host, the number of Containers in a single host can easily exceed 100. It is needed to have sophisticated mechanism to interconnect Containers. Broadly, there are two ways for Containers or VMs to communicate to each other. In Underlay network approach, VMs or Containers are directly exposed to host network. Bridge, macvlan and ipvlan network drivers are examples of this approach. In Overlay network approach, there is an additional level of encapsulation like VXLAN, NVGRE between the Container/VM network and the underlay network.
Linux Bridge
Linux Bridge acts like a regular hardware switch with learning and also supports protocols like STP for loop prevention. In linux bridge implementation, VMs or Containers will connect to bridge and bridge will connect to outside world. For external connectivity, we would need to use NAT. The following picture shows 2 Containers connected to a Linux bridge with ethx interface providing external connectivity.
Macvlan
Macvlan allows a single physical interface to have multiple mac and ip addresses using macvlan sub-interfaces. This is different from creating sub-interface on a physical interface using vlan. With vlan sub-interfaces, each sub-interface belongs to a different L2 domain using vlan and all sub-interfaces have same mac address. With macvlan, each sub-interface will get unique mac and ip address and will be exposed directly in underlay network. Macvlan interface is typically used for virtualization applications and each macvlan interface is connected to a Container or VM. Each container or VM can directly get dhcp address from a common server as the host would do. This would help customers who want Containers to be part of their traditional network with the IP addressing scheme that they already have. Macvlan has 4 types(Private, VEPA, Bridge, Passthru). Commonly used type is Macvlan bridge which allows for endpoints in a single host to be able to talk to each other without packets leaving the host. For external connectivity, underlay network is used. Following picture shows two Containers using macvlan bridge to communicate with each other as well as to outside world. Both Containers will directly get exposed in underlay network using Macvlan sub-interfaces.
Following is an example for creating 2 macvlan bridge sub-interfaces. These examples were tried on Ubuntu 14.04.
sudo ip link add mymacvlan1 link eth0 type macvlan mode bridge sudo ip link add mymacvlan2 link eth0 type macvlan mode bridge sudo ifconfig mymacvlan1 up sudo ifconfig mymacvlan2 up
Following output shows the interface related details for parent interface and the two macvlan sub-interfaces. As we can see, the two macvlan sub-interfaces get unique mac address that is different from parent interface eth0.
$ ip link show mymacvlan1
19: mymacvlan1@eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UNKNOWN mode DEFAULT group default
link/ether 7e:33:b3:7d:f9:80 brd ff:ff:ff:ff:ff:ff
$ ip link show mymacvlan2
20: mymacvlan2@eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UNKNOWN mode DEFAULT group default
link/ether de:1a:30:bc:a3:a0 brd ff:ff:ff:ff:ff:ff
$ ifconfig eth0
eth0 Link encap:Ethernet HWaddr 08:00:27:85:47:0a
inet addr:192.168.0.101 Bcast:192.168.0.255 Mask:255.255.255.0
inet6 addr: fe80::a00:27ff:fe85:470a/64 Scope:Link
Following picture shows an example where macvlan sub-interface works together with vlan sub-interface. Containers c1, c2 are connected to underlay interface ethx.1 and Containers c3, c4 are connected to underlay interface ethx.2.
Following commands creates 2 vlan sub-interfaces and creates two macvlan bridge subinterface on each vlan sub-interface as shown in the picture above:
sudo vconfig add eth2 10 sudo vconfig add eth2 20 macvlan with subinterface: sudo ip link add mymacvlan1 link eth2.10 type macvlan mode bridge sudo ip link add mymacvlan2 link eth2.10 type macvlan mode bridge sudo ifconfig mymacvlan1 up sudo ifconfig mymacvlan2 up sudo ip link add mymacvlan3 link eth2.20 type macvlan mode bridge sudo ip link add mymacvlan4 link eth2.20 type macvlan mode bridge sudo ifconfig mymacvlan3 up sudo ifconfig mymacvlan4 up
Following output shows the relevant interface details:
$ ip link show eth2.10
24: eth2.10@eth2: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP mode DEFAULT group default
link/ether 08:00:27:32:b9:d3 brd ff:ff:ff:ff:ff:ff
smakam14@myubuntu:~$ ip link show eth2.20
25: eth2.20@eth2: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP mode DEFAULT group default
link/ether 08:00:27:32:b9:d3 brd ff:ff:ff:ff:ff:ff
smakam14@myubuntu:~$ ip link show mymacvlan1
26: mymacvlan1@eth2.10: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UNKNOWN mode DEFAULT group default
link/ether 26:2a:eb:a8:19:18 brd ff:ff:ff:ff:ff:ff
smakam14@myubuntu:~$ ip link show mymacvlan2
27: mymacvlan2@eth2.10: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UNKNOWN mode DEFAULT group default
link/ether 3e:a4:35:93:90:7c brd ff:ff:ff:ff:ff:ff
smakam14@myubuntu:~$ ip link show mymacvlan3
28: mymacvlan3@eth2.20: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UNKNOWN mode DEFAULT group default
link/ether 3e:07:83:fc:72:2a brd ff:ff:ff:ff:ff:ff
smakam14@myubuntu:~$ ip link show mymacvlan4
29: mymacvlan4@eth2.20: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UNKNOWN mode DEFAULT group default
link/ether fe:92:b8:50:ef:7b brd ff:ff:ff:ff:ff:ff
Comparing Macvlan to Bridge
Bridges can be used for complex scenarios where it is needed to provide connectivity across hosts and be able to run L2 protocols on the bridge. Macvlan is used for more simple scenarios where it is needed to expose endpoints directly to the external network. With Bridge, it is needed to use NAT for external connectivity. With macvlan, since endpoints are directly mapped to underlay network, external connectivity can be done using same dhcp server and switch that the host uses.
ipvlan
ipvlan is similar to macvlan with the difference being that the endpoints have the same mac address. ipvlan supports L2 and L3 mode. In ipvlan l2 mode, each endpoint gets the same mac address but different ip address. In ipvlan l3 mode, packets are routed between endpoints, so this gives better scalability.
Trying out ipvlan:
ipvlan need kernel support > 4.2. I tried out ipvlan in Ubuntu 16.04.
Following commands sets up two ipvlan l2 sub-interfaces:
sudo ip link add myipvlan1 link enp0s3 type ipvlan mode l2 sudo ifconfig myipvlan1 up sudo ip link add myipvlan2 link enp0s3 type ipvlan mode l2 sudo ifconfig myipvlan2 up
Following output shows the interface related outputs for the ipvlan interfaces. As we can see, mac address is same for both ipvlan sub-interface.
$ ip link show
2: enp0s3: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP
mode DEFAULT group default qlen 1000
link/ether 08:00:27:79:da:82 brd ff:ff:ff:ff:ff:ff
3: myipvlan1@enp0s3: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue st
ate UNKNOWN mode DEFAULT group default qlen 1000
link/ether 08:00:27:79:da:82 brd ff:ff:ff:ff:ff:ff
4: myipvlan2@enp0s3: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue st
ate UNKNOWN mode DEFAULT group default qlen 1000
link/ether 08:00:27:79:da:82 brd ff:ff:ff:ff:ff:ff
Comparing ipvlan to macvlan
ipvlan should be used in cases where some switches restrict the maximum number of mac address per physical port due to port security configuration. macvlan needs to be used in cases where common dhcp server is used since dhcp server would need unique mac address which ipvlan does not have.
Sreenivas Makam's Blog 
Question: Wouldn’t macvlan reintroduce the MAC address explosion that gets introduced because of virtualization at more than 5 times the scale when implemented with containers vs VMs?
Yes, thats correct. Macvlan is not a solution for general purpose containers. It is more needed for cases where enterprise is moving from VMs to Containers and they want a similar networking model and want to expose underlay network directly to containers. Another usecase is that the enterprise want to use their current ip address scheme and existing dhcp server.
You’re wrong when you say that bridge interfaces need NAT to connect to the underlay network. a bridge is basically a switch as you say so the VM/container is directly available with it’s own mac-address on the underlay network, just like macvlan do.
hi Laurent
My statement was in the context of Containers with Docker bridge.
With Docker bridge, underlay network is not exposed to Containers. If Container wants to reach external network, masquerading is done. If external world wants to access service exposed by containers, port mapping is done.
With macvlan, containers have an interface directly in underlay network.
Regards
Sreenivas
What would be the difference between – ipvlan l2 mode and creating sub-interface on a physical interface using vlan ? as in both cases endpoint/subinterface get the same MAC address as physical interface.
hi Vikrant
With sub-interface using vlan, the sub-interfaces cannot talk to each other at L2. With ipvlan l2 mode, ipvlan sub-interfaces can talk to each other at L2.
regards
Sreenivas
Amazing! Thanks!
Hi, I have a Windows 10 Pro Laptop. I installed Docker CE 17.09.1-ce-win42 (14687). It is community edition. I am able to create and use macvlan when docker is in “Linux Containers” Mode. But when I switch back to “Windows Containers” mode the mac vlan drivers are not available and unable to create / use macvlan network.
My requirement is to run container images of windows server core / nanoserver … connecting to macvlan driver so that I can assign to these containers IP addresses in the same network range of my local network. Please suggest / guide me how to achieve it.
Thank you,
Vijaya Bhasker G
Can you say that macvlan supports the implementation of RFC compliant VRRP without the need for gratuitous ARP
I’ll be figuring this out shortly but any advice is appreciated.. Is it possible with ipvlan to transfer the host ip to a container? Im setting open VPN up and want to end up with a little cluster where the host isn’t directly accessible. VPN into container, ssh over that link. I’m considering making a seperat bridge for once I’m in and maybe ssh tunnel connects to that so then my workstation or laptop has access to all the web back ends but only by passing 2 layers. It occurs to me I probably need a bridge on the VPN container to nat internet to host for updates but if it works it should be pretty slick, none of the web servers will have anything but 80 and 443 open and only ntp, dns, and VPN on the draw bridge containers external. At the moment though I’m having trouble geting any sort of firewall to behave in the containers though and I have to wonder if ipvlan 3s defeats some of the goal…
Hello, I really enjoy your blog entries as they open my eye to advanced docker networking.
I’m currently trying to deploy some containers as part of our business network.
When I create a macvlan network and start the container I expected the container to be able to ping gateway and internet (e.g. http://www.google.de) without further configuration but seem to fail. Could you point me into the right direction?
“`
docker network create -d macvlan –subnet=192.168.0.0/24 –gateway=192.168.0.1 –ip-range=192.168.0.1.128/26 buisness_net
docker run -it –rm –net=buisness_net alpine ping -c1 8.8.8.8
“`
hello, using the same setup
ip link add myipvlan1 link enp0s3 type ipvlan mode l2
with auto dhcp
Listening on LPF/myipvlan1/08:00:27:d7:50:7b
Sending on LPF/myipvlan1/08:00:27:d7:50:7b
Sending on Socket/fallback
DHCPDISCOVER on myipvlan1 to 255.255.255.255 port 67 interval 7
DHCPDISCOVER on myipvlan1 to 255.255.255.255 port 67 interval 14
DHCPDISCOVER on myipvlan1 to 255.255.255.255 port 67 interval 11
DHCPDISCOVER on myipvlan1 to 255.255.255.255 port 67 interval 8
DHCPDISCOVER on myipvlan1 to 255.255.255.255 port 67 interval 12
DHCPDISCOVER on myipvlan1 to 255.255.255.255 port 67 interval 9
No DHCPOFFERS received.
No working leases in persistent database – sleeping.
The network doesnt recieve a local network ip address from the router (vm debian 9)
However if i use the same setup with macvlan the router will supply a new IP as it has a different mac address.
Shouldnt the IPVLAN supply a new ipaddress or its getting confused with parents mac?
did ipvlan and macvlan support to integrate with KVM？thanks
You kow how to use ip commands, you don’t need to use ifconfig
[
Reblogged this on IT Blog.
Hello! Any idea about how to do only this part:
—
ip link add mymacvlan1 link eth0 type macvlan mode bridge
ip link add mymacvlan2 link eth0 type macvlan mode bridge
—
…with systemd-networkd (Ubuntu 20.04)?
Too bad that Netplan doesn’t support MACVLAN yet.
I tried to follow a similar VLAN example from ArchLinux:
https://wiki.archlinux.org/index.php/VLAN#Persistent_Configuration
So, I created:
/etc/systemd/network/eth0.network with:
—
[Match]
Name=eth0
[Network]
MACVLAN=mymacvlan0
MACVLAN=mymacvlan1
—
/etc/systemd/network/mymacvlan0.netdev with:
—
[NetDev]
Name=mymacvlan0
Kind=macvlan
[MACVLAN]
Mode=bridge
—
/etc/systemd/network/mymacvlan1.netdev with:
—
[NetDev]
Name=mymacvlan1
Kind=macvlan
[MACVLAN]
Mode=bridge
—
…doesn’t work, even after a reboot.
Then, I also tried to add mymacvlan0.network and mymacvlan1.network files, no success.
mymacvlan0.network with:
—
[Match]
Name=mymacvlan0
[Network]
DHCP=no
—
…still nothing.
I found an ugly workaround though!
—
$ cat /etc/networkd-dispatcher/routable.d/10-macvlan-interfaces.sh
#! /bin/bash
ip link add mymacvlan0 link eth0 type macvlan mode bridge
ip link add mymacvlan1 link eth0 type macvlan mode bridge
—
LOL – But I would like to know how to do it via systemd files…
Cheers!
Any theoretical performance, latency or overhead difference between macvlan and ipvlan? Appreciate any input on this.