Bangalore meetup – Docker Networking Troubleshooting presentation

Docker Bangalore meetup is a very active group dedicated to topics around Docker and the ecosystem around it. There was a meetup conducted yesterday at IBM office. There was a mix of topics presented including Moby, Linuxkit, Docker for Windows and Docker multi-stage builds. Thanks to Neependra for organizing the meetup, Neependra created this meetup group and has been passionately doing it from the early days of Docker. Thanks to IBM for hosting the meetup, IBM has been a very active contributor to Container and Docker ecosystem.

I did a presentation on “Docker Networking – Common issues and troubleshooting considerations”. The feedback received in the session was very positive. Following were some reasons that I did this presentation:

  • Networking is a complex topic and Docker networking is continuously evolving in every release and this makes it difficult for folks to figure out the right feature to use for their use-case.
  • When applications are taken from development to production, networking needs change and the typical approaches don’t always help.
  • Enterprise customers have lot of legacy applications and the networking needs to satisfy connecting legacy non-containerized applications with the new container based micro services.

Following is the approach and references I took to prepare the presentation:

  • Over the years, I have used some techniques for common networking stuff including Docker containers and I have captured this.
  • I went through the discussions in stackoverflow and docker forums and tried to find the common networking questions that folks ask.
  • Docker has a great documentation that includes Docker’s internal documentation, white papers and great set of blogs. I referred this extensively.
  • Dockercon videos around networking topics.

Following link has the slides and this link has the video recording to my presentation. The slides in the recording is not very clear. I would very much appreciate if folks can provide feedback and also provide suggestions on additional Docker networking tips that can be covered. I will try to keep the slides updated based on what I learn/understand in the future.

Following are some of my earlier blogs/presentations that you can refer to get more understanding on Docker Networking recent features:

Docker CE – Installing Test(“RC”) version

Starting with Docker 17.03, Docker introduced Community edition(CE) and Enterprise edition(CE) version of their Docker software. The release numbering also changed. From Docker 1.13.1, we jump to 17.03 version. Docker CE is the free version while Docker EE is the commercially supported enterprise version. Docker enterprise edition comes in different flavors based on the cost. Please refer this link for details on comparison between different Docker editions and the supported platforms for each editions. Both Docker CE and EE follow time based release schedule. Docker CE has 3 editions. CE “stable” edition gets released once every 3 months. CE “edge” edition gets released once every month. CE “test” edition is a release candidate that gets folded into “edge” and “stable” versions. I have used Docker release candidate(“test” edition) to try out new features before they get released. The steps to install release candidate Docker version is slightly different from installing “stable” and “edge” versions. Docker CE 17.06.0-ce-rc2 got released few days back and I have started trying out the new features in this version. This is a precursor to 17.06 release that will happen in few weeks. In this blog, I will cover installation steps for Docker CE edition release candidate software versions. I have focused on 17.06.0-ce-rc2, but the steps applies to any release candidate versions. The 3 approaches I have tried are installation from Docker static binaries, Docker machine with boot2docker and installation in Ubuntu platform with package manager.

Installation using Docker machine

When Docker RC version is released, the corresponding boot2docker image also gets released. I used the steps below to to the installation.

docker-machine create -d virtualbox --virtualbox-boot2docker-url https://github.com/boot2docker/boot2docker/releases/download/v17.06.0-ce-rc2/boot2docker.iso docker-rc2

I have used docker-machine 0.10.0. I have tried the above steps in both Linux and Windows platforms.

Installation using Package manager

This approach is used to install on native Linux systems. I tried this on Ubuntu 14.04 system, the steps below are specific to Ubuntu platform. The steps should be similar for other Linux flavors as well using the corresponding package manager associated with the flavor. To make it easy to move between Docker “stable”, “edge” and “test” versions, I remove the old Docker version and then install the new version. Following are the steps I followed to move from Docker “edge” 17.05-ce version to “test” 17.06-ce-rc2.

Remove old Docker version:

sudo apt-get -y remove docker-ce

Remove “edge” from repository list:

sudo add-apt-repository --remove \
       "deb [arch=amd64] https://download.docker.com/linux/ubuntu \
       $(lsb_release -cs) \
       edge"

Add “test” to repository list:

sudo add-apt-repository \
       "deb [arch=amd64] https://download.docker.com/linux/ubuntu \
       $(lsb_release -cs) \
       test"

Update and install Docker:

sudo apt-get update
sudo apt-get -y install docker-ce

The install would pick the latest version associated with “stable”, “edge” or “test”. The procedure above can be used to migrate from any latest combination of “stable”, “edge” or “test” channels.

Installation using Static binary

This approach is advisable only for testing purposes. I followed the steps in this link for the installation.

Following are the commands I used for installation in Ubuntu 14.04:

export DOCKER_CHANNEL=test
curl -fL -o docker.tgz "https://download.docker.com/linux/static/${DOCKER_CHANNEL}/x86_64/docker-17.06.0-ce-rc2-x86_64.tgz"
tar --extract --file docker.tgz --strip-components 1 --directory /usr/local/bin/

Docker binaries would be in /usr/local/bin. When Docker is installed using package manager, docker binaries are in /usr/bin. If /usr/local/bin is higher up in the path, this version would be picked. This approach allows us to switch between versions easily.

Following is the Docker version running after installation using any of the 3 above approaches:

$ docker version
Client:
 Version:      17.06.0-ce-rc2
 API version:  1.30
 Go version:   go1.8.3
 Git commit:   402dd4a
 Built:        Wed Jun  7 10:04:51 2017
 OS/Arch:      linux/amd64

Server:
 Version:      17.06.0-ce-rc2
 API version:  1.30 (minimum version 1.12)
 Go version:   go1.8.3
 Git commit:   402dd4a
 Built:        Wed Jun  7 10:03:45 2017
 OS/Arch:      linux/amd64
 Experimental: true

If there are any Docker topics that you would like more details, please let me know.

Compare Docker for Windows options

As part of Dockercon 2017, there was an announcement that Linux containers can run as hyperv container in Windows server. This announcement made me to take a deeper look  into Windows containers. I have worked mostly with Linux containers till now. In Windows, I have mostly used Docker machine or Toolbox till now. I recently tried out other methods to deploy containers in Windows. In this blog, I will cover different methods to run containers in Windows, technical internals on the methods and comparison between the methods. I have also covered Windows Docker base images and my experiences trying the different methods to run Docker containers in Windows. The 3 methods that I am covering are Docker Toolbox/Docker machine, Windows native containers, hyper-v containers.

Docker Toolbox

Docker Toolbox runs Docker engine on top of boot2docker VM image running in Virtualbox hypervisor. We can run Linux containers on top of the Docker engine. I have written few blogs(1, 2) about Docker Toolbox before. We can run Docker Toolbox on any Windows variants.

Windows Native containers

Folks familiar with Linux containers know that Linux containers uses Linux kernel features like namespaces, cgroups. To containerize Windows applications, Docker engine for Windows needs to use the corresponding Windows kernel features. Microsoft worked with Docker to make this happen. As part of this effort, changes were made both on Docker and Windows side. This mode allows Windows containers to run directly on Windows server 2016. Windows server 2016 has the necessary container primitives that allows native Windows containers to run on it. Going forward, Microsoft will port this functionality to other flavors of Windows.

hyper-v containers

Windows Hyper-v container is a windows server container that runs in a VM. Every hyper-v container creates its own VM. This means that there is no kernel sharing between the different hyper-v containers. This is useful for cases where additional level of isolation is needed by customers who don’t like the traditional kernel sharing done by containers. The same Docker image and CLI can be used to manage hyper-v containers. Creation of hyper-v containers is specified as a runtime option. There is no difference when building or managing containers between windows server and hyper-v container. Startup times for hyper-v container is higher than windows native container since a new lightweight VM gets created each time. 1 common question that comes up is how is hyper-v container different from a general VM with virtualbox or hyper-v hypervisor and running container on top of it? Following are some differences as I see it:

  • hyper-v container is very light-weight.  This is because of the light-weight OS and other optimizations.
  • hyper-v containers do not appear as VMs inside hyper-v and cannot be managed by regular hyper-v tools.
  • The same Docker  CLI can be used to manage hyper-v containers. To some extent, this is true with Docker Toolbox and Docker machine. With hyper-v containers, its more integrated and becomes a single step process.

There are 2 modes of hyper-v container.

  1. Windows hyper-v container – Here, hyper-v container runs on top of Windows kernel. Only Windows containers can be run in this mode.
  2. Linux hyper-v container – Here, hyper-v container runs on top of Linux kernel. This mode was not available earlier and it was introduced as part of Dockercon 2017. Any Linux flavor can be used as the base kernel. Docker’s Linuxkit project can be used to build the Linux kernel needed for the hyper-v container. Only Linux containers can be run in this mode.

We cannot use Docker Toolbox and hyper-v containers at the same time. Virtualbox cannot run when “Docker for Windows” is installed.

Following picture shows illustration of different Windows container modes

windows_container_types

Following table captures the difference between different Windows Container modes

Windows mode/Feature Toolbox Windows native container hyper-v container
OS Type Any Windows flavor Windows 2016 server Windows 10 pro, Windows 2016 server
hypervisor/VM Virtualbox hypervisor No seperate VM for container VM runs inside hyper-v
Windows container Not possible Yes Possible in Windows hyper-v container
Linux container Yes Not possible Possible in Linux hyper-v container
Startup time Higher compared to windows native and hyper-v containers Least among the 3 options Between Toolbox and windows native containers

Hands-on

If you are using Windows 10 pro or Windows server 2016, you can install Docker for Windows from here. This installs Docker CE version and runs Docker for Windows in hyper-v mode. We can install using either the stable or edge channel. Docker for Windows was available earlier only for Windows 10. The edge channel added Docker for Windows for Windows server 2016 just recently. Once “Docker for Windows” is installed, we can switch between Linux and Windows mode with just a click of a button. As of now, Linux mode uses mobyLinuxVM, this will change later to hyper-v linux container mode. In order to run Hyper-V containers, the Hyper-V role has to be enabled in Windows. If the Windows host is itself a Hyper-V virtual machine, nested virtualization will need to be enabled before installing the Hyper-V role. For more details, please refer these 2 references(1, 2). As shown in the example of reference, we can start hyper-v containers by just specifying a run-time option in Docker.

docker run -it --isolation=hyperv microsoft/nanoserver cmd

If you are using Windows server 2016, Docker EE edition can be installed using the procedure here. I would advise using Docker EE for Windows server 2016 rather than using hyper-v container.

I have tried Docker Toolbox in Windows 7 Enterprise version. Docker Toolbox can be run in any version of Windows. Docker Toolbox installation also installs Virtualbox if its not already installed. Docker Toolbox can be installed from here. For Docker Toolbox hands-on example, please refer to my earlier blog here.

I tried out Windows native containers and hyper-v containers in Azure cloud. After I created a Windows 2016 server, I used the following commands to install Docker engine. These commands have to be executed from powershell in administrator mode.

Install-Module -Name DockerMsftProvider -Repository PSGallery -Force
Install-Package -Name docker -ProviderName DockerMsftProvider
Restart-Computer -Force

Following are some example Windows containers I tried:

docker run microsoft/dotnet-samples:dotnetapp-nanoserver
docker run -d --name myIIS -p 80:80 microsoft/iis

Since Azure uses hypervisor to host compute VM and the fact that nested virtualization is not supported in Azure, Docker for Windows cannot be used with Windows server 2016 in Azure.
I got following error when I started “Docker for Windows” in Linux mode.

Unable to write to the database. Exit code: 1
   at Docker.Backend.ContainerEngine.Linux.DoStart(Settings settings) in C:\gopath\src\github.com\docker\pinata\win\src\Docker.Backend\ContainerEngine\Linux.cs:line 243
   at Docker.Backend.ContainerEngine.Linux.Start(Settings settings) in C:\gopath\src\github.com\docker\pinata\win\src\Docker.Backend\ContainerEngine\Linux.cs:line 120
   at Docker.Core.Pipe.NamedPipeServer.<>c__DisplayClass8_0.b__0(Object[] parameters) in C:\gopath\src\github.com\docker\pinata\win\src\Docker.Core\pipe\NamedPipeServer.cs:line 44
   at Docker.Core.Pipe.NamedPipeServer.RunAction(String action, Object[] parameters) in C:\gopath\src\github.com\docker\pinata\win\src\Docker.Core\pipe\NamedPipeServer.cs:line 140

I was still able to use hyper-v containers in Azure in Windows mode in Windows server 2016. I am still not fully clear how this mode overcame the nested virtualization problem.

From Azure perspective, I would like to see these changes from Microsoft:

  • Azure supporting nested virtualization.
  • Allowing Windows 10 in Azure without MSDN subscription.

There was an announcement earlier this week at Microsoft Build conference that Azure will support nested virtualization in selected VM sizes. This is very good.

Windows base image

Every container has a base image that contains the needed packages and libraries. Windows containers supports 2 base images:

  1. microsoft/windowsservercore – a full blown Windows server with full .NET Framework support. The size is around 9 GB.
  2. microsoft/nanoserver – a minimal Windows server and .NET Core Framework. The size is around 600 MB.

Following picture from here shows the compatibility between Windows server OS, Container type and Container base image.

baseimage

As we can see from the picture, with hyper-v container, we can use only nanoserver container base image.

FAQ

Can I run Linux containers in Windows?

  • The answer depends on which Docker windows mode you are using. With Toolbox and hyper-v Linux containers, Linux containers can be run in Windows. With Windows native container mode, Linux containers cannot be run in Windows.

Which Docker for Windows mode should I use?

  • For development purposes, if there is a need to use both Windows and Linux containers, hyper-v container can be used. For production purposes, we should use Windows native container. If there is a need to have better kernel isolation for additional security, hyper-v container can be used. If you have a version of Windows that is neither Windows 10 or Windows server 2016, Docker Toolbox is the only option available.

Can we run Swarm mode and Overlay network with Windows containers?

  • Swarm mode support was added recently in Windows containers. Multiple containers across Windows hosts can talk over the Overlay network. This needs Windows server update as mentioned in the link here. The same link also talks about a mixed mode Swarm cluster with Windows and Linux nodes. We can have a mix of Windows and Linux containers talking to each other over the Swarm cluster. Using Swarm constraints scheduling feature, we can place Windows containers in Windows nodes and Linux containers in Linux nodes.

Is there an additional Docker EE license needed for Windows server 2016?

  • According to the article here, it is not needed. It is better to check as this might change. Obviously, Windows license has to be taken care separately.

References

Linux Docker base images

Recently, someone asked me how to create a Docker base image for a Linux variant that they are creating. In this blog, I will cover what a Linux base image is and how to create new base images.

Linux Docker base image

Following is a sample Dockerfile to create a Python webserver Docker container image.

FROM ubuntu:14.04

# Update the sources list
RUN apt-get update

# Update
RUN apt-get install -y python2.7 python-pip

# Install app dependencies
RUN pip install Flask==0.9.0

# Bundle app source
COPY simpleapp.py /src/simpleapp.py

EXPOSE  8000
CMD ["python", "/src/simpleapp.py", "-p 8000"]

In the above example, we have given “FROM ubuntu:14.04” in the first line. ubuntu:14.04 is the base image used here. 1 common question that I get asked is that does it mean that this container contains everything present in Ubunbu VM? That is not true. What ubuntu container image contains is the packages, libraries and tools associated with Ubuntu along with the root filesystem. To give a size comparison, Ubuntu container image is around 180mb while the Ubuntu VM size is around 1GB.

Docker hub contains base container images for all major distributions including Ubuntu, rhel, centos, debian etc. It is always better to take the official images as they would have the latest security patches. The official images are maintained by the distribution vendor who works closely with Docker.

Creating Linux Docker Container base image

Docker repository provides a script “mkimage.sh” that can be used to create base images for different Linux variants. When Docker is installed in Ubuntu 14.04, “mkimage.sh” is available in “/usr/share/docker-ce/contrib/”. Following output shows the “help” for “mkimage.sh”.

$ ./mkimage.sh --help
usage: mkimage.sh [-d dir] [-t tag] [--compression algo| --no-compression] script [script-args]
   ie: mkimage.sh -t someuser/debian debootstrap --variant=minbase jessie
       mkimage.sh -t someuser/ubuntu debootstrap --include=ubuntu-minimal --components=main,universe trusty
       mkimage.sh -t someuser/busybox busybox-static
       mkimage.sh -t someuser/centos:5 rinse --distribution centos-5
       mkimage.sh -t someuser/mageia:4 mageia-urpmi --version=4
       mkimage.sh -t someuser/mageia:4 mageia-urpmi --version=4 --mirror=http://somemirror/
       mkimage.sh -t someuser/solaris solaris

Each Linux distribution provides a helper script to create the base filesystem. For example, “debootstrap” is used for debian/ubuntu variants, “rinse” is used for centos variants. “mkimage.sh” uses the root filesystem created by helper script to import that to a Docker container. The helper script installs the necessary packages and sets up the root filesystem. If you are creating a new flavor of Linux and if it based on 1 of the major distributions, we can extend the helper script. Otherwise, we can create a new helper script based on the current examples. It will be good to contribute the new helper script back to “mkimage” in Docker repository.

Following is an example to create Debian Jessie base image with mkimage.sh:

sudo ./mkimage.sh -t smakam/debian:jessie debootstrap jessie

The above command will create a new container image “smakam/debian:jessie”. It pulls the necessary files from the repository.

References

Dockercon 2017 – My experiences

Dockercon 2017 was the first Docker global conference that I attended. The conference was hosted in Austin, Texas. It was a memorable experience and I had lot of fun attending the conference. In this blog, I will share some of my experiences from Dockercon 2017. I have covered details on important announcements, keynote demos, Cool hacks, Sessions that I attended, Security workshop conducted by me and Docker team and key takeaways for me.

Key announcements

Following were key announcements as part of Keynote sessions:

  • Moby opensource projectMoby is a framework to assemble specialized container systems. Docker is 1 of the assembled container systems from Moby. There can be other container systems that users can create. For example, 1 of the example demonstrated in keynote is to use Moby to build a container system to run Kubernetes on Mac. Moby is an effort to keep Docker open source projects and Docker product separate.
  • LinuxkitLinuxkit is a toolkit for building custom, minimal and immutable Linux distributions. This is used by Microsoft to run Linux containers in Windows. Linuxkit is 1 of the components of Moby that allows us to build a bootable container system. This system can be run either on bare-metal or on cloud.
  • IBM is running Docker in their powerpc and Z systems.
  • Oracle enterprise DB is available in Docker store and can be tried free for personal use.
  • I am glad to mention the Cisco announcements.  Cisco and Docker are partnering on Modernizing traditional applications(MTA) program. Contiv 1.0  is available as GA.

Keynote Demos

Live demos are a key part of Dockercon. These demos were done as part of keynote sessions from Solomon and Ben.

  • Multistage Docker build to reduce Docker image size and desktop to cloud integration for moving applications across Swarms.
  • Deploying an application securely with multiple services on Docker swarm cluster. The application was deployed with Docker compose using TLS, Secrets.
  • Secure supply chain using DDC, Security scan and Docker secrets
  • Deploying 3rd party VM applications with containers using image2docker and Docker datacenter. image2docker can do migration of VMs to Containers and this would be helpful for migrating legacy applications.

Hacks

Following 2 hacks were done by Docker captains. These were selected from the many hacks submitted for Dockercon.

  1. PWD – play with Docker
    PWD is a great tool for running Docker containers using browser without having to install Docker. This is great for workshops and it is also a good Docker beginner tool. For more details on PWD, please refer to my earlier blog here.
  2. FaaS – This is a framework for building serverless functions on Docker Swarm. The demo was a cool one with integration with Alexa service.

Sessions attended

Following are the sessions that I attended over the Dockercon week:

  • Cilium: Network and application security using BPF and XDP –
    • Berkeley packet filter(BPF) and extended data processing(XDP) runs in Linux kernel.
    • Learnt use cases of BPF where policy can be forced at network layer inside linux kernel using BPF.
    • Cilium can be used as Docker networking plugin.
    • XDP extends BPF to network drivers which makes packet filtering even faster. Facebook says that XDP is 10 times faster with switching packets.
  • Solving the storage problem for cloud-native applications – Portworx
    • Portworx is a container storage solution that is trying to solve the big problem of persistent container storage. This is a complex problem to solve and there are many players trying to address this problem.
  • Scaling App defense with intent based security – Twistlock
    • This session went into details of Twistlock container security platfrom. Dynamic secure policies can be created by Twistlock automatically.
  • Docker networking: from application plane to Data plane
    • Covered Docker networking from beginning to now including tools to debug common Docker networking issues.
  • Infinit’s next generation key value store
    • Covered how Infinit’s solution is unique, distributed and scalable. Object store and file system can be on top of key-value store, this is targeted for 4th quarter of this year.
  • Journey to Docker production: evolving your infrastructure and processes
    • Talk from Docker Captain Bret fisher – Explained the production considerations for small and big Docker clusters.
  • Container performance analysis – Netflix
    • Netflix tools to debug container performance. Covered tools like Netflix victor, titus, flame graphs.
  • From ARM to Z: multi-platform Docker swarm
    • Cross-platform containers using manifest tool. Same container image can be used across multiple platforms so that developer don’t need to remember platform details.
  • Building a secure app with Docker
    • Best practices to be followed for building secure applications

I am eagerly waiting to watch the recording of the other sessions.

Security workshop

I conducted Docker Security workshop along with Nigel, Nass, Matt. Nigel is a Docker captain and Nass and Matt are from Docker team. Around 50 folks attended the session. It was a 3 hr session with presentations and labs on different Docker security topics including Swarm mode, Content trust, Security scan, Networking, Secrets and Linux container security features. The labs were done on AWS cloud. The session was interactive and we got interesting questions from the audience. The labs and the slides are posted here.

What I enjoyed the most

  • Meeting the folks in person with whom I have interacted over mails and slack
  • Keynote demos
  • Interacting with other Docker captains. Docker has an amazing Captains group and I am privileged to be part of the group. From Bangalore, 2 other Docker captains Neependra and Ajeet also attended the conference. Following picture was taken in the Captains summit.

captains_picture

  • Captains discussion with Solomon Hykes.
  • Presenting and interacting with folks in Docker Security workshop
  • Seeing overall Docker excitement with attendees
  • Talking to companies in their booth and understanding container ecosystem
  • Everyday after conference party…

 

 

Comparing Docker deployment options in public cloud

Few weeks back, I gave a presentation in Container conference, Bangalore comparing different solutions available to deploy Docker in the public cloud.

Slides are available here. I have also put the steps necessary along with short video for each of the options in the github page here.

Abstract of the talk:

Containers provide portability for applications across private and public clouds. Since there are many options to deploy Docker Containers in public cloud, customers get confused in the decision making process. I will compare Docker machine, Docker Cloud, Docker datacenter, Docker for AWS, Azure and Google cloud, AWS ECS, Google Container engine, Azure Container service. A sample multi-container application will be deployed using the different options. The deployment differences including technical internals for each option will be covered. At the end of the session, the user will be able to choose the right Docker deployment option for their use-case.

Note:

  • I have focused mainly on Docker centric options in the comparison.
  • There are few CaaS platforms like Tectonic, Rancher that I have not included since I did not get a chance to try them.
  • Since all the solutions are under active development, some of the gaps will get covered by the solutions in the future.

Comparing Docker compose versions

In this blog, I have captured some of my learnings on Docker compose files and how they differ between versions. Docker compose is a tool used for defining and running multi-container Docker applications. I have used the famous multi-container voting application to illustrate the differences with compose versions.

Following are some questions that I have to tried to answer in this blog:

  • What is the difference between Compose versions 1, 2 and 3?
  • What is the difference between compose, stack and dab formats?
  • What are different ways to run compose files with different compose versions?
  • How does “docker stack deploy” really work?

Compose versions:

Following table captures the main differences between Compose versions:

Continue reading Comparing Docker compose versions