Vault – Use cases

This blog is a continuation of my previous blog on Vault. In the first blog, I have covered overview of Vault. In this blog, I will cover some Vault use cases that I tried out.

Pre-requisites:

Install and start Vault

I have used Vault 0.6 version for the examples here. Vault can be used either in development or production mode. In development mode, Vault is unsealed by default and secrets are stored only in memory. Vault in production mode needs manual unsealing and supports backends like Consul, S3.

Start Vault server:

Following command starts Vault server in development mode. We need to note down the root key that will be used later.

vault server -dev

As the name suggests, development mode is strictly for trying out Vault.

Enable authentication backends

Here, we enable authentication backends needed for this usecase. “Token” backend is enabled by default. We can enable backends by “vault auth-enable <backend name>”. Following command lists the enabled authentication backends for the use cases in this blog.

$ vault auth -methods
Path       Type      Default TTL  Max TTL  Description
approle/   approle   system       system   
github/    github    system       system   
token/     token     system       system   token based credentials
userpass/  userpass  system       system  

Enabling secret backends

Next, we enable secret backends needed for this use case. We have enabled “mysql”, other backends are enabled by default. Following command lists the enabled secret backends for the use cases in this blog.

$ vault mounts
Path        Type       Default TTL  Max TTL  Description
cubbyhole/  cubbyhole  n/a          n/a      per-token private secret storage
mysql/      mysql      system       system   
secret/     generic    system       system   generic secret storage
sys/        system     n/a          n/a      system endpoints used for control, policy and debugging

Use Cases

Use case 1 – AWS backend

This use case is for getting dynamic AWS IAM access keys. AWS Identity and Access Management (IAM) provides granular access to AWS account based on user and group. AWS allows creation of IAM policy to restrict access to specific AWS resources. AWS IAM credentials being present in configuration file exposes a security risk. Vault allows dynamic creation of AWS IAM credentials with specific lease period so that the application can either revoke the credential after use or Vault will automatically delete the IAM credential after lease expiry.

Following is the workflow:

  • Register AWS root credentials with Vault.
  • Create IAM policy based on the access needed.
  • Create AWS role in Vault with the IAM policy created before.
  • Get dynamic AWS IAM credentials using the role created in previous step.
Pre-requisite:

We need to have AWS account to try this. Creating new IAM account does not have AWS charges.

Example:

Following command configures the lease period for AWS IAM dynamic keys.

$ vault write aws/config/lease lease=1200s lease_max=3600s
Success! Data written to: aws/config/lease

In the above command, we have configured lease period as 1200 seconds, this causes secret deletion after 20 minutes.

Following AWS IAM policy allows the user to touch only EC2 resource in AWS.

policy.json:
{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Sid": "Stmt1426528957000",
      "Effect": "Allow",
      "Action": [
        "ec2:*"
      ],
      "Resource": [
        "*"
      ]
    }
  ]
}

Register root AWS key:

vault write aws/config/root \
    access_key=xxx \
    secret_key=xxx

Replace access key and secret key with your own AWS root keys.

Register IAM policy for EC2 access with Vault:

vault write aws/roles/deploy policy=@policy.json

“policy.json” is the file specified in previous step.

Following command shows the dynamic key generation from Vault:

$ vault read aws/creds/deploy 
Key            	Value
---            	-----
lease_id       	aws/creds/deploy/c44fb09f-7465-d1e5-bbb1-59291013ac12
lease_duration 	20m0s
lease_renewable	true
access_key     	AKIAJQKLF72XDNV74SIQ
secret_key     	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
security_token 	

In the above command output, we can see that the lease duration is 20 minutes as we specified in AWS configuration. This key will become invalid after 20 minutes. Using above access and secret key, we can access only AWS EC2 services. We cannot access any other AWS service like S3 with this set of credentials.

If we list IAM users from aws IAM cli (“aws iam list-users”), we can see the new IAM user created by Vault:

    {
            "UserName": "vault-root-deploy-1475168892-6220", 
            "Path": "/", 
            "CreateDate": "2016-09-29T17:08:14Z", 
            "UserId": "AIDAJBH4XTDI3HU5HV5I4", 
            "Arn": "arn:aws:iam::173760706945:user/vault-root-deploy-1475168892-6220"
        }

Use case 2 – Passing secrets to Container

One approach to pass secret tokens to Container is by using environment variable. This approach is not secure as environment variables gets logged and this can easily get into the hands of malicious users. Vault provides a new backend called Cubbyhole to overcome this issue.

Following is the workflow for using Cubbyhole backend:

  • Enable Cubbyhole secret backend. Vault enables this by default.
  • Create temporary token with use count of 2. The use count specifies number of times the specific token can be used.
  • Create permanent token.
  • Store permanent token in temporary token’s cubbyhole.
  • Pass temporary token to container application using environment variable.
  • Application reads permanent token from cubbyhole using temporary token. Even if temporary token is read by malicious user later, there is no use for it since the use count for temporary token would have expired. Out of the specified initial use count of 2 for temporary token, first count is used when writing the permanent token and the second count is used when reading the permanent token.
Cubbyhole manual example

Create temporary token as shown below:

$ vault token-create --policy=policy1 --use-limit=4
Key            	Value
---            	-----
token          	a20119a5-7954-8045-e6df-01615adee8ab
token_accessor 	abd78995-788f-6158-a6b0-258b895ae870
token_duration 	720h0m0s
token_renewable	true
token_policies 	[default policy1]

In the above command, we have used use-limit of 4. This means this token would expire after 4 accesses. I have created “policy1″in Vault before-hand.

Create permanent token:

$ vault token-create
Key            	Value
---            	-----
token          	fc8576ef-0d7a-87d8-f2de-ef4ad37c6dd9
token_accessor 	1ea0240f-5512-e10d-645c-91993d860877
token_duration 	0s
token_renewable	false
token_policies 	[root]

Store permanent token in temporary token’s cubbyhole:

$ vault auth 27e83eeb-beef-cdd1-f406-92ba3fb229a7
Successfully authenticated! You are now logged in.
token: 27e83eeb-beef-cdd1-f406-92ba3fb229a7
token_duration: 2591989
token_policies: [default, policy1]
sreeni@ubuntu:~/vault$ vault write cubbyhole/app  app-token=fc8576ef-0d7a-87d8-f2de-ef4ad37c6dd9
Success! Data written to: cubbyhole/app

Retrieve permanent token using temporary token:

$ vault auth 27e83eeb-beef-cdd1-f406-92ba3fb229a7
Successfully authenticated! You are now logged in.
token: 27e83eeb-beef-cdd1-f406-92ba3fb229a7
token_duration: 2591951
token_policies: [default, policy1]
sreeni@ubuntu:~/vault$ vault read cubbyhole/app
Key      	Value
---      	-----
app-token	fc8576ef-0d7a-87d8-f2de-ef4ad37c6dd9

If we do the token read 1 more time, it will fail since token access count is exceeded:

$ vault read cubbyhole/app
Error reading cubbyhole/app: Error making API request.

URL: GET http://127.0.0.1:8200/v1/cubbyhole/app
Code: 403. Errors:

* permission denied

In the above example, I have used use count of 4 since Vault CLI does not allow 1 command to read and authenticate at the same time. First use count is used to authenticate, second is to write the permanent token, third is used to authenticate and fourth is used to read the permanent token.

Cubbyhole Programmatic example

Following example uses programmatic way to use cubbyhole using Ruby APIs. I found this complete example here.
Create temporary token with use count of 2:

temp = deployer_client.auth_token.create({ :ttl => '15s', :num_uses => 2 })[:auth][:client_token]

Create permanent token:

# permanent token can be used any number of times w/ no ttl.
perm = deployer_client.auth_token.create({})[:auth][:client_token]

Store permanent token in cubbyhole using tempoary token:

# using the first use of token #1, store the permanent token in cubbyhole
temp_client = Vault::Client.new(address: vault_address, token: temp)
temp_client.logical.write("cubbyhole/app-token", { :token => perm })

Fetch permanent token using temporary token:

# get the permanent token to use to grab real secrets
app_temp_client = Vault::Client.new(address: vault_address, token: temp)
puts "using temporary token #{temp} to access permanent token"
perm_token = app_temp_client.logical.read("cubbyhole/app-token")[:data][:token]
Cubbyhole wrap response

With Vault 0.6 version, there is a new capability called wrap response where any secret can be wrapped inside a Cubby hole.

Following is an example:

Wrap permanent token:

$ vault token-create --policy=policy1 --wrap-ttl=60s --use-limit=2
Key                          	Value
---                          	-----
wrapping_token:              	8c92f8d9-1035-d135-531e-e44396a560ec
wrapping_token_ttl:          	1m0s
wrapping_token_creation_time:	2016-09-29 09:14:52.462898166 -0700 PDT
wrapped_accessor:            	13cdb1d2-3c6c-6a36-fd19-2fab10d570b9

Unwrap permanent token:

$ vault unwrap 8c92f8d9-1035-d135-531e-e44396a560ec
Key            	Value
---            	-----
token          	1e474c4c-2f11-5c69-043d-473b218022e9
token_accessor 	13cdb1d2-3c6c-6a36-fd19-2fab10d570b9
token_duration 	720h0m0s
token_renewable	true
token_policies 	[default policy1]

In the above command, we have unwrapped using wrapping token. “8c92f8d9-1035-d135-531e-e44396a560ec” is the temporary token and “1e474c4c-2f11-5c69-043d-473b218022e9” is the permanent token.

Use case 3 – Mysql secret access

In this example, we will generate Mysql role based secrets dynamically. Following picture illustrates the flow.

vault1

Following are the goals of this use case:

  • The application will use Vault to generate username and password with specific roles to access Mysql database. “Readonly” role should be able to only read mysql database entries. “Superuser” role should be able to read and modify mysql database entries.
  • mysql user credentials will be generated dynamically with a specific lease time using Vault. The application can destroy the credentials after its use. In case the application does not destroy, Vault will automatically destroy credentials after the lease time expiry of the secret.
  • Different authentication backends like userpass, Token, Github and Approle will be used to achieve the same goal of dynamically generating mysql credentials. “mysql” secret backend will be used here.
Example

Start mysql server:

We will start mysql server as Docker container mapping the container port to host port.

docker run --name mysql -p 3306:3306 -e MYSQL_ROOT_PASSWORD=mysql -d mysql:latest

Port 3306 gets exposed on the host machine.

Add mysql credentials to Vault:
Following command sets up the connection to mysql server with username “root” and password “mysql”. The username and password here should match with the root credentials we used when starting mysql server.

vault write mysql/config/connection \
    connection_url="root:mysql@tcp(127.0.0.1:3306)/"

Configure mysql secrets lease duration to 1 hour:

vault write mysql/config/lease \
    lease=1h \
    lease_max=24h

Create roles for mysql access:

In this step, we will associate mysql roles for “readonly” and “modify” policy. “readonly” role is allowed only select access in mysql. “modify” role is allowed all access to the database. Following command creates roles in Vault.

vault write mysql/roles/readonly \
    sql="CREATE USER '{{name}}'@'%' IDENTIFIED BY '{{password}}';GRANT SELECT ON *.* TO '{{name}}'@'%';"
vault write mysql/roles/modify \
    sql="CREATE USER '{{name}}'@'%' IDENTIFIED BY '{{password}}';GRANT ALL ON *.* TO '{{name}}'@'%';"

Create policy for mysql:
Following is the mysql readonly policy “mysqlreadonly.json” that allows read access to “mysql/creds/readonly”.

path "mysql/creds/readonly" {
  policy = "read"
}

Following is the mysql modify policy “mysqlmodify.json” that provides read access to “mysql/creds/modify”.

path "mysql/creds/modify" {
  policy = "read"
}

Write the policy to Vault:

Following set of commands writes the JSON policy specified in previous section to Vault.

vault policy-write mysqlmodify mysqlmodify.json
vault policy-write mysqlreadonly mysqlreadonly.json
Using Token based authentication

In this section, we will use token based authentication scheme to access mysql secrets.

Following 2 commands creates tokens for the 2 types of users:

vault token-create --policy=mysqlmodify
vault token-create --policy=mysqlreadonly

Following output shows the results:

$ vault token-create --policy=mysqlmodify
Key            	Value
---            	-----
token          	92456126-c387-199b-fb51-306eb1eeb921
token_accessor 	e44602ad-397d-e4b4-826a-c540e0e16853
token_duration 	720h0m0s
token_renewable	true
token_policies 	[default mysqlmodify]

$ vault token-create --policy=mysqlreadonly
Key            	Value
---            	-----
token          	778455d8-08a9-6407-4b99-c039f503b377
token_accessor 	c266a977-279f-e12f-d5b5-d17ce9e5cd2b
token_duration 	720h0m0s
token_renewable	true
token_policies 	[default mysqlreadonly]

Lets authenticate using readonly token:

$ vault auth 778455d8-08a9-6407-4b99-c039f503b377
Successfully authenticated! You are now logged in.
token: 778455d8-08a9-6407-4b99-c039f503b377
token_duration: 2591649
token_policies: [default, mysqlreadonly]

Now that we have authenticated, lets get the mysql credentails for readonly user:

$ vault read mysql/creds/readonly
Key            	Value
---            	-----
lease_id       	mysql/creds/readonly/18bfe188-f36b-88d3-d61c-5ca4842265da
lease_duration 	1h0m0s
lease_renewable	true
password       	00ee87a7-a2e8-1959-b94a-56431e6e71cf
username       	read-toke-1e1806

If we try to access credentials for modify user, we wont be able to get it since this user has access to only the readonly credentails.

$ vault read mysql/creds/modify
Error reading mysql/creds/modify: Error making API request.

URL: GET http://127.0.0.1:8200/v1/mysql/creds/modify
Code: 403. Errors:

* permission denied

Now, we can access the mysql database using the mysql client with the above credentials. We can read the database, but we are not able to modify the database as shown below.

$ mysql -h127.0.0.1 -P3306 -uread-toke-1e1806 -p00ee87a7-a2e8-1959-b94a-56431e6e71cf
Warning: Using a password on the command line interface can be insecure.
Welcome to the MySQL monitor.  Commands end with ; or \g.
Your MySQL connection id is 16
Server version: 5.7.15 MySQL Community Server (GPL)

Copyright (c) 2000, 2016, Oracle and/or its affiliates. All rights reserved.

Oracle is a registered trademark of Oracle Corporation and/or its
affiliates. Other names may be trademarks of their respective
owners.

Type 'help;' or '\h' for help. Type '\c' to clear the current input statement.

mysql> show databases;
+--------------------+
| Database           |
+--------------------+
| information_schema |
| alpha              |
| alpha1             |
| mysql              |
| performance_schema |
| sys                |
+--------------------+
6 rows in set (0.01 sec)

mysql> create database alpha2;
ERROR 1044 (42000): Access denied for user 'read-toke-1e1806'@'%' to database 'alpha2'

Now, lets authenticate as “modify” user using the appropriate token:

$ vault auth 92456126-c387-199b-fb51-306eb1eeb921
Successfully authenticated! You are now logged in.
token: 92456126-c387-199b-fb51-306eb1eeb921
token_duration: 2591261
token_policies: [default, mysqlmodify]

Now, lets try to get credentials to modify database.

$ vault read mysql/creds/modify
Key            	Value
---            	-----
lease_id       	mysql/creds/modify/85380045-5951-993c-ef3d-04f9c5b4bbed
lease_duration 	1h0m0s
lease_renewable	true
password       	5e73da55-df7b-ef04-0556-0674fc4c42f8
username       	modi-toke-42e9e3

Lets access the database and try to create a new database. Since we have used “modify” role, we are able to make changes to the database with this username and password.

$ mysql -h127.0.0.1 -P3306 -umodi-toke-42e9e3 -p5e73da55-df7b-ef04-0556-0674fc4c42f8
Warning: Using a password on the command line interface can be insecure.
Welcome to the MySQL monitor.  Commands end with ; or \g.
Your MySQL connection id is 17
Server version: 5.7.15 MySQL Community Server (GPL)

Copyright (c) 2000, 2016, Oracle and/or its affiliates. All rights reserved.

Oracle is a registered trademark of Oracle Corporation and/or its
affiliates. Other names may be trademarks of their respective
owners.

Type 'help;' or '\h' for help. Type '\c' to clear the current input statement.

mysql> show databases;
+--------------------+
| Database           |
+--------------------+
| information_schema |
| alpha              |
| alpha1             |
| mysql              |
| performance_schema |
| sys                |
+--------------------+
6 rows in set (0.00 sec)

mysql> create database alpha2;
Query OK, 1 row affected (0.01 sec)

Lease duration:

We can query the token and we can see that ttl is getting decremented. The “creation_ttl” reflects initial creation duration(720 hrs/2592000 seconds) and the “ttl” reflects pending time(2590930 seconds). The ttl for the token reflects the lease expiry of authentication token.

$ vault token-lookup 778455d8-08a9-6407-4b99-c039f503b377
Key             	Value
---             	-----
accessor        	c266a977-279f-e12f-d5b5-d17ce9e5cd2b
creation_time   	1474217534
creation_ttl    	2592000
display_name    	token
explicit_max_ttl	0
id              	778455d8-08a9-6407-4b99-c039f503b377
meta            	
num_uses        	0
orphan          	false
path            	auth/token/create
policies        	[default mysqlreadonly]
renewable       	true
ttl             	2590930

There is also a lease period for database authentication credentials. In this example, we have used 1 hour as the lease period. After this period, database credentials will be destroyed.

Destroy tokens and mysql credentials:

After we have accessed the database, we can revoke the secret as well as the token as shown below.

vault revoke mysql/creds/modify/85380045-5951-993c-ef3d-04f9c5b4bbed
vault revoke mysql/creds/readonly/18bfe188-f36b-88d3-d61c-5ca4842265da
vault token-revoke 778455d8-08a9-6407-4b99-c039f503b377
vault token-revoke 92456126-c387-199b-fb51-306eb1eeb921
Userpass authentication

In this approach, we will use Vault authentication using username and password.
Following 2 commands creates “readonlyuser” with “mysqlreadonly” policy and “superuser” with “mysqlmodify” policy:

vault write auth/userpass/users/readonlyuser \
    password=foo \
    policies=mysqlreadonly

vault write auth/userpass/users/superuser \
    password=foo \
    policies=mysqlmodify

Following command and the associated output shows authentication using the “readonlyuser”. We can see that the returned policy is “mysqlreadonly”.

$ vault auth -method=userpass \
>     username=readonlyuser \
>     password=foo
Successfully authenticated! You are now logged in.
The token below is already saved in the session. You do not
need to "vault auth" again with the token.
token: a6f5315a-1e97-fe03-b7e0-24505157b498
token_duration: 2591999
token_policies: [default, mysqlreadonly]

Following command and the associated output shows authentication using the “superuser”. We can see that the returned policy is “mysqlmodify”

$ vault auth -method=userpass \
>     username=superuser \
>     password=foo
Successfully authenticated! You are now logged in.
The token below is already saved in the session. You do not
need to "vault auth" again with the token.
token: 613ac80a-1fa8-7494-5212-182b4b5eadf2
token_duration: 2592000
token_policies: [default, mysqlmodify]
Using Approle authentication approach

In this approach, we will use Vault authentication using Approle.

This authentication approach is mainly used for machines where some machine property like mac address will be used to authenticate.
First step is to create the roles “readonlyuser” and “superuser” and associate with the respective policies.

vault write auth/approle/role/readonlyuser policies=mysqlreadonly
vault write auth/approle/role/superuser policies=mysqlmodify

Nest step is to create roleid for both type of roles:

$ vault read auth/approle/role/readonlyuser/role-id
Key    	Value
---    	-----
role_id	adebbe05-c929-107e-6164-e86e2dc563f7

$ vault read auth/approle/role/superuser/role-id
Key    	Value
---    	-----
role_id	6ec1966c-b00b-b37c-48db-16084bc57f64

We need to get secrets corresponding to the role as shown below.

$ vault write -f auth/approle/role/readonlyuser/secret-id
Key               	Value
---               	-----
secret_id         	ea99d1ec-ea78-cf4c-92ef-cb0964126960
secret_id_accessor	4253940a-235e-de26-ff94-0c6c1c7d339a

vault write -f auth/approle/role/superuser/secret-id
Key               	Value
---               	-----
secret_id         	1b87e79a-328d-9b10-6cd2-9b2e5e0e8dfa
secret_id_accessor	a71163ef-302b-71f4-8187-65f062d2a3e6

Now, we can authenticate using the above roleid and secretid. This will return the appropriate policies.

$ vault write auth/approle/login role_id=adebbe05-c929-107e-6164-e86e2dc563f7 secret_id=ea99d1ec-ea78-cf4c-92ef-cb0964126960
Key            	Value
---            	-----
token          	0634efb3-d630-2bcb-657a-db017873f7e6
token_accessor 	335a46dc-60ba-0973-b8f6-4e282da8bc30
token_duration 	20m0s
token_renewable	true
token_policies 	[default mysqlreadonly]

$ vault write auth/approle/login role_id=6ec1966c-b00b-b37c-48db-16084bc57f64 secret_id=1b87e79a-328d-9b10-6cd2-9b2e5e0e8dfa
Key            	Value
---            	-----
token          	2e958510-9f46-3b18-9022-45c2145edd8b
token_accessor 	c8c761d2-8264-0c86-124c-480da0b26f7a
token_duration 	20m0s
token_renewable	true
token_policies 	[default mysqlmodify]

Typically, roleid would be a property of the machine programmed into it by some configuration management system.

Using github authentication

In this approach, we will use Vault authentication using Github.

Github authentication scheme is used for user based authentication. Following are some internals on this scheme:

  • Github has concept of organizations, teams and members. Organization can have many teams and each team can have many members.
  • The first step in this scheme is generation of personal access token for the github user account. This token is needed while authenticating to Vault.
  • In this scheme, when user tries to authenticate to Vault, Vault contacts github server with specified username and authentication token. Github returns the teams that this specific user is part of. Vault will then return the policy associated with the specific team.

Following is the Org structure that I created in github along with policy returned by Vault.

vault2

When user “smakam” logs in, github returns team “local” to Vault. Vault would then identify policy associated with team “local”.

Following commands creates the organization, team and associates policy with the team.

vault write auth/github/config organization=sreeniorg
vault write auth/github/map/teams/local value=mysqlreadonly

Following command shows the output when we try to authenticate using token for user “smakam”. Here, we get policy returned as “mysqlreadonly”.

$ vault auth -method=github token=xxxx
Successfully authenticated! You are now logged in.
The token below is already saved in the session. You do not
need to "vault auth" again with the token.
token: 4803ddf8-e419-d1e5-7aae-8d2d42834966
token_duration: 2591999
token_policies: [default, mysqlreadonly]

References

Vault Overview

I have always loved Hashicorp’s Devops and cloud tools. I have used Vagrant, Consul, Terraform, Packer and Atlas before and I have written about few of them in my previous blogs. Vault is Hashicorp’s tool to manage secrets securely in a central location. Secret could be database credentials, AWS access keys, Consul api key, ssh private keys etc. It is necessary for secrets to be managed centrally and having strict control and audit policies. By having a separate tool to manage secrets, application developer don’t need to worry about security internals and leave it to Vault to manage secrets. In this blog, I will cover Vault overview and internals and in the next blog, I will cover some use cases that I tried out.

Vault Principles

Vault uses the following principles:

Key rotation/expiration

Vault can provide keys based on limited ttl and use count. Vault also allows for key renewals and rotation. These approaches keeps keys from staying permanent so that chances of malicious users using the keys is reduced. If Vault provided a secret with ttl duration of 1 hour, Vault will take care of revoking the secret after 1 hour.

Secure distribution

Vault provides TLS for secure access and wrapping using cubbyhole to distribute keys securely. This prevents man in the middle attacks.

Limit exposure

Vault’s default policy is to block all access and policy needs to be applied to selectively add access.

Access Detection

All access to Vault can be audited. This can assist with detection of any malicious activities. If a particular application or user or machine has accessed the secret, Vault will have the audit records for that access.

Break glass procedure

Vault provides facilities like locking out all access and key rotation in case the system gets compromised. The actual break glass procedure to be followed would be based on specific enterprise policy. This is very critical as enterprises needs to know what exactly has been compromised and to be sure that the system has been locked down after the compromise.

Vault Internals

Following are some internals on Vault:

Unsealing:

All secrets including keys, certificates, passwords are stored with encryption in central location.Vault’s initial unseal process is done manually using multiple unseal keys. This is done to prevent key compromise from a single source. Every other operation in Vault can be automated except the unseal process.

Policies:

Access to Vault is controlled using policies. Policies can be written in HCL(Hashicorp configuration language). HCL is compatible with JSON, so policies can be written in JSON. Vault root token has a root policy that provides access to all secrets in Vault. Access to secrets stored in Vault can be restricted with ACL policy by using authentication.

Authentication:

Authentication is necessary to tell Vault who we are. The authentication can either be user based or machine based. Vault has a common scheme for handling authentication and by using authentication backends, it keeps the frontend for authentication the same and the backend takes care of the specifics.

For user based authentication scenario, Vault provides username/password, token, github methods to authenticate. For machine scenario, Vault provides methods like approle, certificates to authenticate.

After successful authentication, Vault assigns a authorization policy which dictates what a particular user/machine can do with Vault. Based on the policy returned, user/machine can access secret backend to fetch the secrets.

Secret backend and Dynamic secrets:

Vault’s secret backend takes care of secret generation. Vault has a common scheme for generating secrets for different types of needs like AWS access, database access like mysql etc. The secrets are generated based on the type of backend. For example, with generic backend, secrets are manually generated by user. For AWS backend, dynamic secrets are generated by Vault using AWS IAM module.

Dynamic secrets are used with backends like AWS, mysql, cassandra, postgres. For dynamic secrets, it is necessary to feed in the backend root secret into vault. For example, AWS root access key should be registered to Vault for AWS backend. The advantage with dynamic secrets is that these secrets have a lease period which expire. Also, since unique dynamic secrets are generated per application, it is easier to audit and do breakglass procedure with dynamic secrets.

Auditing:

Vault’s audit backend takes care of logging all access and events in Vault. Vault supports file and syslog backends. Vault can be accessed externally using HTTP API with or without TLS. Vault can also be accessed programmatically using language libraries supported by Hashicorp.

Vault and Nomad Integration:

In a typical Container orchestration solution, orchestrator takes care of secret management and feeding the secrets to the container. Nomad is Hashicorp’s orchestration solution. In the current scheme, Vault token is specified as a environment variable in the Nomad job. This is not secure. Vault to Nomad integration is currently being developed and will be available soon. Following link walks through the details of Vault and Nomad integration.

Workflow

Following is a typical workflow for Vault administrator and user/machine to manage secrets:

Administrator:

  • Start vault and unseal it.
  • Add credentials for backend based on the type of backend(aws, mysql). For example, with AWS, we need to add AWS secret root keys to Vault.
  • Create policy to control access to backend based on user or machine identifier.

User/Machine:

  • Either user or machine logs into Vault based on credentials they have.
  • Vault associates a policy based on user/machine credentials with authentication backend.
  • User or application fetches necessary secrets from secret backend based on available policy.
  • For dynamic secrets, the invoked application revokes the secret or Vault destroys the secrets after lease duration.

In the next blog, I will cover some use cases that I tried with Vault.

References

Docker for AWS – Deployment options

In this blog, I will cover 5 different options to deploy Docker Containers in AWS infrastructure. There are pros and cons of each option and the goal in this blog is not to suggest that some options are better than others, but to highlight the suitable option for a particular use case. I have taken a sample multi-container application and deployed in all the 5 different models to illustrate this. Following are the 5 options/models discussed in this blog:

  1. Docker Machine for AWS
  2. Docker for AWS
  3. Docker cloud for AWS
  4. Docker Datacenter for AWS
  5. AWS ECS

I have separate blog for each of the above deployment options which are linked to this blog.

Sample application

Following is the sample application used in this blog:

docker_aws10

“client” service has 1 client container task. “vote” service has multiple vote container tasks. Both these services are deployed on a multi-node cluster. “client” service is used to access multi-container “vote” service. “vote” service can also be accessed through external load balancer. The goal of the sample application is to illustrate multi-node cluster, multi-container application, orchestration, container networking across hosts, external load balancing, service discovery and internal load balancing.

Docker-machine for AWS

Docker-machine has EC2 driver for creating a Docker node out of AWS. Docker node in this context means a AWS VM instance with Docker pre-installed. Docker-machine also sets up secure ssh access to the EC2 instance. Once the basic node setup is done, the user can either use traditional Swarm or Swarm mode for orchestration. In terms of integration, this approach provides minimal integration with AWS. This option is very easy to start with and useful for developers who want to try out Docker Containers in the AWS cloud. For more details on Docker-machine for AWS, please refer here.

Docker for AWS

As part of Docker 1.12 announcement, Docker released AWS Docker integration as beta software. With this software, Docker is trying to  simplify AWS integration by better integrating Docker with AWS services like load balancer, security groups, cloudwatch etc. Compared to docker-machine, this option provides close integration with AWS services. System containers running in the EC2 instances provides tight integration between user containers and AWS services. These system containers are added by Docker. For example, 1 of the system container listens to host exposed ports and automatically adds it to the AWS ELB. Currently, there are limited options to change the configuration setup. Hopefully, this will be improved when this comes out of beta phase. This option is useful for developers and operations folks who are used to both Docker tools as well as AWS services.  For more details on Docker for AWS, please refer here.

Docker Cloud for AWS

Docker cloud is a paid hosted service from Docker to manage Containers. Docker cloud can be used to manage nodes in the cloud or in local data center. By providing AWS credentials, Docker cloud can create and manage AWS EC2 instances and Docker containers will be created on these EC2 instances. Since Docker cloud was an acquisition, it does not use some of the Docker ecosystem software. In terms of integration with AWS, Docker cloud provides minimal integration at this point. Docker cloud provides a lot of value in terms of simplifying infrastructure management and deployment of complex micro-services. This option is useful for folks who want a simple hosted solution with minimal integration around AWS services. For more details on Docker cloud for AWS, please refer here.

Docker Datacenter for AWS

Docker Datacenter is Docker’s enterprise grade CaaS(Container as a service) solution where they have integrated their open source software with some proprietary software and support to make it into a commercial product. Docker Datacenter is an application comprised of Universal control plane(UCP), Docker Trusted registry(DTR), Docker engine and supporting services running as Containers. Docker Datacenter for AWS means running these system services on AWS EC2 instances along with running the application containers which the system services manages. Docker Datacenter is an enterprise grade solution with multi-tenancy support and it provides nice integration with Light weight directory access protocol(LDAP) and Role based access control(RBAC). Docker Datacenter for AWS provides a secure solution with clear separation between private and public subnet. Docker Datacenter also provides high availability with multiple UCP controllers and DTR replicas. This option is useful for Enterprises who want a production grade Docker deployment with tight integration around AWS services. For more details on Docker Datacenter for AWS, please refer here.

AWS ECS

AWS has EC2 Container service(ECS) for folks who want to deploy Docker containers in AWS infrastructure. With ECS, Amazon provides its own scheduler to manage Docker containers. ECS integrates very well with other AWS services including load balancer, cloudwatch, cloudformation templates etc. The workflow is little different for folks used to Docker tools. For folks who want to use the Docker ecosystem tools, this option is not suitable.This option can be very powerful once ECS integrates with all AWS services, it can allow seamless movement between VMs and Containers.  The task and service definition file formats does not seem flexible.  The good thing with ECS is users are not charged for Containers or for ECS, but charged only for the EC2 instances. This option seems more suitable for folks who have been using AWS for a long time and want to try out Docker containers. For more details on AWS ECS, please refer here.

Following table is a brief comparison between the 5 solutions:

Property/Solution Docker Machine for AWS Docker for AWS Docker Cloud for AWS Docker Datacenter for AWS AWS ECS
Docker version Latest Docker version(1.12.1 in my case), no flexibility to select Docker version Latest Docker version(1.12 in my case), no flexibility to select Docker version Uses 1.11, no flexibility to select Docker version Uses 1.11, no flexibility to select Docker version Uses 1.11, no flexibility to select Docker version
Orchestration Traditional Swarm using external discovery or Swarm mode can be used. Needs to be setup manually. Swarm mode is integrated and available automatically. Uses proprietary scheduler. Traditional Swarm is used. KV store is automatically setup. Uses AWS proprietary scheduler. There is a plan to integrate external schedulers.
Networking Docker Libnetwork Docker Libnetwork Uses Weave. Docker Libnetwork AWS VPC based networking
Application definition Compose and DAB Compose and DAB Stackfile Compose Task and Service definition files
Integration with AWS Services Very minimal integration Good integration. VPC, ELB, Security groups, IAM roles gets automatically setup. Minimal integration. Good integration. Availability zones, VPC, ELB, Security groups, IAM roles gets automatically setup. Very good integration. Integration available with classic or application load balancer, Cloudwatch logs, autoscaling groups.
Cost (This is in addition to EC2 instance cost) Free Beta phase currently, not sure of the cost. 1 node and 1 private repository free, charges applicable after that. Paid service, free for 30 day trial period Free

Following are some things that I would like to see:

  • AWS ECS allowing an option to use Swarm scheduler.
  • Docker for AWS, Docker cloud and Docker Datacenter using a common networking and orchestration solution.
  • It will be good to have a common task definition format for applications or an option to automatically convert between the formats internally. This allows for users to easily move between these options and use the same task definition format.

References

AWS ECS – Docker Container service

In this blog, I will cover AWS ECS Docker Container service. ECS is an AWS product. This blog is part of my Docker for AWS series and uses the sample voting application for illustration.

AWS has EC2 Container service(ECS) for folks who want to deploy Docker containers in AWS infrastructure. For basics of AWS ECS, you can refer to my previous blog here. With ECS, Amazon provides its own scheduler to manage Docker containers. ECS integrates very well with other AWS services including load balancer, logging service, cloudformation templates etc. AWS recently introduced Application load balancer(ALB) that does L7 load balancing and this integrates well with ECS. Using ALB, we can load balance services directly across Containers. With ECS, users get charged for the EC2 instances and not for the Containers.

To demonstrate ECS usage, we will deploy voting service application in ECS cluster.

Continue reading AWS ECS – Docker Container service

Docker Datacenter for AWS

In this blog, I will cover Docker datacenter usage with AWS. This blog is part of my Docker for AWS series and uses the sample voting application for illustration.

Docker Datacenter is Docker’s enterprise grade CaaS(Container as a service) solution where they have integrated their open source software with some proprietary software and support to make it into a commercial product. Docker Datacenter can be deployed on-premise or in cloud providers like AWS. Docker Datacenter is available free for 30 day trial period.

Docker Datacenter Architecture

Following picture shows the core components of Docker Datacenter:

docker_aws2

(Image from Docker webpage)

Following are the key components of Docker Datacenter:

UCP(Universal control plane):

UCP provides controller to manage the platform and it integrates well with Light weight directory access protocol(LDAP) and Role based access control(RBAC). UCP also provides multi-tenancy support. This allows enterprises to integrate Docker Datacenter with their current user management solutions. Swarm is used for orchestration. UCP provides a nice GUI and same Docker APIs can be used to control UCP. Multiple UCP controllers can operate in HA mode. UCP also provides monitoring and logging support for Containers.

DTR(Docker Trusted registry):

DTR provides a secure Docker image repository and it integrates well with UCP and Docker engine.

Docker engine:

This is the open source Docker engine with commercial support.

Docker Datacenter for AWS

Docker Datacenter is an application comprised of UCP, DTR, Docker engine and supporting services running as Containers. Docker Datacenter for AWS means running these system services on AWS instances along with running the application containers which the system services manages. System services and application containers will be run on separate instances for resiliency purposes.

Following picture shows the architecture of Docker Datacenter for AWS:

docker_aws3

(Picture from Docker Datacenter documentation)

Following are some important notes on Docker Datacenter for AWS architecture:

  • There are 3 kinds of EC2 cluster instances created. UCP controller cluster, DTR cluster, application cluster. UCP controller cluster runs UCP application, DTR cluster runs DTR application, application cluster runs user containers. UCP and DTR are developed as a micro-service with multiple containers. The EC2 instances are spread between 2 availability zones for HA.
  • Each of the cluster resides in the private network for security purposes. For each of the cluster, there is a ELB in public network that exposes the cluster to the outside world.
  • Docker in collaboration with AWS has created a cloudformation template to simplify the creation of Docker Datacenter infrastructure. Cloudformation template takes care of creation of the EC2 instances, deploying UCP, DTR and all associated system containers into the nodes. In addition, Cloudformation takes care of creation of ELB, VPC and all other supporting infrastructure.

Pre-requisites

Following are some pre-requisites to be done before we can deploy the voting application:

  • Get Docker datacenter 30 day trial license from here.
  • We need a registered DNS domainname to use Docker datacenter. UCP and DTR runs on sub-domains of the main registered domain name. In my case, i had “sreeniweb.net” domain already registered with Godaddy. I created a Route53 hosted zone in AWS and updated the nameservers in Godaddy to use AWS nameservers. The other option is to get domainname directly from AWS itself. The cost for domain name varies based on the provider.

Creation of Docker Datacenter infrastructure in AWS

Use the Cloudformation template to create the infrastructure. We need to input number and size of EC2 instances for UCP controller, DTR and application instances. This takes around 30-45 minutes. Following picture shows the user inputs that are needed:

docker_aws4

For my case, i used m3.medium for EC2 instances and cluster size as 3. “dtr.sreeniweb.net”for DTR FQDN,  “ucp.sreeniweb.net” for UCP FQDN,  “*.apps.sreeniweb.net” for application FQDN. We need to enter license in JSON format in the license field.

Following is the screenshot of successful creation along with the resource outputs:

docker_aws5

UCP can be accessed from “https://ucp.sreeniweb.net&#8221; and DTR can be accessed from “https://dtr.sreeniweb.net&#8221;. (These are not available currently as I have destroyed the cluster)

 

Following output shows the main UCP dashboard:

docker_aws6

Following output shows the different EC2 instances created. This can be seen from AWS EC2 console.

docker_aws9

In the above output, we can see 3 UCP controller instances, 3 DTR instances, 3 application node instances and 1 Jumphost. Jumphost is useful to ssh to other EC2 instances.

Following output shows the 3 classic ELBs that gets created. This can be seen from AWS EC2 console. The 3 ELBs are for UCP, DTR and application containers.

docker_aws8

Voting application deployment

Following is the voting compose file that I tried to deploy in Docker Datacenter:

client:
  image: 'smakam/myubuntu:v4'
  command: 'ping docker.com'
  links:
    - vote:vote
vote:
  image: 'instavote/vote:latest'
  ports:
    - '80:80'

Following output shows the application created:

docker_aws7

In the above output, we can see that there are 3 containers in this application. I have scaled the “vote” container to have 2 instances.

Voting application exposes port 80 on each node instance. Following output shows that port 80 access to the ELB connected to application nodes getting load balanced between the 2 containers.

$ curl DockerDat-UCPNodes-1XH06TRPK2SWQ-1768209315.us-westamazonaws.com  | grep -i "container id"
          Processed by container ID 50055eec5656
$ curl DockerDat-UCPNodes-1XH06TRPK2SWQ-1768209315.us-west-2.elb.amazonaws.com  | grep -i "container id"
           Processed by container ID 8c8190f2fe9b

ssh into the nodes

Since the EC2 instances are not directly exposed to the outside world, we need to first ssh into the jumphost and then ssh from jumphost into the individual instances. Jumphost is automatically created by Cloudformation template. ssh access helps in debugging.

Following output shows the Docker version in 1 of the application nodes:

$ docker --version
Docker version 1.11.2-cs4, build 7d4e626

Swarm mode is not yet available in Docker Datacenter as it is available only from version 1.12. It will be interesting to see how Swarm mode will be integrated to Docker Datacenter and if that will be kept as a default option instead of traditional Swarm mode.

Following output shows the running Containers in 1 of the application node:

$ docker ps
CONTAINER ID        IMAGE                    COMMAND                  CREATED             STATUS              PORTS                     NAMES
847dd233475b        smakam/myubuntu:v4       "ping docker.com"        40 minutes ago      Up 40 minutes                                 myvotingapp_client_1
50055eec5656        instavote/vote:latest    "gunicorn app:app -b "   40 minutes ago      Up 40 minutes       0.0.0.0:80->80/tcp        myvotingapp_vote_1
db588e83584a        docker/ucp-swarm:1.1.2   "/swarm join --discov"   3 hours ago         Up 3 hours          2375/tcp                  ucp-swarm-join
e03f9cad5763        docker/ucp-proxy:1.1.2   "/bin/run"               3 hours ago         Up 3 hours          0.0.0.0:12376->2376/tcp   ucp-proxy

In the above output, first 2 are application containers and next 2 are UCP system containers.

Accessing UCP through CLI

We can use the procedure here to get CLI access. This gives us access to the Swarm manager from where we can control the whole cluster. Following output shows that the complete cluster summary including the 9 nodes using the regular Docker command:

$ docker info
Containers: 61
 Running: 60
 Paused: 0
 Stopped: 1
Images: 128
Server Version: swarm/1.2.3
Role: primary
Strategy: spread
Filters: health, port, containerslots, dependency, affinity, constraint
Nodes: 9

Integrating UCP with DTR

This link walks through the steps to integrate UCP with DTR. This would allow UCP to securely push and pull images from DTR.

For folks who want to try Docker Datacenter in the trial period, there is no option to try it out without having a registered domain name. It will be good if there is a way to workaround this.

 

References

Docker Cloud for AWS

In this blog, I will cover Docker cloud usage with AWS. This blog is part of my Docker for AWS series and uses the sample voting application for illustration.

Docker cloud is a hosted service from Docker to manage Containers. Docker cloud is free to try for 1 private repository and node and is chargeable after that. Docker cloud was originally an acquisition from Tutum. Docker cloud can be used to manage infrastructure nodes in the cloud or nodes in the local data center. For basics on Docker cloud/Tutum, please refer to my earlier blog here. Since Docker cloud was an acquisition, it does not use some of the Docker ecosystem software. Following are some important differences:

  • Docker cloud has its own orchestration engine rather than using Swarm.
  • Docker cloud uses Stackfile rather than using compose file for describing the application. There is an attempt to make Stackfile as close to compose file as possible.
  • Docker cloud uses Weave as the underlying overlay network rather than Docker overlay using libnetwork.
  • Service Discovery is in-built and uses DNS based load balancing rather than Service discovery approach used in Docker 1.12.

Following are the steps to create the voting application in Docker cloud:

  • Create multi-node cluster in AWS. We need to register AWS credentials first before we can create the cluster.
  • Create stackfile and deploy the stackfile in the multi-node cluster. Stackfile defines the multi-container application. Docker cloud uses Stackfile and also creates endpoints for individual services.

Pre-requisites

Following are some pre-requisites before we can deploy the voting application in AWS:

  • Create Docker cloud account.
  • Register AWS credentials in Docker cloud.
  • Install Docker-cloud CLI. Docker cloud can be managed either using web interface or docker-cloud CLI.
  • To ssh into individual nodes, we need to deploy “authorized_keys” service on all nodes of the cluster. This runs as a Container. For more details on this, please refer here.

Application deployment

Create multi-node cluster in AWS
I created 2 node cluster using Docker cloud GUI. Following command shows details of the cluster:

$ docker-cloud nodecluster ls
NAME     UUID      REGION     TYPE      DEPLOYED     STATUS           CURRENT#NODES    TARGET#NODES
awstest  843c802d  us-west-2  t2.micro  14 days ago  Empty cluster                0               0
myaws    2f0533b9  us-west-2  t2.micro  3 hours ago  Deployed                     2               2

We are using cluster “myaws” for this example.

Following command shows details of the nodes in the cluster:

$ docker-cloud node ls
UUID      FQDN                                                    LASTSEEN        STATUS      CLUSTER    DOCKER_VER
60e1600a  60e1600a-b6f4-4bd1-859f-57f72128027f.node.dockerapp.io  21 minutes ago  ▶ Deployed  myaws      1.11.1-cs1
c4aba51d  c4aba51d-1581-4beb-8208-81f8ee5008c2.node.dockerapp.io  21 minutes ago  ▶ Deployed  myaws      1.11.1-cs1

Above, we can see the 2 nodes.

Let’s deploy the voting application as shown below:

$ docker-cloud stack up -f voting.yml 
2858f507-b735-4459-80ea-1cd3b799e14f

Following are the contents of “voting.yml” that shows the 2 services “client” and “vote”:

client:
  image: 'smakam/myubuntu:v4'
  command: 'ping docker.com'
  restart: always
vote:
  image: 'instavote/vote:latest'
  deployment_strategy: high_availability
  ports:
    - '8080:80'
  restart: always
  target_num_containers: 2

Following are some important specifics on the YML file above:

  • “restart” option allows for container restart in case it dies.
  • “target_num_containers” describes number of containers for a particular service. Default value is 1. In this example, we have used 2 instances of “vote” container.
  • “high_availability” strategy spreads the multiple instances of containers across nodes.

Following command lists the services deployed:

$ docker-cloud service ps
NAME            UUID      STATUS           #CONTAINERS  IMAGE                              DEPLOYED        PUBLIC DNS                                               STACK
authorizedkeys  a5424263  ◼ Not running              0  dockercloud/authorizedkeys:latest  2 hours ago     authorizedkeys.authorizedkeys.a5424263.svc.dockerapp.io  authorizedkeys
vote            bb5b21bc  ▶ Running                  2  instavote/vote:latest              27 minutes ago  vote.dockercloudstack.bb5b21bc.svc.dockerapp.io          dockercloudstack
client          d0f342fa  ▶ Running                  1  smakam/myubuntu:v4                 26 minutes ago  client.dockercloudstack.d0f342fa.svc.dockerapp.io        dockercloudstack

Above, we can see that the “client” and “vote” service is running.

Following command shows the Docker version running in the node:

# docker --version
Docker version 1.11.1-cs1, build bfd1f99

Docker cloud deploys custom AMI with Docker installed as well as the necessary system services.

Following command shows the containers running in node1:

# docker ps
CONTAINER ID        IMAGE                                   COMMAND                  CREATED             STATUS              PORTS                  NAMES
1b70e324c8d7        smakam/myubuntu:v4                      "ping docker.com"        30 minutes ago      Up 30 minutes                              client-1.dockercloudstack.825b9f18
9926c0c82868        instavote/vote:latest                   "gunicorn app:app -b "   30 minutes ago      Up 30 minutes       0.0.0.0:8080->80/tcp   vote-2.dockercloudstack.c50f2bcc
e61c7cfcf7a5        dockercloud/cleanup:latest              "/run.sh"                3 hours ago         Up 3 hours                                 cleanup-59378.47087957
2b7f0c29ded8        dockercloud/logrotate:latest            "crond -f"               3 hours ago         Up 3 hours                                 logrotate-89401.514eac74
0f9bd75b318c        dockercloud/events:latest               "/events"                3 hours ago         Up 3 hours                                 events-70700.de5bba90
ed4e763aec8e        dockercloud/ntpd:latest                 "/run.sh"                3 hours ago         Up 3 hours                                 ntpd-75129.9cf0bf86
78c3b4bebab1        weaveworks/plugin:1.5.2                 "/home/weave/plugin"     3 hours ago         Up 3 hours                                 weaveplugin
64c3778e58eb        weaveworks/weave:1.5.2                  "/home/weave/weaver -"   3 hours ago         Up 3 hours                                 weave
734129420b6f        dockercloud/network-daemon:1.11.1-cs1   "/run.sh"                3 hours ago         Up 3 hours                                 weave-49562.4bf4a770

Except the first 2 containers above, remaining containers are Docker cloud system containers that helps with orchestration, networking, logging etc. Only 1 instance of “vote” service is running in node1, another instance runs in node2, Docker cloud takes care of orchestration and health check of containers in the service.

Lets get the service endpoint for the voting application:

$ docker-cloud service inspect vote | grep -i end
      "endpoint_uri": "http://vote.dockercloudstack.bb5b21bc.svc.dockerapp.io:8080/"

Following output shows that access to the endpoint gets load balanced between the 2 “vote” containers:

$ curl http://vote.dockercloudstack.bb5b21bc.svc.dockerapp.io:8080/ | grep -i "container id"
           Processed by container ID vote-2
sreeni@ubuntu:~/dockercloudstack$ curl http://vote.dockercloudstack.bb5b21bc.svc.dockerapp.io:8080/ | grep -i "container id"
          Processed by container ID vote-1

We can also access the “vote” service from “client” service using hostname since they are part of the same stack.

References

 

Docker for AWS beta

In this blog, I will cover “Docker for AWS” beta service launched by Docker. This blog is part of my Docker for AWS series and uses the sample voting application for illustration.

As part of Docker 1.12 announcement, Docker released AWS Docker integration as beta software. With this software, Docker is trying to  simplify AWS integration by better integrating Docker with AWS services like load balancer, security groups, logs etc. Docker launched similar integration service with Microsoft Azure as well. Docker 1.12 RC4 is available as part of this integration, so Swarm mode feature can be used.
Following are some features that Docker has added as part of this integration with AWS:

  • EC2 instances + Autoscaling groups – Instances and Autoscaling groups are created based on the number of manager and worker nodes specified by the user.
  • IAM profiles – This allows for modification of different services provided by AWS based on user identity.
  • DynamoDB Tables – It is not clear how exactly this is used. Looks like this is an database used for internal communication.
  • SQS Queue – This facilitates upgrade handling as mentioned here.
  • VPC + subnets – This allows for network management in an isolated domain.
  • ELB – For services that expose external ports, automatic integration is done with AWS ELB.

Following are the summary of steps:

  • Register for AWS beta.
  • Start Docker cluster using cloudformation template.
  • Deploy application.

Registering for Docker AWS beta
New users can register here with AWS account details. Docker team will reply back with registration details. The link provided as part of successful beta invite takes to a cloudformation template that allows for automatic creation of Docker Swarm mode cluster with associated AWS services.

Following is the GUI snapshot of Cloudformation template:

docker_aws11.PNG

Following are some details that needs to be entered:

  • Number of manager and worker nodes.
  • Instance type for manager and worker nodes.
  • Security key to ssh.

At this point, there is no option to select Docker version and it comes installed with Docker 1.12 RC4 by default.

Cloudformation gives ELB DNS name and option to ssh to master node as 2 outputs. Following is the output for my AWS Docker cluster.

docker_aws1

The way Docker creates security groups, ssh access is allowed only to master node and ssh access is prevented to worker nodes. This is on purpose to provide better security and was discussed here. To debug issues, suggestion provided is to use cloudwatch logs or by looking at Container logs from master node. Container logs of worker nodes can be monitored from master using:

docker -H <hostname> logs  <containerid>

I selected 1 manager node and 2 worker nodes during the initial template input process. Following is the Swarm mode cluster that got created.

$ docker node ls
ID                           HOSTNAME                                      MEMBERSHIP  STATUS  AVAILABILITY  MANAGER STATUS
4obcqdkd1zu3g2eb3lzs0iw0q    ip-192-168-34-65.us-west-2.compute.internal   Accepted    Ready   Active        
b9t9zpj4ydc67wr4uhh1z0kz3 *  ip-192-168-33-175.us-west-2.compute.internal  Accepted    Ready   Active        Leader
dtbrn7e9orq8u72okwg9tmnkn    ip-192-168-33-7.us-west-2.compute.internal    Accepted    Ready   Active        

To do better integration between Docker and AWS, Docker creates some system containers in each node. Following output shows system containers running in master node:

$ docker ps
CONTAINER ID        IMAGE                                       COMMAND                  CREATED             STATUS              PORTS                NAMES
abf8951cd712        docker4x/controller:aws-v1.12.0-rc4-beta2   "controller run --log"   7 hours ago         Up 7 hours          8080/tcp             editions_controller
8e72ace27e55        docker4x/shell-aws:aws-v1.12.0-rc4-beta2    "/entry.sh /usr/sbin/"   7 hours ago         Up 7 hours          0.0.0.0:22->22/tcp   nauseous_wozniak
fdaf4d4f61d0        docker4x/guide-aws:aws-v1.12.0-rc4-beta2    "/entry.sh"              7 hours ago         Up 7 hours                               infallible_jones

Controller container is responsible for talking to Docker engine and AWS backend services. For example, when ports get externally exposed by Docker services, controller container takes care of automatically updating AWS ELB to load balance between the EC2 instances. Since routing mesh exposes the port in each Docker node, it takes care of automatically load balancing between the individual containers associated with that service.

Deploying application

Following set of commands deploys the “client” and “vote” service in the overlay network “overlay1”

sudo docker network create --driver overlay overlay1
sudo docker service create --replicas 1 --name client --network overlay1 smakam/myubuntu:v4 ping docker.com
sudo docker service create --name vote --network overlay1 --replicas 2 -p 8080:80 instavote/vote

Following output shows the running services:

$ docker service ls
ID            NAME    REPLICAS  IMAGE               COMMAND
87qk6yt5za8c  client  1/1       smakam/myubuntu:v4  ping docker.com
cj4iznzzjtex  vote    2/2       instavote/vote     

Following output shows the request sent to ELB domain name and getting load balanced between the “vote” containers associated with “vote” service.

$ curl Docker-ELB-1271640601.us-west-2.elb.amazonaws.com:8080 | grep -i "container id"
          Processed by container ID b8a467211555
sreeni@ubuntu:~$ curl Docker-ELB-1271640601.us-west-2.elb.amazonaws.com:8080 | grep -i "container id"
          Processed by container ID 7d34e8032693

We can access the “vote” service from “client” service as well. By default VIP based load balancing would be done in that case.

References