I have always loved Hashicorp’s Devops and cloud tools. I have used Vagrant, Consul, Terraform, Packer and Atlas before and I have written about few of them in my previous blogs. Vault is Hashicorp’s tool to manage secrets securely in a central location. Secret could be database credentials, AWS access keys, Consul api key, ssh private keys etc. It is necessary for secrets to be managed centrally and having strict control and audit policies. By having a separate tool to manage secrets, application developer don’t need to worry about security internals and leave it to Vault to manage secrets. In this blog, I will cover Vault overview and internals and in the next blog, I will cover some use cases that I tried out.
Vault uses the following principles:
Vault can provide keys based on limited ttl and use count. Vault also allows for key renewals and rotation. These approaches keeps keys from staying permanent so that chances of malicious users using the keys is reduced. If Vault provided a secret with ttl duration of 1 hour, Vault will take care of revoking the secret after 1 hour.
Vault provides TLS for secure access and wrapping using cubbyhole to distribute keys securely. This prevents man in the middle attacks.
Vault’s default policy is to block all access and policy needs to be applied to selectively add access.
All access to Vault can be audited. This can assist with detection of any malicious activities. If a particular application or user or machine has accessed the secret, Vault will have the audit records for that access.
Break glass procedure
Vault provides facilities like locking out all access and key rotation in case the system gets compromised. The actual break glass procedure to be followed would be based on specific enterprise policy. This is very critical as enterprises needs to know what exactly has been compromised and to be sure that the system has been locked down after the compromise.
Following are some internals on Vault:
All secrets including keys, certificates, passwords are stored with encryption in central location.Vault’s initial unseal process is done manually using multiple unseal keys. This is done to prevent key compromise from a single source. Every other operation in Vault can be automated except the unseal process.
Access to Vault is controlled using policies. Policies can be written in HCL(Hashicorp configuration language). HCL is compatible with JSON, so policies can be written in JSON. Vault root token has a root policy that provides access to all secrets in Vault. Access to secrets stored in Vault can be restricted with ACL policy by using authentication.
Authentication is necessary to tell Vault who we are. The authentication can either be user based or machine based. Vault has a common scheme for handling authentication and by using authentication backends, it keeps the frontend for authentication the same and the backend takes care of the specifics.
For user based authentication scenario, Vault provides username/password, token, github methods to authenticate. For machine scenario, Vault provides methods like approle, certificates to authenticate.
After successful authentication, Vault assigns a authorization policy which dictates what a particular user/machine can do with Vault. Based on the policy returned, user/machine can access secret backend to fetch the secrets.
Secret backend and Dynamic secrets:
Vault’s secret backend takes care of secret generation. Vault has a common scheme for generating secrets for different types of needs like AWS access, database access like mysql etc. The secrets are generated based on the type of backend. For example, with generic backend, secrets are manually generated by user. For AWS backend, dynamic secrets are generated by Vault using AWS IAM module.
Dynamic secrets are used with backends like AWS, mysql, cassandra, postgres. For dynamic secrets, it is necessary to feed in the backend root secret into vault. For example, AWS root access key should be registered to Vault for AWS backend. The advantage with dynamic secrets is that these secrets have a lease period which expire. Also, since unique dynamic secrets are generated per application, it is easier to audit and do breakglass procedure with dynamic secrets.
Vault’s audit backend takes care of logging all access and events in Vault. Vault supports file and syslog backends. Vault can be accessed externally using HTTP API with or without TLS. Vault can also be accessed programmatically using language libraries supported by Hashicorp.
Vault and Nomad Integration:
In a typical Container orchestration solution, orchestrator takes care of secret management and feeding the secrets to the container. Nomad is Hashicorp’s orchestration solution. In the current scheme, Vault token is specified as a environment variable in the Nomad job. This is not secure. Vault to Nomad integration is currently being developed and will be available soon. Following link walks through the details of Vault and Nomad integration.
Following is a typical workflow for Vault administrator and user/machine to manage secrets:
- Start vault and unseal it.
- Add credentials for backend based on the type of backend(aws, mysql). For example, with AWS, we need to add AWS secret root keys to Vault.
- Create policy to control access to backend based on user or machine identifier.
- Either user or machine logs into Vault based on credentials they have.
- Vault associates a policy based on user/machine credentials with authentication backend.
- User or application fetches necessary secrets from secret backend based on available policy.
- For dynamic secrets, the invoked application revokes the secret or Vault destroys the secrets after lease duration.
In the next blog, I will cover some use cases that I tried with Vault.