Managing Unseal Keys for HashiCorp Vault

After finding a need for a new secrets management platform at CoverMyMeds and evaluating several tools, we decided on Vault by HashiCorp. Vault boasts an impressive number of secret and authentication “backends” which give it impressive flexibility for storing and generating secrets, as well as dynamically generating credentials. It also integrates well with Consul service discovery and is able to use Consul’s key/value store as a storage backend. These features, along with its easy deployment in a containerized environment, heavily influenced our decision and we continue to find new use cases and features as roll-out progresses. You can check out more information about Vault at vaultproject.io.

One challenge we encountered during our initial testing was what to do with Vault’s unseal keys. When Vault is initialized, an encryption key is generated and then split into five key shares which are output to the console. In order to use Vault after the server is started, three of these five keys must be presented to Vault in order to unseal it. This functionality is dual-purpose; it serves as an extra layer of security by preventing an unauthorized instance of Vault being attached to the cluster and is also the encryption key itself which must be presented to each Vault instance at runtime. This method of keysharing is known as Shamir’s Secret Sharing.

The process of unsealing posed an interesting problem: what do we do with these keys and how do we get them from where they are stored to the Vault instance at runtime? It is tempting to store these keys on the host’s filesystem and write a script to feed them to Vault when it is restarted. Not only does this defeat the purpose of splitting the keys into shares, but makes an inviting target in the event of a security breach. Storing them in a password repository also seemed less than ideal as well. If they are all in one place and that place is compromised the proverbial “keys to the kingdom” are exposed. These concerns led to an interesting solution.

It quickly became clear that we did not want the keys all in one place nor in the charge of one person. The first decision we made was that we wanted to delegate five people to hold one of these keys each. When creating a new Vault instance we would reach out to three of these people to send their keyshare to Vault. This immediately led to some questions. “Where can I store this key? Is Lastpass ok? A GPG-encrypted file on my laptop?” We wanted to standardize this process and make it as simple, secure and automated as possible. One thing we all had was our company Macbook, and it has built in functionality for storing secrets in the keychain. We are also heavy users of Ansible at CoverMyMeds, so we decided to leverage that to automate the process.

The keychain can be accessed via the terminal with a not-so-obviously named utility called “security”. Using Ansible’s shell and uri modules, we created playbooks that would allow the person who created the new Vault instance to take the keys that were generated and distribute them to the home directory of the five key-holders on a secure server. Those key-holders can then run a second playbook that will read the key from where it was left in their home directory, store that value with a descriptive name in the keychain, read the key back from the keychain and compare it to what was distributed to verify the key had been stored successfully, and then remove the key from their home directory. A third playbook can be run by the key-holder that uses the uri module to send the stored key to the Vault unseal API endpoint. All of these playbooks are contained in one repository and called with a Makefile to simplify the process.

We have found this process to be pretty painless. It also keeps the unseal keys safe, decentralized, and encrypted until they are needed. When a new Vault node is restarted or created the key-holders can simply run one command from their laptop and their key-share is transmitted. We hope this is useful information for those looking to deploy Vault in their environment, and we have published the framework for this process at github.com/covermymeds/vault_unseal. Feel free to comment here or on our GitHub pages with comments and suggestions. We look forward to your feedback.