America in the 1920s

Bill Bryson (A Short History of Almost Everything) knows how to write rich and engaging history books (he’s a remarkable storyteller). One Summer is a spectacular book for understanding America in the mad 1920s (and all the craziness around Charles Lindbergh, including early aviation history). A must read.

Using Ansible Vault with environment variables

This is a common trend. You’ve been using Ansible to provision your infrastructure for some time and all of a sudden you will have a couple of secrets to manage, usually SSL/SSH private keys, API credentials, passwords, etc. Because you don’t want these secrets to be stored “in the clear” on your git repository, you will declare them as variables inside yaml files and then use Ansible Vault to encrypt them using an AES symmetric key. You can then run ansible-playbook with –ask-vault-pass, so yaml var files will get decrypted on the fly when running the playbook.

Sometimes I use Ansible together with other tools under the same repository. For example, I prefer to provision AWS infrastructure with Terraform and then call Ansible as a provisioner to customize an EC2 instance and Cloudflare to update the DNS record . Or use Packer to bake an AMI and use Ansible as a local provisioner. In this case, it is common practice to use environment variables for secrets in use by Terraform providers (AWS_ACCESS_KEY_ID, AWS_SECRET_ACCESS_KEY, etc) instead of storing them directly in .tf files that are then pushed to github. What if I also want to have an easy way to encrypt these ENV variables on a file? Why not use Ansible Vault with Terraform? Just discovered that Ansible Vault encrypts any kind of text file, not only yaml type. We just have to create a secrets.txt file with one ENV variable per line:

[email protected]

And then encrypt secrets.txt with Ansible Vault:

ansible-vault encrypt secrets.txt

If all goes well, secrets.txt should now begin with “$ANSIBLE_VAULT;1.1;AES256” followed by the encrypted text. Your secrets.txt can be safely added to the repository. Now, before you run terraform plan, you can easily export your secrets as ENV variables by doing:

for i in `ansible-vault view secrets.txt` ; do export $i ; done

Ansible Vault will ask you for the password so it can decrypt secrets.txt and will output the contents so we can use it with export. If you’re using Makefile, a “make export-secrets” job can make it even more easier. This is just a quick way to store credentials in use by Terraform or other tools in a file encrypted using a shared secret. If you have a big infrastructure team working under the same repository and you don’t want to use Ansible, there are tools like StackExchange’s Blackbox that allow you to easily encrypt files using GPG, making use of a team keyring. Also, I assume there will be a human executing Terraform. If you’re running Ansible/Terraform within a CI/CD environment, there are better ways to use credentials (by using job or role assigned tokens and something like Hashicorp Vault). To be explored.

Upstart and resolvconf cache

I’ve recently found this when I was trying to fix a nameserver config issue with resolvconf on Ubuntu. When resolvconf populates /etc/resolv.conf, it will read what we have configured in /etc/resolvconf/resolv.conf.d (head, base, tail, etc) and also any dns-server declared in /etc/network/interfaces. I had a conflict with something I was populating in the head file (with Puppet) from something that was configured under /etc/network/interfaces. So I removed the conflicting dns-server declaration from the interfaces file and run “resolvconf -u” to update the config. To my surprise, the “deleted” nameservers from /etc/network/interfaces were still included in /etc/resolv.conf. After some debugging, I have noticed that resolvconf’s Upstart script now keeps a cache file under /run/resolvconf/interface that is a copy of the previous /etc/network/interfaces. You need to delete this file and restart resolvconf to make it work: “stop resolvconf ; start resolvconf”.