Disaster Recovery with Terraform, AWS and a few lessons learned

"Licensed under CC 2.0" What would happen if you, unexpectedly, had to build your entire production infrastructure from scratch? Would you be able to perform a full recover off all services and dependencies to an acceptable level? How long would it take? Hours? Days? Would the engineering team know what to do? What problems would … Continue reading Disaster Recovery with Terraform, AWS and a few lessons learned →

Monitoring Docker Thin Pool Usage with Prometheus

Update (12/4/20): I highly recommend using the latest Amazon ECS-optimized Amazon Linux 2 AMI. It uses Docker's OverlayFS (overlay2) storage driver. The same partition is used for OS, Docker images  and metadata. It's easier to monitor filesystem usage using the Prometheus Node exporter. If you still have to use the older ECS AMI v2015.09.d or … Continue reading Monitoring Docker Thin Pool Usage with Prometheus →

Using EC2 Spot Instances with ECS

Had the opportunity to write an article for AWS Startups Blog, explaining how we use EC2 Spot Instances with ECS at Signal: "Every day, Signal ingests millions of documents from a growing number of publishers, including online media, print newspapers, broadcast, regulation and legislation. Our text analytics pipeline processes these documents in real time, applying … Continue reading Using EC2 Spot Instances with ECS →

America in the 1920s

Bill Bryson (A Short History of Almost Everything) knows how to write rich and engaging history books (he's a remarkable storyteller). One Summer is a spectacular book for understanding America in the mad 1920s (and all the craziness around Charles Lindbergh, including early aviation history). A must read.

Using Ansible Vault with environment variables

This is a common trend. You've been using Ansible to provision your infrastructure for some time and all of a sudden you will have a couple of secrets to manage, usually SSL/SSH private keys, API credentials, passwords, etc. Because you don't want these secrets to be stored "in the clear" on your git repository, you will declare them … Continue reading Using Ansible Vault with environment variables →

Upstart and resolvconf cache

I've recently found this when I was trying to fix a nameserver config issue with resolvconf on Ubuntu. When resolvconf populates /etc/resolv.conf, it will read what we have configured in /etc/resolvconf/resolv.conf.d (head, base, tail, etc) and also any dns-server declared in /etc/network/interfaces. I had a conflict with something I was populating in the head file (with … Continue reading Upstart and resolvconf cache →

Discovering jemalloc and debugging native Java memory leaks

I've joined ThoughtWorks last August (awesome!) and I've been working with the tech team on everything related to infrastructure automation, code deployment and all things "DevOps" for GOV.UK Verify (part of the Government Digital Services). The last few months were very rewarding to me as I got exposed to a lot of different technologies, although I do … Continue reading Discovering jemalloc and debugging native Java memory leaks →