"Licensed under CC 2.0" What would happen if you, unexpectedly, had to build your entire production infrastructure from scratch? Would you be able to perform a full recover off all services and dependencies to an acceptable level? How long would it take? Hours? Days? Would the engineering team know what to do? What problems would … Continue reading Disaster Recovery with Terraform, AWS and a few lessons learned
Monitoring Docker Thin Pool Usage with Prometheus
Update (12/4/20): I highly recommend using the latest Amazon ECS-optimized Amazon Linux 2 AMI. It uses Docker's OverlayFS (overlay2) storage driver. The same partition is used for OS, Docker images and metadata. It's easier to monitor filesystem usage using the Prometheus Node exporter. If you still have to use the older ECS AMI v2015.09.d or … Continue reading Monitoring Docker Thin Pool Usage with Prometheus
Using EC2 Spot Instances with ECS
Had the opportunity to write an article for AWS Startups Blog, explaining how we use EC2 Spot Instances with ECS at Signal: "Every day, Signal ingests millions of documents from a growing number of publishers, including online media, print newspapers, broadcast, regulation and legislation. Our text analytics pipeline processes these documents in real time, applying … Continue reading Using EC2 Spot Instances with ECS
America in the 1920s
Bill Bryson (A Short History of Almost Everything) knows how to write rich and engaging history books (he's a remarkable storyteller). One Summer is a spectacular book for understanding America in the mad 1920s (and all the craziness around Charles Lindbergh, including early aviation history). A must read.
Using Ansible Vault with environment variables
This is a common trend. You've been using Ansible to provision your infrastructure for some time and all of a sudden you will have a couple of secrets to manage, usually SSL/SSH private keys, API credentials, passwords, etc. Because you don't want these secrets to be stored "in the clear" on your git repository, you will declare them … Continue reading Using Ansible Vault with environment variables
Upstart and resolvconf cache
I've recently found this when I was trying to fix a nameserver config issue with resolvconf on Ubuntu. When resolvconf populates /etc/resolv.conf, it will read what we have configured in /etc/resolvconf/resolv.conf.d (head, base, tail, etc) and also any dns-server declared in /etc/network/interfaces. I had a conflict with something I was populating in the head file (with … Continue reading Upstart and resolvconf cache
marques.cx -> fmarques.org
Not long ago I decided to move to a new personal domain and registered fmarques.org. I am in the process of moving everything from marques.cx to my new domain, which will cease to exist in a few months. If you are one of the brave souls still keeping an eye on my feed, I advise to change … Continue reading marques.cx -> fmarques.org
Discovering jemalloc and debugging native Java memory leaks
I've joined ThoughtWorks last August (awesome!) and I've been working with the tech team on everything related to infrastructure automation, code deployment and all things "DevOps" for GOV.UK Verify (part of the Government Digital Services). The last few months were very rewarding to me as I got exposed to a lot of different technologies, although I do … Continue reading Discovering jemalloc and debugging native Java memory leaks
Flash Storage Failure Rates
Real data is so much better than speculation A very interesting read on Flash Storage Failure Rates in a very large population (Facebook data).
Easy way to to stop kernel updates on Ubuntu
One easy command (as root): echo `dpkg -l "*$(uname -r)*" | grep image | awk '{print $2}'` hold | dpkg --set-selections
