Infrastructure as Code Doesn't Mean You Should Stop Understanding Infrastructure

Managing your infrastructure using a tool such as Ansible, Chef, or Terraform makes your life considerably easier; it gives you repeatability, it gives you version control and it lets you review your infrastructure changes in the same way you'd review application code. But most importantly, it means you don't have to spend your afternoon manually logging into twelve servers to make the exact same change twelve times, it makes your changes repeatable.

The downside is, it comes with a potential pitfall: the more infrastructure you automate, the easier it becomes to forget what the automation is actually doing.

Infrastructure as Code is an Abstraction

IaC means you don't have to configure your servers manually, but it doesn't mean that you no longer can. At the end of the day, most IaC is an abstraction over operations that could otherwise be performed manually. Let's take the following playbook:

- name: Install nginx
  ansible.builtin.apt:
    name: nginx
    state: present

That's an amazingly simple way of installing nginx onto a server, but it doesn't do anything overly fancy. There's no magic building of packages going on behind the scenes (at least for this example), there's no sacrificing to the great IT gods, it simply abstracts one simple command away from you: apt install nginx. There is one thing Ansible does provide in this instance, which is the ability to use ansible.builtin.package rather than ansible.builtin.apt, this makes your playbook potentially usable across multiple distributions; CentOS, Ubuntu, Debian, FreeBSD, your playbook could work on all of them with few (if any) changes (with the large caveat that the package name must match across all distros that you target).

The same rule above applies regardless of which tool you use for the most part; Chef, for example, has the same capabilities. It can also manage your packages, files, services, and everything in-between, every operation in a cookbook or a playbook can be translated directly into one or more commands or operations that can be carried out manually, just as Terraform resources eventually translate into API calls against a cloud provider or your hypervisor, or even your servers, and Kubernetes manifests just result in the creation of containers using images running on a container runtime.

The abstraction of these processes is useful precisely because you don't want to think about those details every time you do something, you want to be able to do a task repeatedly, and consistently. Imagine applying configuration across 50 servers, only for one of those servers to have a typo in a hostname, or a configuration variable, that results in a 1 in 50 chance of a request failing.

The Happy Path Is Very Good at Hiding Complexity

One of the nicest things about good infrastructure automation is also one of its biggest dangers; when everything works, infrastructure becomes boring.

• You commit a change
• CI runs
• Terraform produces a sensible plan
• You apply it
• Your configuration management runs
• Everything goes green
• You're done

After you've made changes enough times, it's very easy to start seeing the abstraction itself as the infrastructure, you see your cookbooks as the thing that runs your service, not the servers that actually handle requests and provide the resources. This is where understanding the layer underneath your automation becomes important. Your IaC tooling can tell you whether the infrastructure matches the state you've described, but it can't necessarily tell you whether the state you've described is actually correct. If there's a typo in the configuration in your Chef cookbook, that typo will make it to every server that runs the cookbook, but as far as Chef is concerned, unless a service failed to start, everything is as expected.

Eventually, You're Going to SSH Into the Box

We've spent years trying to eliminate the need to SSH into production servers, and for good reason. Nobody wants to be manually changing production configuration and creating a drift between the state of 2 separate servers. You also reach a problem where those configuration changes aren't actually documented. Your documentation says that server Y should be sending database traffic to 10.0.0.1, but that got changed somewhere down the line, it's sending its traffic to 10.0.0.2 now, and nobody knows why, and somebody will inevitably see that difference between the current state and the documented state, change it, and walk away... Inevitably resulting in saying goodbye to your production environment until somebody realises that 10.0.0.1 hasn't been used in 5 years. That problem gets compounded when changes get made at 2am to fix an incident, before an exhausted on call engineer goes back to bed and says "I'll document that change in the morning" (and we've all been in the situation where that task falls by the wayside because hundreds of other things come along the next morning).

Despite these problems with manually managing your infrastructure, none of this is an argument against knowing how to actually administer your estate. When something goes wrong, and some day it will, being able to actually investigate on a specific server is invaluable. You might start with:

systemctl status nginx
journalctl -u nginx

Maybe nginx is fine, so you check whether it's actually listening with ss -lntp. It is, but can you reach it locally? curl -v http://localhost responds with a 200, but can it be hit from another server on the same VLAN? Aha, curl failed on the second box! So you begin to dig into the network; firewall rules, routing, unexpected IPv6, DNS? (It's always DNS, right?)

You get the point, executing commands, checking configuration directly, all of this is extremely key to actually being able to investigate problems that might occur at some point down the line. To be able to debug your infrastructure you need to understand your infrastructure, first.

Learn to Go Down a Layer

It's a useful engineering principle in general, and it isn't necessarily tied to IaC, but when an abstraction stops behaving the way you'd expect it to, then go down a layer.

If Terraform is behaving strangely, understand the API calls it's ultimately making, and maybe even try to replicate those calls yourself.

If Chef fails to apply a change, understand the command(s) it's running, and try running them yourself, and see what changes need to be made in order to fix the issue.

Abstractions are incredibly useful while they're working, but once they fail, you need to understand what's going on underneath the hood to figure out where to go next, otherwise you'll be reduced to changing configuration and code until something works, and never understanding why that change fixed your problem... That's not debugging, that's just gambling with your infrastructure.

Build It Manually at Least Once

By build it manually at least once, I don't mean build your whole infrastructure, I mean prototype part of your infrastructure first. Let's say you want to set a new nginx server up - jumping straight to IaC might sound enticing, but sometimes the best solution is to build that nginx server first, manually installing the packages you need, configuring repositories, creating users, setting file permissions, writing the config the first time round and checking that everything works, iterating that configuration until you're in a state that you're happy with. Does it perform? Does it work securely? Does it even serve the right resources without throwing 404s out of nowhere?

Once you manage to build a working service manually, you can migrate that into your preferred IaC tool, and you can apply that across your estate (and ditch the prototype, don't let your prototype run in production, that's another surefire way to get config drift!). When you understand the infrastructure itself, you can quickly write the definitions around it, and you benefit from being able to fix it a year down the line when everything goes from running smoothly to failing to function, whether that's from a botched update, or just a package incompatibility.

Understanding Infrastructure Makes Your IaC Better

Understanding the underlying infrastructure isn't only useful when things go wrong; it also makes the automation you write better in the first place.

If you understand systemd, you'll write better service definitions. If you understand Linux permissions, you're less likely to solve a permissions problem by throwing chmod 777 at it. If you understand networking, you'll design better firewall rules and have a better idea of which services actually need to communicate with each other. If you understand DNS and TLS, you'll be much better prepared to configure them correctly rather than repeatedly changing things until the errors disappear.

The same applies at a larger scale. Understanding how the individual components of your infrastructure interact makes it much easier to decide where your abstractions should sit, what should be configurable, what should be shared, and what should remain specific to an individual service.

You don't need to memorise every command or become an expert in every subsystem underneath your automation. You just need enough understanding that the automation you're writing is describing infrastructure you actually understand, rather than infrastructure you only know how to create through a particular tool.

Automation Should Remove Repetition, Not Understanding

Going back to what I said before, your IaC is there to remove repetition, and introduce consistency across your environment. web-01 to web-50 will all have the exact same nginx configuration, with the exact same certificates, and the exact same folder structures.

This also means that you can version your whole infrastructure. Changed the nginx configuration and nothing works anymore? No need to panic, just revert the latest commit and re-run your tooling, you'll be back to a working state in no time at all, compared to manually correcting configuration across all of your boxes. That's a matter of minutes vs hours for getting production environments back up and running, and that difference matters massively when trading in massive amounts of money and traffic. Every minute your site is down is an impact on your reputation and your bottom line.

It also makes your on-call support much easier when it comes to diagnosing issues with infrastructure. An engineer at 2am might understand the infrastructure inside and out, but being able to diff the changes made earlier that day makes their job of diagnosing issues much easier and changes the timeframe for them being able to restore the service.

In conclusion, abstraction doesn't remove the underlying system, it just gives us a better interface for controlling it. And when that interface stops giving us the answers we need, we still need to understand what's going on underneath. Infrastructure as Code should mean you don't have to configure servers manually, not that you no longer can.