Invisible Infrastructure

After our random power outage a few weeks ago, I had a thought:

The more reliable something is, the less you notice it.

Take the power, for example. You probably don’t actively think about the power. I know I don’t. And that’s why I was caught off guard when the power went out. The reliability of stable underground power lulled me into a lack of awareness. I wasn’t prepared for being without power because it wasn’t top of mind. But what about if you live in a more unstable area where the power does go out with some regularity? Then you would think about it.

You’d be less surprised and more prepared.

The same thing happened when there was a GitHub outage. GitHub Actions, specifically, were down most of the day. We use them to automatically run tests and deploy our code to production. And we couldn’t do that because GitHub was experiencing issues.

It’s moments like those where dependencies become painfully obvious. Increased reliability lowered awareness until the outage revealed the infrastructure dependency.

Invisible infrastructure becomes visible.

The same thing happens with outages in software. The infrastructure you rely on is immediately brought into question. So then you’re left wondering what happened, why, and how you can prevent the next outage.

Reliability matters. If you want people to trust your product, it needs to work well.

Look for the infrastructure and web of dependencies supporting your product:

• The software packages in the code.
• The cloud-hosted services.
• The team’s knowledge.

Infrastructure isn’t only in the software. It’s also the system and the people working on it.

That’s why site reliability is the perfect way to understand your systems. You get a first-hand look at the failure points. You see what breaks, who knows how to fix it, and how the team responds to make it better each time. It’s one of my favorite ways to assess the product, the team, and the system driving them both.

Don’t get me wrong. I don’t want downtime. It is stressful and zaps an entire day when it happens. But it’s also how you build resilience into your team, product, and process.

Infrastructure Types

Infrastructure can be physical, like the power example. It can also be digital, like GitHub and any other software. But it can also be human, like the tacit knowledge of your team.

Physical + Digital + Human

To make infrastructure visible, find the:

1. Visible Structures: The observable arrangement of the system.
2. Invisible Conditions: The conditions allowing the structure to function.

If you focus only on the Visible Structures, then you miss the Invisible Conditions that make a system work. It’s like an electrical current. It generates power and flows through the system. Without it, a system stagnates and gravitates toward entropy. You have to look deeper. You can’t simply list who does what and call it a system. There’s always underlying conditions you can’t easily see. Someone is often covering deficiencies through their own skills and intuition. It’s unseen, but felt.

Resilience Loop

Now let’s get back to the site outage.

The product is down or performance is degraded enough to trigger a response.

Does your team know about it?

How do they handle it?

How do they improve the system?

I call this the Resilience Loop. And, running a lot of software teams, I’ve seen all flavors of it in practice. But the best teams:

1. Know about issues before anyone else.
2. Methodically address issues while communicating what’s happening.
3. Find the root cause and harden the system and response process each time.

It’s a loop. Every outage is an opportunity to learn, to iterate, to improve. Done enough times, reliability grows and the underlying conditions reveal themselves. Dependencies surface so you can see them and address them.

The Resilience Loop Formula:

Resilience = Prediction × Process × Prevention

Prediction is what happens before an outage.

Process is what happens during an outage.

Prevention is what happens after an outage.

Prediction

Uptime monitoring continuously monitors your product and generates alerts when something happens. It can be slow response times or errors. You let computers do the monitoring so you don’t have to manually check it yourself. Manual testing is still critical, though.

Ideally, uptime monitoring identifies trends that can lead to an issue. You want to catch the problems before they get bad. Most modern software services provide auto-scaling when volume exceeds capacity. And they’re integrated with several external systems, allowing you to funnel alerts to email, Slack, Teams, Linear, or even as text messages.

The best way to do this is evaluate the metrics leading up to, during, and after the incident. If you know what to look for, you can add checks at those specific failure points.

Process

Even with extreme reliability and every redundancy in place, 100% uptime is impractical. Software will go down. So you need to make sure you have a reliable process for handling outages when they occur.

You need to know:

1. Who you notify to handle the incident.
2. How you communicate to customers.
3. Where you coordinate the work.

One is the on-call system. You need a group of engineers who know the back-end systems.

Two is the communication system. You need a status page and a way to maintain updates as the team works to resolve the incident.

Three is your team chat. I’ve also been on teams that join a video call to debug live.

Either way, you need:

• Someone quarterbacking the engineering team and calling out who is doing what.
• Someone who is keeping the status page and any other channels updated.

After each incident, you should always evaluate the process. Determine what broke down and how it can be better. This is often the most important part that gets skipped.

Prevention

My first question after an outage incident is:

What was the root cause?

Once you’ve identified the root cause, you can guard against it. Add more memory to a server; or increase the number of processes; or implement a fix in the code.

It’s also good to keep track of the root cause and prevention solution for each incident. If you see clusters of issues on sequential days, you should dig in further. Sometimes sliding a toggle in AWS or Redis can solve the problem, but it could also be a temporary solution. If you want reliability, it requires diligence. Get to the root cause and improve the system.

The Principle

Make the invisible visible before it becomes unavailable and unreliable.

The Practice

Here’s a simple practice you can use after every incident to reveal the weak points:

1. Reconstruct. Establish what happened, when, and through which dependencies.
2. Prediction. Find the leading indicator for the outage to improve monitoring or alerts.
3. Process. Identify friction in detection, ownership, communication, mitigation, or recovery.
4. Prevention. Make a structural change that removes, contains, or reduces the failure mode.
5. Compound. Feed what was learned back into monitoring and future response.