The Load-Bearing Wall Problem

A single point of failure is not simply a thing that breaks. Everything breaks. A single point of failure is a component whose failure has a blast radius equal to your entire product — one process dies at 3 a.m. and every customer sees the same white screen.

The uncomfortable part is that these components are almost never the flaky ones. They are usually the most reliable thing you own. That is precisely how they got the job: it worked, so you routed one more thing through it, and then another, and after two years of that it is holding up the building.

01 / WHY YOU CANNOT SEE YOUR OWN

They hide behind their own reliability

Nobody sits down and designs a chokepoint. They accumulate. A shared Redis that started as a cache and is now also your session store, your rate limiter, and your job queue. A payment provider that was fine for three years. One Postgres primary that has never once gone down, which is also the reason nobody has restored a backup since the day it was created.

Uptime is not evidence of resilience. It is often just evidence that you have not been unlucky yet. The question worth asking is never "is this reliable?" — it is "what happens to my customers when this is gone for four hours?" If you cannot answer that in one sentence, you have found something worth looking at.

02 / TAXONOMY

The five places they hide

Most teams audit the first two and stop, which is a shame because the last three cause the outages that make the news.

Infrastructure

The single server, the single availability zone, the one load balancer, the NAT gateway everything egresses through.

Data

One primary database with no replica. Or a replica that exists but has never been promoted, which is the same thing with extra confidence.

Third parties

Payments, SMS, email, maps, auth providers, courier APIs. You do not control their uptime, their deploys, or their decision to rate-limit you on a Friday.

The control plane

DNS. TLS certificate renewal. Your CI/CD pipeline. Your secrets manager. Your identity provider. These are the sneakiest category, because they are not in the request path — until you need to ship a fix and discover you cannot deploy, cannot log in, and cannot roll back.

People

One engineer who understands billing. One person with production credentials. One person who knows why that cron job exists. A bus factor of one is a single point of failure with a calendar and a right to take holiday.

Field note

The most expensive outages usually involve two categories at once: a data-layer failure during a window when the one person who knows the recovery procedure is unreachable.

03 / DISCOVERY

Three exercises that actually surface them

Write the dependency inventory. List every external thing a request touches — services, databases, caches, queues, APIs, DNS, config sources. Beside each one write a single sentence: what the customer sees if this returns errors forever, starting now. The rows where you hesitate are your real backlog.

Rank by blast radius, not by likelihood. Probability estimates for rare events are close to worthless. Score each dependency on three things instead: how many users are affected, how long recovery takes, and whether the damage is reversible. A component with a small chance of causing permanent data loss outranks a flaky one that self-heals in thirty seconds.

Unplug it on purpose. A failover you have never triggered is a hypothesis, not a mechanism. Pick a Tuesday morning, tell everyone, and kill the replica, block the payment provider, expire the token. Game days feel absurd until the first one, which typically finds three broken assumptions inside an hour.

04 / DEMONSTRATION

Watch the blast radius

Here are two versions of the same small commerce system. Take a component offline in each and compare what the customer experiences.

05 / RESPONSE

The ladder, not the switch

"Eliminate single points of failure" is advice that reliably produces nothing, because full elimination is expensive and most teams correctly decide they cannot afford it — and then do nothing at all. The useful framing is a ladder. Every dependency sits on a rung, and you move the important ones up one rung at a time.

L0

Unmonitored

It fails, and you learn about it from a customer on social media. Recovery time is unbounded.

L1

Monitored, manual recovery

You get paged, there is a runbook, and the backups have actually been restored at least once in a drill.

L2

Graceful degradation

The feature stops. The product does not. Checkout queues, search falls back to the database, the page renders from cache.

L3

Automatic failover

A replica gets promoted, traffic reroutes, a secondary provider takes over. Users see a blip.

L4

No single instance

Stateless, horizontally scaled, spread across zones or regions. Losing one is a non-event.

Here is the part worth internalising: L2 is where most of the value is, and it is by far the cheapest rung to reach. Multi-region failover is a quarter of engineering time. Deciding that a payment provider timeout should park the order in a queue instead of returning a 500 is an afternoon, and it converts a total outage into a delay nobody complains about.

06 / PATTERNS

What buys you graceful degradation

• Timeouts on every remote call. The most common cause of a full outage is not a crash — it is an unbounded wait. A dependency that hangs holds your worker threads, the pool fills, and pages that never needed that dependency stop rendering too.
• Circuit breakers. After N consecutive failures, stop calling and fail fast with a known response. This protects you from the dependency and protects the dependency from you while it is trying to recover.
• Bulkheads. Separate connection pools, worker queues, and thread budgets per dependency, so one slow integration cannot consume the capacity the rest of the system needs.
• Turn synchronous into durable-asynchronous. Accept the request, write the intent to a queue, return a confirmation, retry with backoff. This single move converts a large class of hard dependencies into soft ones.
• Idempotency keys. Retries are only safe if a duplicate is harmless. Without them, your resilience mechanism becomes a double-charging mechanism.
• Read-only mode. If writes are impossible, serving reads is enormously better than serving nothing. Build the flag before you need it.
• Kill switches. A feature flag that disables a dependency in seconds beats a hotfix deploy that takes twenty minutes — assuming your deploy pipeline is even up.

07 / TRAPS

Redundancy that is not redundancy

• Shared failure domains. Two instances on the same host, two hosts in the same rack, two zones on the same power feed, two services behind the same DNS record. You have two of something and one of what matters.
• Correlated config. Identical replicas receiving an identical bad config push fail identically, at the same instant. Most large outages are this.
• The failover mechanism as a new SPOF. The consensus layer, the health checker, the orchestrator. Adding HA can add a smarter, more central thing to lose.
• Retry storms. Every client retrying at once turns a brief blip into a sustained outage. Use exponential backoff with jitter, and cap total attempts.
• Untested restores. A backup is a claim about the past. A restore drill is evidence. The gap between them is where companies die.

08 / MONDAY

A short, concrete starting list

• Write down every external dependency in your request path, with one sentence on what the customer sees when it is gone.
• Find every remote call with no timeout configured. Set one. This is usually the single highest-value hour of the whole exercise.
• Restore your production backup into a scratch environment and time it. Record the number.
• Name the one thing that only one person on the team can do, and schedule a session where a second person does it.
• Pick your riskiest third-party dependency and move it from L1 to L2: queue it, cache it, or give it a defined fallback response.
• Book a game day. Take one real component offline, in working hours, with the team watching.

The goal was never zero single points of failure. Some are rational: you are not going to run a second payment provider in your first year, and you should not. The goal is zero unknown single points of failure — a written list, each with a known blast radius, a known recovery time, and a deliberate decision about which rung of the ladder it deserves to be on.

Outages you have thought about in advance are incidents. The other kind are surprises, and surprises are what turn a bad hour into a bad quarter.

Shefat Shohan

Software Engineer, Codevioso

Fullstack developer passionate about building scalable, user-focused web applications and AI-powered solutions.