From 'It Runs' to 'It Can Be Maintained': Three Foundations of Reliable Systems

A system being able to start only proves that it passed the earliest gate. Once it reaches production, teams care about three questions: Can we locate a problem quickly? Will a local fault spread? Can every release be controlled?

Reliability is not a one-time acceptance check before launch. It is an engineering capability that keeps shortening the time from discovery to containment to recovery. These three foundations connect in order: make the system explain what happened, keep a fault inside its boundary, and give every change a safe way out.

Establish observability first

#

Logs, metrics, and traces are not three unrelated tools. They answer different layers of the same question:

• metrics tell us where something is abnormal;
• traces tell us what a request went through;
• logs explain why it happened.

The most valuable starting point is not collecting everything. Define a service-level objective for one core user journey first. Only after “normal” is explicit can an alert avoid becoming noise no one handles.

Work backward from the user journey

#

Choose a path that genuinely affects the business—for example, “sign in → place an order → payment succeeds”—and define a small number of actionable signals for every step.

1
2
3
4
5
6
7
8

slo:
  availability: 99.9%
  latency_p95: 300ms
  window: 30d
alert:
  burn_rate:
    fast: 14.4
    slow: 2

The important part of this configuration is not the exact number. It is the connection between an alert and the rate at which the error budget is being consumed: fast burn wakes the on-call engineer quickly, while slow burn exposes sustained degradation.

Make one request traceable

#

Metrics, traces, and logs must share the same trace_id or request identifier. The ideal troubleshooting path goes from an SLO alert to an abnormal service, down to a slow trace, and finally to the structured log for the same request.

Limit the blast radius

#

Timeouts, retries, circuit breakers, and rate limits all address the same problem: do not let a local anomaly become a system-wide collapse. Be especially careful with unbounded retries; they often amplify traffic while the downstream service is already overloaded.

Reliability design is not about eliminating failure. It is about making failure predictable, observable, and recoverable.

Give dependencies an explicit budget

#

If the end-to-end budget for a request is 800 ms, every downstream timeout cannot also be 800 ms. Reserve time for queuing, network transfer, dependency calls, and local work. An upstream timeout should always exceed the downstream timeout plus cleanup time.

Retries need three boundaries: retry only idempotent operations, cap the total attempts, and stay within the overall time budget. Otherwise a retry is not a recovery mechanism but a traffic amplifier.

Make degradation a product decision

#

Degradation should not be debated for the first time during an incident. Whether search can return fewer fields, recommendations can fall back to popular content, or an uncertain payment can enter a pending state should be decided by product and engineering in advance and verified in a rehearsal.

Make change safer

#

Small-batch releases, automatic rollback, and configuration audits often improve production stability faster than a large refactor. Every incident review should result in a system change, not just a reminder to be more careful.

Turn a release into an experiment with an exit condition

#

A trustworthy delivery pipeline should answer four questions: What changed? Who is affected first? How will we detect an abnormal result? How quickly can we return? Canary releases become genuinely useful only after those decisions and rollback steps are automated.

Configuration and database changes especially need to be reversible. Prefer an expand-and-contract sequence—support both shapes, switch traffic, then clean up—so a code rollback does not encounter an unknown new structure.

A review must change the system

#

A high-quality review should produce at least one verifiable improvement: add an automated guardrail, shorten detection time, reduce recovery steps, or remove a dangerous manual action. If the conclusion is only “raise awareness,” the same incident is likely to return.

In short

#

Complete observability for the core path, set explicit boundaries around dependencies, and reduce the impact of every change. These three basic steps form the skeleton of most reliability work.

Mature reliability does not mean “nothing ever fails.” It means the team can see a problem quickly, contain it locally, and return safely from every change.