Reliability engineering runs on a handful of small calculations that everyone half-remembers. This is a tour of them, using the SLA, Percentile, Backoff, Capacity and CIDR tools — most of which you can also run inline from the command palette: type sla 99.99 or cidr 10.0.0.0/22 or bare arithmetic like 0.999 * 0.999 and the answer appears without opening anything.
Error budgets: what a nine actually buys
An SLA percentage only means something converted into time you're allowed to be down. Type sla 99.9 and read it off: about 43m 49s per 30-day month, 8h 46m per year. The neighbours:
99% 7h 18m / month — a bad afternoon
99.9% 43m 49s / month — one decent incident
99.95% 21m 54s / month
99.99% 4m 23s / month — no human is paging fast enough
99.999% 26s / month — automation or nothing
That last column is the real content of the table: at four nines, your error budget is spent before a human finishes reading the alert. The budget framing also makes deploys legible — every risky change spends from the same 43 minutes that incidents do.
Availability composes downward. A request that traverses two 99.9% services in series succeeds only when both are up: 0.999 × 0.999 = 0.998001, so the pair is ≈99.8% — the chain is worse than its weakest link, not equal to it. Four such hops and you're at 99.6%, over 21 hours a year, even though every individual service is "three nines". The Capacity tool's availability panel does this for series chains and for the inverse case — parallel redundancy, where the system is only down when every replica is down at once, which is how two 99% boxes compose to 99.99%.
Averages lie; p99 is a user
Paste this into the Percentile tool (or p99 ... in the palette) — 20 response times in ms:
48 50 51 52 53 53 54 55 55 56
56 57 58 58 59 60 62 64 900 2400
Mean: ~220ms. Looks acceptable on a dashboard. But p50 is 56ms and the worst two requests took 0.9s and 2.4s — the mean is a number no request actually experienced. The tool computes nearest-rank percentiles (p50/p75/p90/p95/p99), and nearest-rank is honest in a way interpolation isn't: it reports a latency some real request had.
Two consequences worth internalizing. First, p99 is a user-experience number, not a tail curiosity: a page that fans out to 30 backend calls hits at least one per-call p99 on 1 - 0.99^30 ≈ 26% of page loads. Your p99 is a quarter of your users. Second, you need samples to even see a p99: with nearest-rank, the p99 of 50 samples is just the 50th value — the max. Below a few hundred samples per window, your "p99 spike" alert is really a "worst single request" alert; either widen the window or accept that you're alerting on noise.
Backoff with jitter
When a dependency fails, every client that failed at the same moment will, under naive exponential backoff, retry at the same moment — the thundering herd, arriving in synchronized waves at exactly the cadence the struggling service can least afford. The fix is randomness. Full jitter, the usual best default:
delay_n = min(cap, base × factor^(n-1))
sleep_n = random(0, delay_n)
The Backoff tool plans this as a table — per-attempt delay, the min–max range each jitter mode can produce, and the cumulative expected wait — for none, full, equal (half fixed, half random, keeping a floor under every wait) and decorrelated (the window grows off the previous sleep). It shows ranges rather than rolling dice, so you can read the same schedule twice and reason about worst cases: with base 100ms, factor 2, cap 30s, attempt 8 has a 12.8s delay and full jitter puts the sleep anywhere in 0–12.8s.
Cap both the delay and the attempts — an uncapped exponential reaches minutes by attempt 12 and your caller gave up long ago. Better than a fixed attempt count is a retry budget: allow retries up to, say, 10% of request volume, so retries can't more-than-double load on a service that's already drowning. And never retry non-idempotent operations without an idempotency key; the herd you create will be a herd of duplicate charges.
Headroom: the n+1 arithmetic
If your fleet runs n nodes at utilisation u, losing one node sends the survivors to u × n/(n-1). For the fleet to stay under a target utilisation t after a single failure:
u ≤ t × (n - 1) / n
Four nodes with a 75% target: run at no more than 75 × 3/4 ≈ 56% day to day. Three nodes: 50%. The palette calculator does these in one line — 75 * 3 / 4. The Capacity tool's Little's Law panel (L = λ × W) pins the third of throughput, latency and concurrency from the other two, which is how you convert "we serve 400 req/s at 250ms" into "there are 100 requests in flight, so a 100-worker pool has zero headroom".
Subnet sizing is the same shape of arithmetic. cidr 10.0.0.0/22 in the palette: 1,022 usable hosts, 10.0.0.1–10.0.3.254. Planning an autoscaling group that could reach 300 instances, each holding an address, plus load balancer interfaces that consume several per subnet? A /24 (254 usable) silently caps your scale-out; the /22 doesn't. Check before the VPC is immutable, not after.
The trap: fleet p99 isn't the average of node p99s
Percentiles do not compose. If you scrape a per-node p99 from each of ten servers behind a load balancer and average them — or worse, alert on the mean of p99s — you get a number with no operational meaning. One node with a dying disk can have a p99 of 4s while nine healthy nodes sit at 60ms; the "average p99" reads 454ms and looks like a mild fleet-wide regression, when the truth is one broken machine hurting ~10% of traffic badly. The reverse failure also happens: every node's p99 looks fine while the fleet p99 is awful, because the slow requests all share a cause (one hot shard, one bad AZ) that no single node sees enough of.
The only correct fleet percentile comes from pooling — merge the raw samples (or mergeable sketches: HDR histograms, t-digests) and take the percentile of the union. That's what the Percentile tool computes when you paste everything into one box: one distribution, one honest p99. Keep per-node p99s too — as an outlier detector across nodes, where their non-composability is exactly what makes a sick node stick out.