Resilience Patterns: Circuit Breaker, Bulkhead, Timeout Budgets

How does cascading failure begin?

Always with the same scenario: a dependency slows down — it does not crash, it slows. Threads waiting on it accumulate, pools fill, those waiting on the waiters accumulate, and within five minutes even healthy services stop answering. Every resilience pattern serves one goal: stopping a component's failure at that component's boundary. This article builds the four core patterns and the way they work together.

The pattern map

Timeouts: every call has a due date

A remote call without a timeout is a thread pawned indefinitely. Two disciplines: first, an explicit timeout on every outbound call (HTTP, database, queue, cache) — never trust library defaults, most mean 'forever'. Second, timeouts designed as a budget: if the user is promised an answer in 2 seconds, the chain's timeouts must fit within that budget — if service A waits 1.5s and then gives B a 1.5s timeout, the budget is blown at the first link. The budget travels along the call chain as remaining time; if what remains cannot suffice, the call is not made at all — fail fast.

Retries: medicine and dosage

Retries hide transient failures (network blips, momentary congestion) from users; overdosed, they are poison that multiplies load. The rules are crisp: retry only idempotent, transient failure classes (connection errors yes, 400 never, and timeouts with the awareness that the operation may have succeeded); exponential backoff + jitter (to break synchronized retry waves); a retry budget (a small percentage of total traffic — as error rates rise, retrying itself is throttled); and one layer in the chain retries — if every layer retries independently, one failure in a three-layer chain explodes into 27 calls.

Circuit breaker: stop beating the dead

When a dependency is truly sick, retrying is not medicine but torture. The breaker is a three-state machine: Closed (normal flow; error rate watched) → threshold crossed, Open (calls not made at all, fallback returns instantly; breathing room for the sick system) → after a cooldown, Half-Open (limited trial traffic; success returns to Closed, failure back to Open). Two engineering subtleties: thresholds must be volume-aware (5 errors in 10 requests vs 5,000 in 10,000 — same ratio, different meaning), and breaker state must be visible on a dashboard — 'which circuits are open?' is the first screen checked during an incident. Fallback design matters as much as the breaker: stale data from cache, a default answer, or gracefully hiding the feature — a blank screen is not a fallback.

Bulkhead: compartmentalizing the ship

Named after ship partitions, the pattern is resource isolation: each external dependency gets its own pool/concurrency limit — payment calls may use at most 50 concurrent permits, recommendations 20. When recommendations slow down, they exhaust only their own 20 permits; the payment path is untouchable. In the virtual-thread era, semaphore-based permits replace thread pools, but the principle stands: shared resources mean shared fate — every non-isolated dependency is a silent partner of your most critical flow. The critical/non-critical split applies here too: the order path and reporting must not share a connection pool.

Orchestrating the patterns: order and ownership

The four patterns nest, and their order matters: outside-in bulkhead (permit available?) → circuit breaker (circuit closed?) → timeout (budget left?) → the call → retry decision by failure class. Libraries like Resilience4j compose the chain declaratively; what is critical is that configuration is written deliberately per dependency — the same copy-pasted values on every service is decoration, not resilience. Each dependency's SLA, failure character and business criticality differ; parameters derive from that reality and are validated with chaos and load tests.

Business impact: failure local is an incident; failure spread is a crisis

The commercial value of resilience patterns is a smaller blast radius: when recommendations crash, recommendations disappear — sales continue; when the payment provider slows, payments queue — the site stays up. That difference is the difference between 'partial service impact' and 'full outage' in incident reports, and it converts directly to money across SLA penalties, lost revenue and reputation. Once built, the patterns must not remain undrilled: dependency-failure simulations (chaos tests) are the only proof that fallbacks actually work.

Frequently asked questions

Should every external call get a circuit breaker?

Every significant dependency that crosses a network and can fail — yes. The cost is low and the payoff at failure time is large; but parameters must be tuned per dependency.

How do I choose timeout values?

From the dependency's real latency distribution (p99 + margin) and backwards from the end-to-end budget. A default of 30 seconds means, in practice, 'no timeout'.

What is the relation between retries and idempotency?

Retries are only legitimate for safely repeatable operations. For writes without an idempotency key, a post-timeout retry risks double effects (double order, double charge).

Are these patterns needed in a monolith?

Yes — a monolith also calls the database, the payment provider, email and search services. Cascading failure is not a microservices monopoly.

Resilience checklist

1. Explicit timeout on every outbound call; end-to-end latency budget defined

2. Retries only for idempotent+transient classes; backoff+jitter+budget in place

3. Retries at one layer of the chain; no cross-layer multipliers

4. Circuit breakers + meaningful fallbacks on critical dependencies

5. Bulkhead/permit limits per dependency; critical paths isolated

6. Breaker states and dependency health on dashboards

7. Chaos/failure drills regular; fallbacks proven

SSH Yazılım designs resilience patterns and implements them on Resilience4j end to end. Let us shrink your blast radius together.