Integration Resilience Patterns

Integration resilience is the discipline of expecting things to fail and degrading gracefully when they do. The patterns are well-documented — circuit breakers, retries, timeouts, bulkheads, idempotency. The patterns being well-known doesn't mean they're well-applied; most enterprise integration incidents we participate in involve at least one of these patterns being implemented incorrectly or missing entirely.

This is the year-end synthesis from a year of integration delivery: which patterns survive contact with real failures, the implementation mistakes that recur, and the operational habits that determine whether the patterns produce the resilience they promise.

The failure modes you are designing against

Before discussing patterns, the failure modes worth designing against:

Transient transport failures. Network hiccups, momentary unavailability, intermittent timeouts. These resolve on their own within seconds; the architectural response is graceful retry.

Sustained dependency outages. A downstream system is down for minutes or hours. Retry doesn't help; the architectural response is circuit-breaking and graceful degradation.

Slow dependency. A downstream system is up but slow. This is the most common cause of cascade failures in integration estates because callers wait, accumulate connections, and exhaust their own capacity. The architectural response is timeout discipline and bulkheading.

Partial dependency failure. A downstream system is working for some operations but failing for others. The architectural response is operation-level circuit breaking, not platform-level.

Catastrophic upstream demand. Inbound traffic exceeds the platform's design capacity. The architectural response is rate limiting and load shedding.

Internal failure propagation. A failure in one workload cascades to others sharing infrastructure. The architectural response is bulkheading.

Each pattern below addresses specific failure modes. The patterns combine to produce resilient estates; using one pattern in isolation rarely solves the resilience problem.

Circuit breakers, applied correctly

The circuit breaker pattern: when failures from a dependency exceed a threshold, stop calling the dependency for a period and fail fast. After the period, allow trial calls; if they succeed, resume normal operation.

The discipline that makes circuit breakers work:

• Per-operation, not per-system. A dependency may be failing for a specific operation while others work. Circuit-breaking the whole dependency is heavy-handed. Per-operation circuit breakers preserve more of the working surface.
• Threshold tuned to actual failure characteristics. A circuit breaker that trips on the first failure produces too much oscillation. A threshold that requires sustained failure (5 of last 10 calls, error rate above 50% over 30 seconds) produces more useful behaviour.
• Half-open state with limited trial. When the breaker considers re-closing, it allows one or a few trial calls. If they succeed, full reopening. If they fail, back to open. The trial discipline prevents oscillation.
• Observable. The state of every circuit breaker is visible in operational dashboards. Operations engineers can see which breakers are tripped without diagnostic queries.

What goes wrong: circuit breakers that trip but never reset, leaving operations engineers manually clearing them. Or circuit breakers that don't trip because the threshold is too lenient, allowing failures to cascade. Or circuit breakers without observability, leaving operations to discover them indirectly.

Retry, with discipline

Retry is the most common resilience pattern and the one most commonly implemented poorly. Done right, retry hides transient failures from upstream callers. Done wrong, retry amplifies load during incidents and produces worse outcomes than no retry at all.

The disciplines:

• Retry only safe operations. Idempotent operations can be retried freely. Non-idempotent operations cannot — retry may produce duplicate side effects. The application has to know which is which.
• Exponential backoff. First retry after 100ms; next after 500ms; next after 2s; etc. Linear retry produces too much load too quickly.
• Jitter. Add randomness to retry intervals so retries from many callers don't synchronise and produce thundering-herd effects on the recovering dependency.
• Bounded retry count. Three to five retries is enough. Beyond that, the issue is not transient; further retries waste resources.
• Eventual fail-fast. When retries exhaust, return a clear error to the caller. Don't return a partial success. Don't hide the failure.

The estates that retry poorly produce incidents where a moderate dependency slowdown becomes a full outage as retries pile up. The estates that retry well preserve service through dependency issues that would otherwise be visible to users.

Timeout discipline

Timeouts are the most important and most under-set value in integration estates. Without a timeout, a slow dependency holds the caller forever. The caller's resources accumulate; eventually the caller fails too; the failure propagates upstream.

The disciplines:

• Every external call has a timeout. Not "usually." Every. Database calls, HTTP calls, message broker operations, internal service calls. No exceptions.
• Timeouts are smaller than upstream timeouts. If your service has a 30-second SLA, your calls to dependencies need to be less than that — typically substantially less, so retries can fit within the budget.
• Timeouts are tuned to actual latency. A 30-second timeout when the dependency normally responds in 100ms is too lenient — slow responses pile up before the timeout fires. A timeout slightly above p99 normal latency catches actual slowness quickly.
• Connection timeouts separate from read timeouts. Connection timeout (can we reach the dependency?) is usually short; read timeout (is the dependency responding?) is longer.

Estates we audit consistently have timeout gaps. Many "occasional 30-second hangs" trace to a missing timeout on a single call somewhere deep in the integration.

Bulkheading

Bulkheading isolates failure domains. A failure in one part of the system can't bring down other parts that share infrastructure.

Patterns:

• Connection pool isolation. Each downstream dependency has its own connection pool. A dependency that hangs doesn't starve other dependencies' connections.
• Thread pool isolation. Different operations run on different thread pools. A slow operation doesn't consume threads needed for other operations.
• Queue isolation. Different workloads have different queues. A backed-up queue doesn't impact other workloads.
• Cluster isolation. Critical workloads on dedicated clusters; less critical on shared clusters. The dedicated clusters are immune to noisy-neighbour effects.

Bulkheading is unglamorous. It produces estates where individual failures stay local; it doesn't produce visible improvements in steady state. The value is in the incident that doesn't cascade.

Idempotency as a contract

Idempotency — the property that an operation produces the same result whether it's executed once or many times — is the architectural foundation that makes retry safe. Without idempotent operations, retry produces duplicates; with idempotent operations, retry is safe to use liberally.

Implementing idempotency:

• Idempotency keys. Every request carries a unique key that the receiver uses to detect duplicates. If the same key arrives twice, the receiver returns the previous result without re-processing.
• Server-side deduplication. The receiver stores idempotency keys for some retention period. Duplicate requests within that window are detected and handled correctly.
• Documented idempotency contracts. API documentation specifies which operations are idempotent and how. Consumers know which operations they can safely retry.

Many estates we audit have retry without idempotency. The retries succeed; duplicate side effects pile up; reconciliation queues accumulate work to undo the duplicates. The original failure was minor; the duplication is major.

Graceful degradation

When dependencies fail, the application has choices. Fail entirely, or degrade gracefully?

Patterns:

• Cached fallbacks. When the live dependency fails, serve recent cached data. The user sees slightly stale data instead of an error.
• Default values. When personalisation fails, serve the default experience. The user gets a working page even if it's not optimised for them.
• Feature flags for degradation. When a non-critical feature fails, disable it. The rest of the experience works.
• Read-only mode. When the write path fails, accept read traffic. Users can browse even when they can't modify.
• Graceful queue. When a downstream system is overwhelmed, queue requests rather than rejecting them. Process when capacity returns.

The architectural skill is identifying which features can degrade gracefully. Some can't — payment processing, regulatory transactions, security-critical operations. For these, hard failure is the correct response. For others, degradation preserves user experience through dependency issues.

The operational habits

Resilience patterns are necessary; operational habits are what make them work in practice.

Chaos engineering. Deliberately inject failures in production-like environments to verify that resilience patterns work as designed. Netflix-style Chaos Monkey is the well-known version; many enterprises use simpler practices (intentional latency injection, planned dependency outages, controlled-circuit-breaker tests). The practice produces estates where failures are familiar rather than surprising.

Failure runbooks. When the circuit breaker trips for dependency X, the on-call engineer knows what to do. When a queue backs up, the runbook covers diagnosis and remediation. The patterns may be implemented in code; the response is in the runbook.

Postmortem rigour. Every incident gets a postmortem. The postmortem identifies which resilience patterns worked, which didn't, and which were missing. The findings produce engineering work. Resilience improves over years through accumulated postmortem learning.

Periodic resilience audits. Quarterly review: which timeouts are missing? Which retries are non-idempotent? Which bulkheads have eroded? Resilience entropy is real; periodic audit catches it.

What we recommend

For an enterprise integration estate building resilience:

1. Audit timeouts first. Every external call needs one. Missing timeouts are the most common single resilience gap.
2. Audit retry patterns. Are retries bounded? Idempotent? Backed off with jitter?
3. Implement circuit breakers per dependency, per operation. Make them observable.
4. Bulkhead shared infrastructure. Connection pools, thread pools, queues, clusters.
5. Define idempotency contracts for inbound operations. Document them.
6. Identify graceful degradation paths for non-critical features.
7. Establish chaos engineering practice. Start small; the discipline matters more than the sophistication.

For an estate with recurring incidents:

1. Audit the most recent incidents for the resilience patterns that would have prevented or contained them.
2. Identify the recurring gaps. They're usually timeout discipline, idempotency, or bulkheading.
3. Close them systematically rather than per-incident.

Resilience is unglamorous engineering. It produces estates that don't have the dramatic incidents the unprepared estates have. The work is largely invisible when it's working. The value is in the night you don't get paged because the patterns held when they should have.