Engineering guide · Integrations
Reliable System Integrations: Retries, Reconciliation and Monitoring
An integration is not complete when one successful request moves data between two systems. It is complete when duplicate, delayed, invalid and missing events can be detected and recovered without guessing. Reliability comes from designing the failure path as carefully as the happy path.
By Anjula Paulus, Founder and software engineer · Published · 4 min read
Key takeaways
- Assume messages can be duplicated, delayed, reordered or lost.
- Make repeated processing safe with idempotency and stable identifiers.
- Retry only transient failures and cap every retry policy.
- Reconcile business records independently of event delivery.
- Give operators enough context to replay safely without editing databases.
01
Write the contract before the connector
Define the source of truth for each field, identity mapping, required and optional values, ordering expectations and what a successful exchange means. Version the contract and keep representative examples. When external APIs change, contract tests should fail before production data does.
02
Make processing idempotent
Networks time out after a remote system has already accepted work. Webhooks are commonly delivered more than once. Use a stable event or business identifier and record processing outcomes so repeating the same request does not create a second invoice, payment or shipment.
Idempotency is a business rule, not only a header. Decide what counts as the same operation and how long its identity must be remembered.
03
Retry selectively
Retry timeouts, rate limits and temporary server failures with exponential backoff and jitter. Do not repeatedly retry invalid data, expired credentials or a rejected business rule. Classify errors so permanent failures reach an operator instead of consuming the queue forever.
Cap attempts, record the final state and provide a controlled replay path after the cause is fixed. A dead-letter queue without ownership and alerts is only a quieter form of data loss.
04
Reconcile the business state
Even well-designed event delivery can fail in unexpected ways. Run scheduled reconciliation that compares business records across both systems: counts, totals, identifiers and status transitions. Reconciliation detects silent gaps that transport-level monitoring cannot see.
Define which side wins when values differ and whether correction is automatic or reviewed. Financial and inventory differences often require an auditable human decision.
05
Expose an operational view
Logs should connect the source record, integration event, attempts and destination result using correlation identifiers. Metrics should show throughput, latency, error classes, retry depth and oldest pending work. Alerts should describe business impact, not merely that a process returned an exception.
- Which records are affected?
- Is data delayed, duplicated or rejected?
- Will the system recover automatically?
- What action can an operator take safely?
- How will successful recovery be verified?
06
Test failure deliberately
Test duplicate webhooks, timeouts after success, rate limits, malformed data, expired tokens, partial outages and out-of-order events. Verify that alerts fire, retries stop appropriately, replay is safe and reconciliation finds the final discrepancy.
Questions
Common questions about building reliable integrations
What makes an API integration reliable?
A clear data contract, idempotent processing, selective bounded retries, observable failure states, reconciliation and a safe replay process. Success responses alone do not prove business records are consistent.
Should every failed API call be retried?
No. Retry transient conditions such as timeouts, rate limits and temporary server errors. Invalid data, authentication failures and rejected business rules need correction or human action.
Why is reconciliation needed if webhooks are reliable?
Reconciliation independently compares business state and catches silent gaps, manual changes and unexpected failure modes that delivery monitoring may miss.