SfSkills Domains Docs GitHub

Home / integration / retry-and-backoff-patterns

retry-and-backoff-patterns

Implementing resilient integration retry logic in Salesforce: exponential backoff, jitter, idempotency keys, dead-letter queues, and circuit breaker patterns for Apex callouts. Use when designing callout retry behavior, preventing thundering-herd issues, or handling persistent integration failures. NOT for how many callouts fit in a transaction or CalloutException after DML — use integration/callout-limits-and-async-patterns. NOT for chaining or debugging Queueable jobs that make no callout — use apex/apex-queueable-patterns.

Open the skill folder on GitHub

Retry and Backoff Patterns

This skill activates when a practitioner needs to implement resilient retry logic for Apex callouts — covering exponential backoff with jitter, idempotency via External Id upserts, dead-letter queue handling, and circuit breaker patterns using Custom Metadata. It does not cover native platform retry mechanisms (Outbound Messages, Platform Events) beyond documenting them as alternatives.

Before Starting

Gather this context before working on anything in this domain:

Core Concepts

No Thread.sleep() in Apex — Use the Queueable Delay Parameter

Apex has no Thread.sleep(), and a synchronous retry loop cannot introduce delay. But Apex can schedule a delayed retry explicitly — the claim that it can only hope for scheduler jitter is out of date. System.enqueueJob takes an optional delay: "use the System.enqueueJob(queueable, delay) method to add queueable jobs to the asynchronous execution queue with a specified minimum delay (0–10 minutes)."

System.enqueueJob(new MyQueueableClass(), 5);            // 5-minute minimum

AsyncOptions opts = new AsyncOptions();                   // equivalent form
opts.MinimumQueueableDelayInMinutes = 5;
System.enqueueJob(new RetryCalloutJob(orderId, retryCount), opts);

Exponential Backoff with Jitter

Exponential backoff calculates the delay between retries as baseDelay * 2^retryCount. Without jitter, every concurrent failing job wakes at the same moment and hammers the external system simultaneously — the thundering-herd problem. Adding jitter (+ (Math.random() * baseDelay)) scatters retry times across the time window.

Because System.enqueueJob(queueable, delay) takes whole minutes, 0–10, convert the computed backoff to minutes and clamp it — do not pass a seconds value into a minutes parameter:

Integer delayMinutes = (Integer) Math.min(
    10, Math.max(0, Math.ceil((baseDelaySeconds * Math.pow(2, retryCount) + jitter) / 60))
);
System.enqueueJob(new RetryCalloutJob(payload, retryCount + 1, maxRetries), delayMinutes);

The 10-minute ceiling is the real design constraint: 1/2/4/8 minutes fits, anything past that flattens to 10. Log the calculated delay alongside the clamped one so a flattened retry is visible in the data rather than looking like a scheduler anomaly.

Idempotency Key via External Id Upsert

On retry, the same payload may be sent and accepted by the external system, but if the original request succeeded and only the response was lost (network timeout), a naive retry creates a duplicate. The defense is to assign a stable idempotency key before the first attempt — typically a UUID stored in External_Id__c on the driving record — and pass it as a request header or body field. On the Salesforce side, use upsert on External_Id__c to prevent duplicate SObject creation from reprocessing. The key must survive retries unchanged.

Max Retry Guard and Dead-Letter Queue

Every retry implementation must have an explicit maximum retry count (3–5 is typical). When retryCount >= maxRetries, the job must not re-enqueue. Instead, write a dead-letter record — a Failed_Integration_Log__c custom object capturing the payload, error message, HTTP status, retry count, and timestamp — and optionally fire an alert (Platform Event or email). Without this guard, a permanently failing integration exhausts the async Apex limit silently.

Common Patterns

Pattern 1: Queueable Retry Chain with Exponential Backoff

When to use: An Apex callout fails with a transient error (HTTP 429, 503, or timeout) and must be retried automatically with increasing delay.

How it works:

  1. The initial callout attempt is made from a Queueable execute() method.
  2. On failure, increment retryCount on the job instance. Calculate delaySeconds = baseDelay * Math.pow(2, retryCount) + (Math.random() * baseDelay), then convert to whole minutes clamped to 0–10.
  3. If retryCount < maxRetries, call System.enqueueJob(new RetryCalloutJob(payload, retryCount, maxRetries), delayMinutes) — pass the delay rather than relying on scheduler jitter.
  4. If retryCount >= maxRetries, write a Failed_Integration_Log__c record and stop.

Why not a for-loop retry in the same transaction: Each Http.send() call consumes a callout from the 100-per-transaction limit. More critically, you cannot introduce real delay in a synchronous loop, so you just hammer the endpoint repeatedly in milliseconds — worse than no retry.

Pattern 2: Circuit Breaker via Custom Metadata

When to use: An external system is degraded for extended periods. Rather than retrying every call and burning async limits, a circuit breaker detects open-circuit state and skips the callout entirely until the system recovers.

How it works:

  1. Create a Circuit_Breaker_Config__c Custom Metadata record with fields: Is_Open__c (Boolean), Opened_At__c (DateTime), Cool_Down_Minutes__c (Number).
  2. At the start of the Queueable execute(), query (or cache via a static variable) the CMDT record.
  3. If Is_Open__c = true AND Opened_At__c + Cool_Down_Minutes__c > now, skip the callout and write a log entry.
  4. If the cool-down has elapsed, treat the circuit as half-open: attempt one callout. Success → flip Is_Open__c = false (via an Apex update or a named flow). Failure → keep open and reset Opened_At__c.
  5. Toggling Is_Open__c manually also enables operators to manually open or close the circuit without a code deploy.

Why not a flag on a custom object: CMDT records are cached at the platform level and do not consume SOQL queries per transaction (after the first load in a request). Custom objects do.

Pattern 3: Idempotency Key Guard

When to use: The external system does not natively deduplicate requests (no idempotency-key header support), and a duplicate call creates duplicate data.

How it works:

  1. Before the first callout attempt, generate String idempotencyKey = [String UUID or ExternalId from driving record].
  2. Include the key in the request body or a custom header (X-Idempotency-Key).
  3. On the Salesforce side, use Database.upsert(record, Schema.SObject.Fields.External_Id__c, false) to prevent double-insert on reprocessing.
  4. Log the key on Failed_Integration_Log__c so support teams can trace retried requests.

Decision Guidance

SituationRecommended ApproachReason
Callout fails in a trigger or LWC controllerMove callout to Queueable first, then add retry chainSynchronous context cannot support delay or re-attempt patterns
Transient HTTP 429 or 503 errorQueueable retry chain with exponential backoff + jitterHandles temporary unavailability without thundering-herd
Timeout (no HTTP response received)Retry with idempotency key — success confirmation is unknownWithout idempotency key, retry may double-process
External system down for hoursCircuit breaker via CMDT + dead-letter logRetrying endlessly burns async Apex quota for no benefit
Max retries exceededWrite Failed_Integration_Log__c and alertEnables manual intervention; prevents silent data loss
Outbound Messages or Platform EventsUse native retry — no Apex neededOutbound Messages retry for up to 24 hours; Platform Events replay for 3 days

Step-by-step instructions for an AI agent or practitioner implementing retry logic for an Apex callout:

  1. Confirm execution context: Verify that the callout is already in a Queueable or Batch context. If it originates from a trigger or controller, extract it to a Queueable job first before adding retry logic.
  2. Define retry parameters: Decide maxRetries (3–5), baseDelaySeconds (1–5), and maxDelaySeconds (30–60). Document these as Custom Metadata fields in Retry_Config__mdt so they can be adjusted without a deploy.
  3. Add retry counter and idempotency key fields: Add Retry_Count__c (Integer, default 0) and Integration_Idempotency_Key__c (Text, External Id) to the driving SObject. Populate the idempotency key before the first enqueue.
  4. Implement the Queueable retry chain: In the execute() method, wrap the callout in a try-catch. On caught exceptions or non-2xx responses, increment the counter, calculate the backoff, convert it to whole minutes clamped to 0–10, and re-enqueue with System.enqueueJob(job, delayMinutes) if under the limit. Log both the calculated and the clamped delay.
  5. Implement the dead-letter path: When retryCount >= maxRetries, insert a Failed_Integration_Log__c record with payload, error, HTTP status, retry count, and timestamp. Optionally publish a Platform Event to notify an operations flow.
  6. Add the circuit breaker check: At the start of execute(), read Circuit_Breaker_Config__mdt. If the circuit is open and cool-down has not elapsed, log and return without attempting the callout.
  7. Test failure modes explicitly: Write Apex tests that mock HTTP 429, 503, and timeout responses. Assert that retry count increments, dead-letter records are written at max retries, and idempotency keys are preserved across re-enqueues.

Review Checklist

Run through these before marking integration retry work complete:

Salesforce-Specific Gotchas

Non-obvious platform behaviors that cause real production problems:

  1. The enqueue delay is a floor, not an appointment — System.enqueueJob(queueable, delay) accepts a "specified minimum delay (0–10 minutes)". The platform will not run the job before that delay, but does not promise to run it at that delay — execution still waits for a worker. So: passing >10 or a seconds-scaled value is a bug, not a longer wait; and a job enqueued with no delay may still be delayed by the org-wide default (1–600 seconds) in Apex Settings, which an explicit delay "ignores". The retry interval is bounded below and approximate above.
  2. Callout-after-DML rule applies inside Queueable — If you write a Failed_Integration_Log__c record (DML) and then attempt a callout in the same execute() method, you hit the "callout after uncommitted work" exception. Always perform DML after the callout block, or use a separate inner Queueable for the logging path.
  3. Daily async Apex limit is shared across all jobs — Unlimited retry storms (e.g., a broken endpoint during peak processing) can consume the 250,000 daily Queueable executions, blocking all other background processing. The maxRetries guard and circuit breaker are critical safety valves, not optional.
  4. System.enqueueJob() is limited to 50 per transaction — A batch of failing records all trying to enqueue retry jobs in the same transaction will hit this limit. Design retry logic so each job re-enqueues itself (1 per transaction), not so a parent job enqueues N children.
  5. Native platform retries are separate from Apex retries — Outbound Messages retry automatically for up to 24 hours at platform-managed intervals. Platform Events can be replayed for up to 3 days. Do not add Apex retry logic on top of these — it results in double-processing.

Output Artifacts

ArtifactDescription
RetryCalloutJob.clsQueueable Apex class implementing exponential backoff + jitter + dead-letter path
Circuit_Breaker_Config__mdtCustom Metadata type for per-integration circuit breaker state
Retry_Config__mdtCustom Metadata type for maxRetries, baseDelay, maxDelay per integration
Failed_Integration_Log__cCustom object for dead-letter records with payload, error, and retry metadata
RetryCalloutJobTest.clsApex test class covering happy path, max retries, and circuit open scenarios

Dependencies

Package files