SfSkills Domains Docs GitHub

Home / data / cdc-data-sync-patterns

cdc-data-sync-patterns

Use this skill for the replication lifecycle of CDC-based data synchronization: day-0 full load, incremental delta processing, replay ID management, gap event fallback, and ordering/deduplication using transactionKey and sequenceNumber. NOT for CDC entity selection, channel config, or edition limits — use integration/change-data-capture-integration. NOT for checking whether the two systems actually match — use data/data-reconciliation-patterns.

Open the skill folder on GitHub

CDC Data Sync Patterns

Use this skill when designing or debugging the data replication lifecycle for CDC-based synchronization pipelines. It covers the full replication lifecycle: initial day-0 full load via Bulk API, incremental delta processing via CDC events, durable replay ID management, gap event fallback queries, and correct event ordering using transactionKey and sequenceNumber. This skill is distinct from integration/change-data-capture-integration, which covers CDC admin setup and channel configuration.

Trigger Scenarios

Before Starting

Gather this context before working on anything in this domain:

Core Concepts

CDC Event Retention and the 72-Hour Window

Salesforce retains Change Data Capture events for 72 hours (3 days). Events outside this window are permanently purged and cannot be replayed at any price. The retention window is the central constraint around which all recovery strategies must be designed.

A subscriber that reconnects within 72 hours of going offline can replay missed events using its stored replayId. A subscriber that was offline for more than 72 hours has no CDC-based recovery path and must perform a full Bulk API re-sync of the affected objects before resuming CDC.

ReplayId Values and Durable Subscription

The replayId controls where in the event stream a subscriber begins receiving events:

ReplayId ValueMeaning
-1Tip — receive only events published after subscribing (no catch-up)
-2Full catch-up — replay all events retained in the 72-hour window
<specific ID>Resume from a specific position; subscriber must persist this after each successful batch

Choosing the right value depends on subscriber state:

Critical: Salesforce maintains no per-subscriber cursor. The subscriber is solely responsible for persisting the replayId after each successfully processed event batch. Failure to persist it means the subscriber cannot resume from where it left off.

Gap Events: Fallback is Mandatory

When Salesforce cannot generate a complete change event — because the record change was too large (>1 MB), originated from a bulk operation that bypassed the application server, or an internal error occurred — it emits a gap event instead.

Gap event changeType values: GAP_CREATE, GAP_UPDATE, GAP_DELETE, GAP_UNDELETE, GAP_OVERFLOW.

Gap events include the ChangeEventHeader with recordIds but contain no field data in the event body. A subscriber that processes a gap event as if it were a normal change event will see empty or null field values and silently write corrupted data to the external datastore.

Correct gap event handling:

  1. Check changeType for the GAP_ prefix before processing any event.
  2. If a gap event is detected, extract the recordIds from ChangeEventHeader.
  3. Issue a REST API SOQL query for the current full record state using those IDs.
  4. Apply the queried record state to the external datastore.
  5. Use commitTimestamp to guard against overwriting a newer non-gap event that has already been applied.

Event Ordering: transactionKey and sequenceNumber

CDC events within a single transaction are delivered with the same transactionKey and incrementing sequenceNumber values. Across transactions, events are ordered by commitTimestamp.

Two scenarios where ordering matters:

Using event arrival time or subscriber processing time for ordering instead of transactionKey + sequenceNumber + commitTimestamp is a common source of data corruption in CDC replication pipelines.

Common Patterns

Pattern 1: Day-0 Full Load then Switch to CDC

When to use: Bootstrapping a new CDC replication pipeline for objects that already have existing records. CDC only publishes events for changes that occur after the subscription is set up — it does not backfill existing records.

How it works:

  1. Record the current replayId at the tip of the event stream before running the Bulk API load (use Pub/Sub API GetTopic to retrieve the latest replayId).
  2. Run a Bulk API 2.0 query job to extract all existing records from the target object.
  3. Load those records into the external datastore.
  4. Subscribe to the CDC topic using the replayId recorded in step 1. This ensures no events published during the Bulk API load are missed.
  5. Process incoming CDC events to apply incremental deltas.

Why not CDC alone from the start: CDC does not replay events older than 72 hours and has no mechanism to publish events for records that existed before the subscription started.

Pattern 2: Reconnect Within 72 Hours Using Persisted ReplayId

When to use: A subscriber restarts or reconnects after a brief outage (less than 72 hours) and needs to catch up without data loss.

How it works:

  1. On shutdown, ensure the last successfully processed replayId is written to durable external storage (database row, Redis key, etc.).
  2. On reconnect, read the stored replayId from external storage.
  3. Subscribe to the CDC topic using that specific replayId.
  4. Process all events from that position forward before switching back to normal streaming.

Why not -2 on reconnect: Using -2 replays the full 72-hour window, which can produce millions of events for active orgs. Using the stored specific replayId resumes from exactly where processing stopped.

Pattern 3: Post-72-Hour Outage Recovery via Full Bulk API Re-sync

When to use: A subscriber has been offline for more than 72 hours (e.g., infrastructure failure, planned maintenance that ran long). The stored replayId now points to events that are outside the retention window and cannot be replayed.

How it works:

  1. Detect that the stored replayId is stale (the Pub/Sub API returns an error indicating the replay ID is outside the retention window).
  2. Perform a full Bulk API 2.0 query job on each affected object to extract current record state.
  3. Upsert those records into the external datastore (using an external ID or record ID as the upsert key).
  4. Subscribe to the CDC topic at -1 (tip) to resume receiving new changes going forward.
  5. Store the new replayId from the first event received to re-establish a durable position.

Why not partial replay: There is no partial replay for events beyond the retention window. The only authoritative source of current record state after 72+ hours offline is a fresh Bulk API extract.

Decision Guidance

SituationRecommended ApproachReason
Brand-new CDC pipeline, existing recordsDay-0 Bulk API load, then subscribe at recorded tip replayIdCDC does not backfill pre-existing records
Subscriber reconnects < 72 hours after outageResume with persisted replayIdEfficient; replays only missed events
Subscriber reconnects > 72 hours after outageFull Bulk API re-sync, then subscribe at -1Retention window expired; replay impossible
First-ever connection, no historical data neededSubscribe at -1 (tip)No catch-up required
First-ever connection, want all retained eventsSubscribe at -2Gets up to 72 hours of history
changeType has GAP_ prefixExtract recordIds from header, query REST APIGap events have no field data
Multiple events for same record in one transactionOrder by sequenceNumber within same transactionKeyCorrect application order
Deduplicating events across retries or reconnectsUse transactionKey + sequenceNumber as idempotency keyUnique per change within a transaction
  1. Determine subscriber state — Is this a new deployment, a reconnect after a short outage, or a reconnect after a long outage (>72 hours)? The answer determines whether to run a Bulk API load, resume from a stored replayId, or trigger a full re-sync.
  2. Run day-0 load if required — For new pipelines: record the current tip replayId, execute a Bulk API 2.0 query job, load results to the external datastore, then subscribe at the recorded replayId to catch any events published during the load.
  3. Implement replay ID persistence — After each successfully processed batch of events, write the last replayId to durable external storage. Never rely on in-memory state alone. Include a write-confirm step before acknowledging event processing.
  4. Implement gap event detection and fallback — Before processing any event fields, check changeType for the GAP_ prefix. On detection: extract recordIds from ChangeEventHeader, query REST API for current field values, apply those values using commitTimestamp as a guard against stale overwrites.
  5. Implement ordering and deduplication — Within a transaction (transactionKey), apply events in ascending sequenceNumber order. Across transactions, use commitTimestamp for sequencing. Use transactionKey + sequenceNumber as an idempotency key to safely handle event redelivery.
  6. Test recovery scenarios — Explicitly test: subscriber restart within 72 hours (should resume from stored replayId), subscriber restart after simulated 72-hour gap (should trigger Bulk API re-sync), and gap event injection (should trigger REST fallback query).

Review Checklist

Salesforce-Specific Gotchas

  1. 72-hour retention is absolute and non-negotiable — Events purged after the retention window are permanently gone. There is no support case, add-on, or org setting that can extend the window or recover purged events. Design recovery procedures to assume this constraint.
  2. Gap events carry no field data — only recordIds — A GAP_UPDATE event has the same changeType field as any other change event, but the event body contains no field values. Subscribers that do not check for the GAP_ prefix before processing field data will write empty or null values to the external datastore silently.
  3. Salesforce has no per-subscriber cursor — Salesforce does not track where any individual subscriber is in the event stream. The subscriber must persist the replayId externally. Losing the replayId means the subscriber can only start at tip (-1) or replay the full retained window (-2) — both of which may cause data gaps or excessive event volume.
  4. transactionKey + sequenceNumber, not timestamp, for ordering — commitTimestamp is accurate at the transaction level but multiple events within a single transaction share the same timestamp. Using timestamp alone to order intra-transaction events produces non-deterministic results. Always use sequenceNumber within a shared transactionKey.
  5. Stale replayId gives an error, not silent behavior — If the subscriber provides a replayId that is outside the retention window, the Pub/Sub API returns an error rather than silently falling back to tip. Reconnection logic must handle this error explicitly and trigger the Bulk API re-sync path.

Output Artifacts

ArtifactDescription
Day-0 load designBulk API 2.0 query job configuration and timing relative to first CDC subscription
ReplayId persistence designStorage location, write timing, and recovery read procedure
Gap event handling logicDetection check and REST API fallback query design for GAP_* events
Ordering/deduplication designtransactionKey + sequenceNumber usage for correct event application order
Recovery decision treeFlowchart: reconnect < 72h → resume; reconnect > 72h → full re-sync

Official Sources Used

Package files