SfSkills Domains Docs GitHub

Home / integration / change-data-capture-integration

change-data-capture-integration

Use this skill when setting up CDC to stream Salesforce record changes to external systems via CometD or Pub/Sub API. Covers change event channels, entity selection, event structure, replay ID management, gap events, and subscriber management. NOT for Apex CDC trigger subscribers (see apex/platform-events-apex). NOT for publishing custom business events (use integration/platform-events-integration).

Open the skill folder on GitHub

Change Data Capture Integration

Use this skill when an external system needs to receive near-real-time Salesforce record changes without polling. CDC publishes structured change events to a durable event bus that external subscribers consume via Pub/Sub API (gRPC) or the CometD Streaming API. This skill covers the full lifecycle: entity selection, channel topology, event structure, replay, gap handling, and subscriber operations.

Before Starting

Gather this context before working on anything in this domain:

Core Concepts

Change Event Structure

Every CDC message wraps a ChangeEventHeader object alongside the record fields. Key header fields:

FieldTypeDescription
entityNamestringAPI name of the changed object (e.g., Account)
changeTypeenumCREATE, UPDATE, DELETE, UNDELETE, or GAP_* prefixed variants
changedFieldsstring[]Fields changed in an UPDATE (Pub/Sub API only; not in CometD)
nulledFieldsstring[]Fields explicitly set to null in an UPDATE (Pub/Sub API only)
diffFieldsstring[]Fields that changed due to formula or roll-up recalculation (Pub/Sub API only)
recordIdsstring[]Record IDs affected; multiple IDs appear when gap events are merged
transactionKeystringUnique transaction identifier; use to deduplicate or correlate events
sequenceNumberintOrder of change within a transaction
commitTimestamplongEpoch milliseconds when the change committed

changedFields, nulledFields, and diffFields are not available in CometD clients — they require Pub/Sub API or Apex triggers. This is a critical architectural difference.

Enabling CDC and Entity Selection

CDC is enabled per object in Setup > Integrations > Change Data Capture. This page controls entity selections for the default ChangeEvents standard channel only. Custom channels are managed separately via PlatformEventChannel and PlatformEventChannelMember metadata types.

Default entity selection limit is 5 objects (standard + custom combined) across all channels and all editions. If the same entity is selected in multiple channels, it counts once. Exceeding 5 requires purchasing the Change Data Capture add-on.

Custom channels are created with Metadata API (PlatformEventChannel type) or Tooling API and support up to 100 custom channels. Selections made in PlatformEventChannelMember for custom channels are NOT reflected in the Setup UI — query Tooling API to audit selections.

Change Event Channels

Two channel types exist:

Custom channels support event enrichment: you can add fields to the change event payload that were not part of the originating DML change, allowing downstream systems to avoid follow-up queries.

External Subscription Protocols

Pub/Sub API (gRPC) — preferred for new integrations:

CometD (Streaming API) — legacy, still supported:

Replay IDs and Durable Subscription

Change events are retained for 72 hours (3 days). Subscribers can replay events within this window using replay IDs:

Replay ID ValueMeaning
-1Tip — receive only events published after subscribing (no history)
-2All retained — replay all events stored in the current 72-hour window
<specific ID>Resume from a specific position; subscriber must persist this value after each batch

Durable subscription requires the subscriber to persist the last successfully processed replay ID to external state (database, cache, or file). On reconnect, pass the stored replay ID to resume without gaps.

Replay ID storage responsibility lies entirely with the subscriber. Salesforce does not maintain per-subscriber cursors — if a subscriber reconnects without a stored replay ID, it can only start at tip (-1) or replay all retained events (-2).

Gap Events And Overflow Events — Two Different Failures

Gap events (GAP_CREATE, GAP_UPDATE, GAP_DELETE, GAP_UNDELETE) are sent when Salesforce cannot generate a full change event: "The change event size exceeds the maximum 1 MB message size", certain field-type conversions, internal errors, or "Changes that occur outside the application server transaction and are applied directly in the database." They carry real recordIds but no field data — detect the GAP_ prefix, re-query current state by recordIds through REST, and reconcile against commitTimestamp so a newer non-gap event is not overwritten.

GAP_OVERFLOW is not one of these and must not share the code path. It is emitted when a single transaction exceeds 100,000 changes: "The first 100,000 changes generate change events. The set of changes beyond that amount generates one overflow event for each entity type." The weighting is asymmetric — "A record creation, deletion, or undeletion counts as one change toward the threshold", but "In a record update, each field change counts toward the overflow threshold", so a 3-field update across 34,000 records crosses the line. Typical causes: a cascade delete of accounts with many child opportunities and activities, or a recurring event with hundreds of occurrences and attendees.

The overflow payload carries nothing usable: all data fields are null, changedFields/nulledFields/diffFields are empty, and "The record ID for the change is always set to 000000000000000AAA, which is the empty record ID." So there is no per-record recovery path — unlike the other GAP_ types you cannot re-query what you missed. The fix is upstream (batch the originating operation below 100,000 changes) or a scheduled reconciliation job, never consumer scaling or replay tuning.

Common Patterns

Pattern 1: Single External Subscriber — Default Channel with Durable Replay

When to use: One middleware system (e.g., MuleSoft, Boomi) needs all CDC changes for a small set of objects.

How it works:

  1. Enable the target objects in Setup > Integrations > Change Data Capture (max 5 entities without add-on).
  2. Subscribe to /data/ChangeEvents or the per-entity channel /data/AccountChangeEvent.
  3. Use replay ID -2 on first connection to catch up, then switch to persisting the last processed replay ID.
  4. Store the replay ID after each successful batch commit to external state.
  5. On gap event, fetch current record via REST API using the record ID in the header.

Why not just polling: Polling via REST API misses field-level delta information and cannot detect deletes or undeletes. CDC provides explicit changeType and the exact set of changed fields.

Pattern 2: Multiple Subscribers with Isolated Custom Channels

When to use: Two or more downstream systems need CDC but for different entity subsets, or with different event enrichment needs.

How it works:

  1. Create a custom PlatformEventChannel per subscriber group via Metadata API.
  2. Add PlatformEventChannelMember records for the relevant entities per channel.
  3. Configure event enrichment on channel members where subscribers need additional fields beyond the changed ones.
  4. Each subscriber connects to its specific channel path, e.g., /data/ERP_Sync__chn.
  5. Each subscriber manages its own replay ID independently.

Why not the default channel: it broadcasts all selected entities to all subscribers; custom channels filter server-side before delivery.

Pattern 3: CDC → Kafka Bridge for Fan-Out

When to use: Multiple downstream consumers exist and Kafka is the enterprise event backbone.

How it works:

  1. One dedicated integration layer (MuleSoft CDC connector, custom Pub/Sub API gRPC client, or Kafka Connect Salesforce connector) subscribes to Salesforce CDC via Pub/Sub API.
  2. That layer publishes to Kafka topics mapped per entity (account-changes, opportunity-changes).
  3. Kafka consumers subscribe independently; the CDC-to-Kafka bridge manages Salesforce replay ID in its own persistent store.
  4. Kafka provides its own offset management and fan-out — Salesforce event delivery allocation is consumed only by the single bridge subscriber.

Why not multiple Salesforce subscribers: each additional external subscriber counts independently against the delivery allocation; one bridge protects it.

Decision Guidance

SituationRecommended ApproachReason
One integration system needs all CDC changesDefault /data/ChangeEvents channel, CometD or Pub/Sub APISimple; no custom channel overhead
Multiple systems need different entitiesCustom channels per subscriberServer-side filtering; preserves delivery allocation
Need changedFields delta in external systemPub/Sub API (gRPC)CometD does not surface changedFields
Existing MuleSoft or Boomi middlewareCometD via EMP Connector or platform connectorEstablished connector support; migration to Pub/Sub API optional
Tracking > 5 entity typesPurchase CDC add-on or evaluate if all entities truly need CDCDefault limit is 5; add-on removes the cap
Subscriber was offline > 72 hoursFull re-sync via Bulk API 2.0 then re-subscribe at tipEvents beyond retention window are permanently purged
Need enriched fields in event payloadCustom channel with event enrichment on PlatformEventChannelMemberAvoids follow-up REST queries per event
Kafka is the enterprise busSingle Pub/Sub API bridge to KafkaPreserves delivery allocation; Kafka provides fan-out
  1. Confirm scope and limits — Count the entities needed. Verify org Edition and whether the CDC add-on is licensed. Confirm subscriber technology (Pub/Sub API or CometD).
  2. Enable entities and configure channels — For up to 5 entities, enable via Setup > Change Data Capture. For custom channels or more entities (with add-on), create PlatformEventChannel and PlatformEventChannelMember via Metadata API deployment.
  3. Design replay ID strategy — Decide where the subscriber persists the last replay ID (external DB, Redis, config file). Decide initial replay behavior: -2 for full catch-up or -1 for tip-only.
  4. Implement subscriber — For Pub/Sub API: implement a gRPC Subscribe call with FetchRequest carrying the stored replay ID. For CometD: configure the EMP Connector or middleware connector with the channel path and replay extension.
  5. Implement gap event handling — Add conditional logic that detects GAP_ prefixed changeType values, marks records dirty, and fetches current state from REST API using recordIds.
  6. Test replay and reconnection — Validate reconnect behavior by stopping the subscriber, waiting, and replaying using the stored replay ID. Confirm the subscriber resumes from the correct position.
  7. Monitor delivery allocation usage — Check Setup > Event Monitoring or REST API to track daily delivery usage against the org's allocation. Alert before the limit is reached.

Review Checklist

Salesforce-Specific Gotchas

  1. changedFields is absent in CometD — changedFields, nulledFields and diffFields come only from Pub/Sub API or Apex triggers. CometD delivers the full field set, so diff logic there must compare against stored state.
  2. Gap events contain no changed field data — the subscriber gets only recordIds and the gap changeType. Check the GAP_ prefix before touching field values, and branch GAP_OVERFLOW separately (see Core Concepts) — it carries no usable record IDs either.
  3. Entity selections in custom channels are invisible in Setup UI — the CDC Setup page reflects only the default ChangeEvents channel, so a Setup-based audit undercounts event volume. Use SELECT QualifiedApiName FROM PlatformEventChannelMember (Tooling API).
  4. Event delivery allocation is per-subscriber, cumulative — two subscribers on the same 20,000-event stream consume 40,000 of the allocation. See references/gotchas.md Gotcha 4 for which subscriber types consume it and the exact disconnect strings.
  5. 72-hour retention is hard — a subscriber offline longer than that has a gap no replay can close. Recovery is a full Bulk API re-sync followed by re-subscribing at tip.
  6. Formula fields and roll-ups do not generate CDC events — CDC captures explicit DML on stored values. Formula recalculations and roll-up updates driven by child records produce no parent change event, so formula state tracked via CDC silently drifts.

Output Artifacts

ArtifactDescription
Entity selection listObjects enabled for CDC and the channel they are on
Channel subscription pathCometD or Pub/Sub API topic path per subscriber
Replay ID storage designWhere and how the subscriber persists the last replay ID
Gap event handling logicCode or flow logic for detecting and recovering from gap events
Delivery allocation monitoring planSetup dashboard or REST API query for daily usage tracking

Package files