SfSkills Domains Docs GitHub

Home / integration / real-time-vs-batch-integration

real-time-vs-batch-integration

When to use this skill: choosing between real-time (synchronous callouts, Platform Events, CDC, Pub/Sub API) and batch (Bulk API 2.0, scheduled ETL) integration patterns. Trigger keywords: should I use real-time or batch, how to sync high-volume data, when to use Platform Events vs Bulk API, integration latency vs volume tradeoff. NOT for choosing an ETL tool — use data/etl-vs-api-data-patterns. NOT for Batch Apex internals — use apex/batch-apex-patterns.

Open the skill folder on GitHub

Real-Time vs Batch Integration

This skill activates when a practitioner must choose between real-time and batch integration approaches in Salesforce. It provides a decision framework grounded in the Salesforce Integration Patterns guide, covering synchronous callouts, Platform Events, Change Data Capture, Pub/Sub API, and Bulk API 2.0, and produces a defensible pattern recommendation with limits and tradeoffs.

Before Starting

Gather this context before working on anything in this domain:

ContextWhat to gather
VolumeRecords changing per hour at peak. Above ~2,000 records per transaction window is outside Apex callout capacity and requires an async or bulk mechanism.
Latency toleranceSub-second acknowledgement, near-real-time (seconds to minutes), or nightly window acceptable?
Transactionality requirementSynchronous callouts participate in the Salesforce DML transaction and can be rolled back; CDC events and Platform Events are fire-and-forget once published.
Most common wrong assumptionTreating Platform Events and synchronous Apex callouts as equivalent "real-time" options — callouts are synchronous (120s timeout, 100/tx limit); Platform Events are async with up to 72-hour replay but no rollback guarantee.

Core Concepts

Synchronous Callout (Remote Process Invocation — Request and Reply)

An Apex HTTP or SOAP callout fires inline within a DML transaction. The calling transaction blocks until the external system responds or the 120-second timeout elapses. Governor limits: 100 callouts per transaction, 120-second timeout per callout, and callouts cannot be made from triggers that already have DML pending (the "callout after DML" restriction). This pattern is correct when: volume is low (single record or small sets), the caller needs an immediate response to make a commit/rollback decision, and the external system can respond within the timeout.

Asynchronous Event-Driven (Platform Events and CDC/Pub/Sub API)

Platform Events are schema-defined Salesforce objects published via EventBus.publish() or DML insert. Subscribers receive events asynchronously via Apex triggers, CometD streaming, or Pub/Sub API gRPC. Key behavioral facts:

Bulk API 2.0 (Batch Data Load)

Bulk API 2.0 is the correct mechanism for loading or extracting large volumes of records — Salesforce recommends it for datasets above 2,000 records. Jobs are asynchronous and processed in the background. Key behavioral facts:

Hybrid Approach

Many production integrations combine patterns: a Platform Event or CDC subscriber receives a change notification in near-real-time but delegates heavy record processing to a queued Bulk API job or a scheduled batch. This decouples the event detection latency from the volume constraint.

Common Patterns

Pattern 1: Trigger-Based Sync for Low-Volume Transactional Records

When to use: Individual record creates/updates that require immediate confirmation from an external system (e.g., creating a payment authorization, validating an address, checking inventory for a single order line).

How it works:

  1. An Apex trigger or process fires on record insert/update.
  2. A @future(callout=true) method or a queueable with callout support issues an HTTP request to the external endpoint.
  3. The response updates a status field on the Salesforce record in a follow-on DML.
  4. Retry logic is handled by re-queuing or using a Platform Event to signal failure.

Why not Platform Events here: The caller needs a synchronous confirmation to decide whether to commit or surface an error to the user. Platform Events do not provide a reply.

Limits to enforce: Verify fewer than 100 callouts will fire per transaction. Use @future or Queueable to avoid the "callout after DML" error. Set a Named Credential — never hardcode endpoint URLs or credentials.

Pattern 2: Event-Driven Near-Real-Time Sync via Platform Events

When to use: Volume is moderate (hundreds to low thousands of events per hour), latency must be under a few minutes, and the business does not require transactional rollback on the receiving side (e.g., broadcasting order status changes to a fulfillment system).

How it works:

  1. A trigger or Flow publishes a Platform Event on record change.
  2. An external subscriber (MuleSoft, custom app, AWS Lambda) listens on CometD or Pub/Sub API gRPC.
  3. The subscriber processes and writes to the target system.
  4. Replay IDs allow the subscriber to recover from downtime without data loss within the 72-hour window.

Why not synchronous callouts: External system downtime or latency spikes would block the Salesforce transaction and risk timeout errors at scale.

Pattern 3: Bulk API 2.0 Scheduled Batch Sync

When to use: Volume exceeds 2,000 records per sync cycle, latency tolerance is hours (e.g., nightly ERP sync, daily customer master reconciliation), and eventual consistency is acceptable.

How it works:

  1. External ETL tool or scheduled job queries Salesforce (SOQL query job via Bulk API 2.0) or receives a data extract.
  2. Target system data is transformed and staged.
  3. A Bulk API 2.0 ingest job upserts records using an external ID field.
  4. Job status is polled; failed records are captured in the results CSV and re-queued for the next cycle.
  5. Schedule jobs during off-peak hours to avoid resource contention with interactive users.

Why not Platform Events for this volume: At sustained thousands-per-second rates, Platform Events have throughput limits (standard: 250,000 events/day for most editions) and the event bus is not a reliable high-volume ETL channel.

Decision Guidance

SituationRecommended ApproachReason
<100 records, immediate response needed, user-facingSynchronous callout (Apex HTTP/SOAP)Caller needs reply to commit or surface error; volume is within callout limits
Hundreds to low-thousands of events/hour, latency < 5 min, no rollback neededPlatform Events + external subscriberAsync, durable 72-hr replay, decouples systems
External system needs to react to Salesforce record changes automaticallyChange Data Capture + Pub/Sub APIPlatform-generated, field-level diff, 3-day replay, no custom publish code
>2,000 records per cycle, hours latency acceptableBulk API 2.0 scheduled jobPurpose-built for volume; callouts and Platform Events cannot handle this throughput reliably
High event volume + need for immediate detection but deferred processingHybrid: CDC/Platform Event triggers a queued batchDecouples detection latency from processing volume
Bidirectional sync with conflict resolutionMiddleware (MuleSoft) + Bulk API 2.0 or Platform EventsSalesforce-native patterns lack built-in conflict resolution; middleware handles transformation

Step-by-step instructions for an AI agent or practitioner working on this task:

  1. Establish volume and latency requirements: ask the stakeholder for peak records-per-hour and the maximum acceptable delay from source change to destination visibility. These two axes determine the viable pattern set before any implementation begins.
  2. Confirm transactionality needs: determine whether the integration must roll back Salesforce data if the external system rejects the payload. If yes, only a synchronous callout participates in the transaction; all async patterns are eventually consistent.
  3. Apply the decision table: map volume + latency + transactionality to the correct Salesforce mechanism. Flag if volume exceeds callout limits (100/transaction) or Platform Event daily allocations (250,000 standard).
  4. Check governor limits for the chosen pattern: for callouts verify the 120s timeout and 100/transaction limit; for Platform Events confirm daily event allocation; for Bulk API 2.0 confirm the 150M records/24hr limit and plan the off-peak schedule.
  5. Design the error and retry path: synchronous callouts need try/catch + status field; Platform Event subscribers need a dead-letter queue or replay-ID restart strategy; Bulk API 2.0 jobs need results-CSV processing to re-queue failed rows.
  6. Validate with a load estimate: calculate peak event or record rate against the chosen mechanism's limits; if headroom is under 30%, escalate to the next tier or add a hybrid buffer.
  7. Document the chosen pattern and rationale in the integration design using the template in templates/real-time-vs-batch-integration-template.md, including replay window, retry approach, and failure notification path.

Review Checklist

Run through these before marking work in this area complete:

Salesforce-Specific Gotchas

Non-obvious platform behaviors that cause real production problems:

  1. Apex callout 120-second timeout is absolute — there is no way to extend it. If the external system is slow or the payload is large, a synchronous callout will time out even at low volume. The fix is to move to a Queueable or @future callout that fires outside the user transaction, decoupling user-facing latency from external system performance.
  2. "Callout after DML" restriction — if a trigger performs DML before the callout code runs, the callout is blocked with System.CalloutException: You have uncommitted work pending. This forces callouts to @future or Queueable, which means they cannot roll back the DML they follow.
  3. Platform Events do not roll back on transaction rollback by default — as of Spring '25, publishBehavior defaults to PUBLISH_IMMEDIATELY for standard Platform Events; events already written to the bus are not retracted if the publishing transaction fails. Use PHASE_AFTER_COMMIT publish behavior when the event should only fire on successful commit.
  4. CDC 72-hour / 3-day replay window is a hard ceiling — if a subscriber is down for longer than the replay window, events are gone. Design monitoring to alert if a subscriber has not polled within 48 hours so the window does not silently expire.
  5. Bulk API 2.0 has no cross-batch atomicity — a job processing 500,000 records in multiple internal batches will commit each batch independently. If batch 3 of 10 fails, batches 1–2 are already committed and cannot be rolled back. Design idempotent upserts using external IDs so retries are safe.

Output Artifacts

ArtifactDescription
Integration pattern recommendationWritten rationale mapping volume/latency/transactionality to the chosen Salesforce mechanism
Decision tablePopulated version of the decision table in this skill with project-specific values
Governor limit validationCalculations showing chosen pattern is within limits at projected peak
Error handling designRetry approach, dead-letter strategy, and monitoring thresholds per integration leg

Package files