SfSkills Domains Docs GitHub

Home / integration / data-cloud-ingestion-api

data-cloud-ingestion-api

Use when implementing or troubleshooting the Salesforce Data Cloud Ingestion API — covers streaming ingestion (near-real-time micro-batches), bulk ingestion (CSV-based full-replace jobs), schema management in OpenAPI. NOT for standard Salesforce Bulk API, CRM Analytics data import, or Data Cloud data stre — use integration/bulk-api-2-patterns.

Open the skill folder on GitHub

Data Cloud Ingestion API

This skill activates when a practitioner needs to implement or troubleshoot the Salesforce Data Cloud Ingestion API — the REST API for pushing external data into Data Cloud from outside the Salesforce ecosystem. It covers both interaction patterns (streaming micro-batches and bulk CSV jobs), the irreversible nature of schema deployments, the Connected App setup requirements, and error handling for both modes.

Before Starting

Gather this context before working on anything in this domain:

Core Concepts

Streaming vs. Bulk Ingestion

Streaming Ingestion:

Bulk Ingestion:

Schema Management (OpenAPI 3.0.x YAML)

Ingestion API schemas are defined in OpenAPI 3.0.x YAML and uploaded to Data Cloud at registration time. Critical constraints:

Authentication: Connected App with cdp_ingest_api Scope

Ingestion API calls require OAuth 2.0 via a Connected App configured with cdp_ingest_api OAuth scope:

  1. Create a Connected App in Setup > App Manager
  2. Add OAuth scope: cdp_ingest_api (Data Cloud Ingest API)
  3. Configure JWT Bearer Token or Client Credentials flow for server-to-server auth
  4. Register the data source in Data Cloud Setup > Data Stream > Ingestion API using the Connected App

Common Patterns

Pattern 1: Streaming Ingestion for Real-Time Events

When to use: External system generates events (web clickstream, mobile app events, IoT) that must appear in Data Cloud within minutes.

How it works:

  1. Configure Connected App with cdp_ingest_api scope
  2. Define OpenAPI 3.0.x schema for the event object
  3. Register Ingestion API connector in Data Cloud Setup
  4. External system calls streaming endpoint with JSON payload
  5. Data Cloud processes micro-batch within approximately 3 minutes
  6. Use validate endpoint during development to test payload format without writing data

Pattern 2: Bulk Ingestion for Nightly Snapshot

When to use: External data warehouse exports a full snapshot nightly. The snapshot replaces the previous Data Cloud record set for that object.

How it works:

  1. Export data to CSV (UTF-8, comma-delimited, ≤150 MB per file)
  2. Create bulk job via POST .../jobs
  3. Upload CSV files (≤100 files per job)
  4. Close job to trigger processing
  5. Poll job status endpoint until complete
  6. Validate via Data Cloud Query API to confirm record count

Decision Guidance

SituationRecommended ApproachReason
Real-time eventsStreaming IngestionProcessed every ~3 minutes; fire-and-forget
Nightly full snapshotBulk Ingestion (CSV, 150 MB/file max)Full-replace semantics; handles large volumes
Partial record updateStreaming with upsert semanticsBulk is full-replace only
Schema testing in developmentStreaming validate endpointValidates structure without writing data
Schema field removal neededArchitecture review — create new object versionField removal is not supported post-deployment
  1. Design schema before deployment — Define all fields, types, and object categories in OpenAPI 3.0.x YAML. Confirm engagement-category objects include a DateTime field. Review schema with all stakeholders before deployment — post-deployment changes are additive-only.
  2. Configure Connected App — Create a Connected App with cdp_ingest_api scope. Test OAuth token acquisition before schema registration.
  3. Register Ingestion API connector in Data Cloud Setup — Navigate to Data Cloud Setup > Data Streams > New > Ingestion API. Select the Connected App and upload the schema.
  4. Select ingestion mode — For real-time events: streaming endpoint. For nightly snapshots: bulk job pattern.
  5. Implement error handling — Streaming: monitor Data Cloud error logs for processing failures. Bulk: poll job status until complete; errors appear in job status response.
  6. Use streaming validate endpoint during development — Test payload format without writing data to Data Cloud.
  7. Validate records landed — After streaming or bulk job, query Data Cloud Query API with ANSI SQL to confirm record counts.

Review Checklist

Salesforce-Specific Gotchas

  1. Schema changes are largely irreversible after deployment — After a schema is deployed, you cannot remove fields, change data types, or delete objects. Adding fields is the only supported change. An incorrectly designed schema requires creating a new object or living with the bad field indefinitely.
  2. Bulk ingestion uses full-replace semantics — A bulk job replaces the entire dataset for the object. Running a partial file deletes all records not in that file. Bulk is for full snapshots, not incremental updates.
  3. Engagement-category objects require a DateTime field — Schemas for Engagement-type objects must include a field with type: string, format: date-time. Omitting it fails schema validation at registration time.

Output Artifacts

ArtifactDescription
OpenAPI 3.0.x schema YAMLComplete schema file for all ingested objects
Connected App configurationOAuth scope requirements and authentication flow documentation
Ingestion mode designStreaming vs. bulk decision with implementation specification
Error handling runbookStreaming error log monitoring and bulk job status polling

Package files