SfSkills Domains Docs GitHub

Home / data / data-archival-strategies

data-archival-strategies

Use when planning how to archive aging Salesforce records to reduce storage costs, maintain query performance, and meet retention policies.

Open the skill folder on GitHub

Data Archival Strategies

Use this skill when an org is approaching storage limits, experiencing query performance degradation from large object row counts, or needs to implement a formal data retention and archival policy. This skill does NOT cover migrating data to a new org (use data-migration-planning) or cleaning up duplicate records.

Before Starting

Gather this context before working on anything in this domain:

Core Concepts

Salesforce Storage Model

Salesforce separates storage into two buckets:

Both pools are recalculated asynchronously. Salesforce Help states that after adding or removing a large number of records or files, "the change in your org's storage usage isn't reflected immediately." An archival run that appears to have freed nothing has usually just not been recounted yet.

Storage alerts fire at 85% and 100% of your allocation. Exceeding data storage blocks new record inserts org-wide — a critical reliability risk.

Big Objects

Salesforce Big Objects (__b suffix) are a separate, horizontally scalable database that stores and manages massive amounts of data — consistently across 1 million, 100 million, or 1 billion records. They are available in Enterprise, Performance, Unlimited, and Developer Editions (Developer Edition is capped at 1 million records).

Key behavioral constraints:

ConstraintDetail
Immutable after writeStandard DML (insert, update, delete) does not apply. Use Database.insertImmediate() for writes. Updates = reinsert with same index values (upsert semantics). Deletion uses Database.deleteImmediate() or SOAP deleteByExample().
Index-only queriesSOQL must use fields that are part of the defined composite index, in order, without gaps. Arbitrary filter queries are not supported.
No triggers, flows, or processesAutomations do not fire on Big Object writes. Use Queueable or Batch Apex for any post-write processing.
No aggregate functions in SOQLCOUNT() and similar aggregates are not supported. Use Batch Apex to iterate and count manually.
No standard UIBig Objects do not appear in standard list views, reports, or dashboards. To report, extract a subset into a custom object via Bulk API query and report on that.
Idempotent writesReinserting a record with identical index values yields a single record, making retry-on-failure safe.
No encryption support for custom Big ObjectsData archived from an encrypted standard/custom object is stored as clear text. The standard FieldHistoryArchive Big Object DOES respect Shield Platform Encryption.
Mixed DML restriction in Apex testsTests cannot mix DML on sObjects and Big Objects in the same transaction. Use a mocking framework or manually roll back test data.

Field History and Field History Truncation

Salesforce tracks field history on standard and custom objects when Field History Tracking is enabled. History records are stored in a separate History object (e.g., AccountHistory, OpportunityFieldHistory) and are subject to an 18-month rolling retention window by default.

Recycle Bin and Soft Delete

When a record is deleted in Salesforce, it enters the Recycle Bin (soft delete):

Common Patterns

Mode 1: Design an Archival Strategy from Scratch

When to use: Org is growing rapidly, storage is projected to exceed limits within 6–12 months, or a data retention policy requires records older than N years to be moved off the main object.

How it works:

  1. Run Setup > Storage Usage to establish the current data storage and file storage baseline.
  2. Identify the top 3–5 objects by record count. Use SOQL aggregate queries: SELECT COUNT(Id) FROM Opportunity.
  3. For each candidate object, determine: (a) is the data still needed for real-time queries or reports? (b) what is the legal retention requirement?
  4. Choose an archival destination:
    • Big Object if the data must remain queryable from within Salesforce (compliance, audit dashboards) and the query pattern fits the indexed fields.
    • External storage (Heroku PostgreSQL, Amazon S3 via middleware, Data Cloud) if data is rarely accessed and cost-optimized storage is the priority.
    • Soft-delete pattern (IsArchived__c flag) if the data must remain in the same object for reporting but should be excluded from standard views.
  5. Design the archival job: use Batch Apex to read source records in pages of up to 2,000, write to the destination, then hard-delete source records via Database.deleteImmediate() or Bulk API hard delete.
  6. Schedule the archival job to run nightly or weekly depending on growth rate.

Why not archive manually: Manual export-and-delete via Data Loader creates recycle bin pollution that degrades query performance until the 15-day window expires. Batch Apex with hard delete reclaims storage immediately.

Mode 2: Audit Current Storage and Recommend Action

When to use: Storage alert has fired or query performance has degraded on a large object.

How it works:

  1. Check Setup > Storage Usage. Identify whether the pressure is data storage or file storage.
  2. For data storage pressure: run SELECT COUNT(Id) FROM RecycleBin equivalent — check Setup > Recycle Bin for volume. Empty the recycle bin first; this is the fastest win.
  3. Identify the top objects by count. Look for History objects (AccountHistory, CaseHistory) as hidden storage consumers.
  4. Check if field history tracking is enabled on high-volume objects for fields that change frequently (e.g., Stage, Status). Disabling tracking on low-value fields stops future accumulation.
  5. Recommend archival approach based on access pattern (see Decision Guidance table below).

Mode 3: Troubleshoot Storage Alert or Query Performance Degradation

When to use: Org has received a storage alert or users report slow list views and reports on large objects.

How it works:

  1. Empty the Recycle Bin immediately (Setup or Database.emptyRecycleBin()). Soft-deleted records degrade selectivity for query optimization.
  2. Identify if the alert is driven by file storage (Attachments, ContentVersion). If so, the archival strategy is different — files require external storage or Content Delivery Networks.
  3. For query performance: check if the slow query has selective filters on indexed fields. On objects with millions of records, non-selective filters cause full table scans.
  4. For data storage: run a batch archival job targeting the oldest records by CreatedDate that are beyond the retention window.

Decision Guidance

SituationRecommended ApproachReason
Data must remain queryable from Salesforce with known filter patternsBig Object with composite indexConsistent performance at billions of records; stays on-platform
Data is rarely accessed, cost is primary concernExternal storage (S3 / Heroku / Data Cloud)Lower cost per GB; no on-platform storage consumed
Records must stay on the parent object for reports but hidden from usersSoft-delete pattern (IsArchived__c boolean)Keeps records in-org, excludes from views, supports historical queries
Storage alert fired and recycle bin is fullEmpty recycle bin immediatelyFastest storage reclamation, no archival job required
Field history is consuming excessive storageDisable tracking on low-value fields; evaluate Shield for extended retentionField history growth is unbounded on high-churn fields
Compliance requires 7+ year history retentionSalesforce Shield Field Audit Trail + FieldHistoryArchive Big ObjectOnly supported path to extend beyond 18-month default
Large file/attachment storage pressureArchive files to external storage; use Salesforce Files Connect or custom middlewareFile storage limits are separate from data storage

Step-by-step instructions for an AI agent or practitioner activating this skill:

  1. Gather context — confirm the org edition, relevant objects, and current configuration state
  2. Review official sources — check the references in this skill's well-architected.md before making changes
  3. Implement or advise — apply the patterns from Core Concepts and Common Patterns sections above
  4. Validate — run the skill's checker script and verify against the Review Checklist below
  5. Document — record any deviations from standard patterns and update the template if needed

Review Checklist

Run through these before marking archival work complete:

Salesforce-Specific Gotchas

Non-obvious platform behaviors that cause real production problems:

  1. Big Object records cannot be updated or deleted via standard DML — update and delete DML statements throw an exception on Big Object records. Updates require reinserting with the same index values (upsert semantics). Deletion uses Database.deleteImmediate() which requires all index fields to be specified. Treating a Big Object like a standard object in Apex will cause runtime errors.
  2. Async SOQL was retired in Summer '23 — do not design an archive read path around it — The /services/data/vXX.0/async-queries/ endpoint and AsyncQueryJob no longer exist. This bites archival projects specifically and late: the write path into a Big Object (Database.insertImmediate, Bulk API) works fine, so ingestion runs for months before anyone attempts the first compliance read and finds there is no implemented way to get the data back. Per Salesforce Help 000394892: "You must use the Bulk API or batch Apex to query or report on custom Big Objects." Build and test the read path before you delete the source records.
  3. Recycle bin affects query optimizer selectivity — Even though soft-deleted records are excluded from normal SOQL results, they still participate in selectivity calculations used by the query optimizer. A full Recycle Bin on a large object can cause full table scans and slow reports, even if the table appears smaller to users. Empty the Recycle Bin regularly.
  4. Field history on archived records is not archived automatically — When you archive (or delete) parent records, the associated History object rows (AccountHistory, CaseHistory, etc.) are NOT deleted automatically. They remain in the History object and continue to count against data storage until the 18-month automatic truncation window expires.

Output Artifacts

ArtifactDescription
Archival strategy recommendationDecision between Big Object, external storage, or soft-delete pattern with rationale
Archival implementation planSequenced Batch Apex job design, index definition, and schedule
Storage reclamation estimateProjected GB recovered based on record counts and average record size
Recycle bin and soft-delete guidanceSteps to empty recycle bin and implement IsArchived__c pattern

Package files