SfSkills Domains Docs GitHub

Home / data / data-quality-and-governance

data-quality-and-governance

Use this skill when designing data quality gates, auditing data quality posture, troubleshooting validation failures, or implementing governance controls (field history, GDPR erasure, data classification, duplicate strategy).

Open the skill folder on GitHub

Data Quality and Governance

This skill activates when a practitioner needs to design or audit data quality controls, enforce compliance requirements, or troubleshoot failures in validation, deduplication, or audit trail configuration across a Salesforce org. It covers the full governance stack: validation rules, duplicate management strategy, field history, GDPR/retention patterns, data classification, and Shield-layer controls.

Before Starting

Gather this context before working on anything in this domain:

Core Concepts

Validation Rules as Data Gates

Validation rules are formula-based constraints that prevent a record save when the formula evaluates to true. They are the primary declarative mechanism for enforcing data quality at the point of entry.

Use cases:

Custom Permission-based bypass: For integrations and data migrations that must bypass validation rules legitimately, the pattern is:

  1. Create a Custom Permission (e.g., Bypass_Validation_Rules).
  2. Wrap every validation rule formula with AND(NOT($Permission.Bypass_Validation_Rules), <original formula>).
  3. Assign the Custom Permission to a Permission Set used only by integration users.

This is preferable to System Administrator profile bypasses (which are role-based and too broad) and to $Profile.Name checks (which are brittle and hard to audit).

Duplicate Management Strategy

Salesforce Duplicate Management has two components: Matching Rules (define what constitutes a duplicate) and Duplicate Rules (define what to do when a match is found).

Matching Rules:

Duplicate Rules — alert vs block:

When standard Duplicate Rules are not enough:

Apex and Bulk API bypass (critical):

Field History as Audit Trail

Field history tracking records every change to a tracked field — who changed it, when, from what value, to what value. Changes are stored in the associated XxxHistory sObject (e.g., AccountHistory, OpportunityFieldHistory).

Key limits and behaviors:

Querying field history:

SELECT Field, OldValue, NewValue, CreatedById, CreatedDate
FROM AccountHistory
WHERE AccountId = '001...'
ORDER BY CreatedDate DESC

GDPR and Data Retention

Right to erasure: Salesforce recommends data anonymization rather than hard deletion. Hard deletion of a Contact or Account that has related records (Opportunities, Cases, Activities) will fail due to referential integrity or cause orphaned records. The recommended pattern is:

  1. Receive erasure request.
  2. Replace PII field values with non-reversible tokens (e.g., ERASED-<UniqueId> or a UUID).
  3. Clear or tokenize: First Name, Last Name, Email, Phone, Address fields.
  4. Soft-delete the record if no related records exist, or leave it with tokenized values if related records must be preserved.
  5. Log the erasure event in a custom Data_Erasure_Log__c object for compliance audit.

Retention periods: Salesforce does not enforce custom data retention periods natively. Implement retention enforcement via:

Data classification using field attributes: Salesforce metadata supports the following attributes on field definitions (available in Enterprise Edition and above with Data Classification metadata):

Einstein Data Detect (Shield): Available with the Salesforce Shield add-on. Scans existing org data to automatically identify fields that contain PII patterns (email addresses, SSNs, phone numbers, credit card numbers) and recommends classification. Use this to bootstrap a PII field inventory on an existing org.

Salesforce Shield Platform Encryption: Encrypts field values at rest using Salesforce-managed or customer-managed keys (Bring Your Own Key / BYOK). Encrypts: specific standard fields and all custom fields designated as Encrypted. Key behaviors:

Common Patterns

Mode 1: Designing Data Quality Gates for a New Org or Object

When to use: Starting fresh — new object, new org, or onboarding a new business process. Goal is to build quality controls before bad data accumulates.

How it works:

  1. Identify all fields that have conditional-required or format constraints. Map them to validation rule formulas before any record is created.
  2. Implement Custom Permission bypass from day one. Do not wait for an integration to complain.
  3. Select the 20 most audit-critical fields for field history tracking. Prioritize: status/stage fields, owner fields, financial amount fields, compliance-relevant date fields.
  4. Set Data Sensitivity Level and Compliance Categorization on all PII fields in field metadata.
  5. Create a Duplicate Rule in Block mode for the primary object (Account, Contact) from the start. Alert mode can be used initially but should be upgraded to Block after a validation period.
  6. Define a Retention_Expires__c date formula or trigger-set field for any object subject to a data retention policy.

Why not skip it: Retrofitting validation rules into a mature org is painful because existing data violates new rules. Adding field history after millions of records exist provides no history of past changes. Starting clean is dramatically cheaper.

Mode 2: Auditing Existing Data Quality Posture

When to use: Taking over an existing org, preparing for a compliance audit, or responding to a data quality incident.

How it works:

  1. Run scripts/check_data_quality.py --manifest-dir <metadata_dir> against the unpackaged org metadata to identify: validation rules without Custom Permission bypass, objects with fewer than the expected number of history-tracked fields, Duplicate Rules in alert-only mode on critical objects.
  2. Export the field inventory and cross-reference against the PII field register. Identify unclassified fields.
  3. Query XxxHistory objects for your top 5 critical objects to confirm history is actually capturing changes (a zero-row result on an active object is a red flag).
  4. Review Duplicate Record Sets (DuplicateRecordSet and DuplicateRecordItem objects) to understand the current duplicate backlog.
  5. If Shield is licensed, run Einstein Data Detect to surface unregistered PII.

Output: A gap report covering: missing bypass conditions, unclassified PII fields, history tracking gaps, duplicate rule coverage gaps.

Mode 3: Troubleshooting Data Quality Failures

When to use: A specific validation rule is firing unexpectedly, field history is not capturing changes, or duplicates are getting through.

Decision tree:

SymptomLikely CauseFix
Validation rule fires during migration loadNo bypass mechanism; rule fires on API updatesAdd Custom Permission bypass; grant to integration user
Field history not recording changeField type not trackable (formula, encrypted, auto-number)Switch to a trackable field type or use a trigger to log changes manually
Field history not recording first valuePlatform behavior — first value on create is not loggedAccepted behavior; use Apex trigger on insert to log initial value to custom audit object if needed
Duplicate rule not blocking in ApexDMLOptions.DuplicateRuleHeader not setSet dmlOptions.DuplicateRuleHeader.allowSave = false before DML
Duplicate rule not blocking in Bulk API 2.0Bulk API 2.0 cannot trigger Duplicate RulesRun deduplication as a separate pre-load step
Duplicate rule in alert mode letting duplicates throughRule is configured as Alert, not BlockChange rule action to Block; communicate change to users

Decision Guidance

SituationRecommended ApproachReason
Field required only under specific conditionsValidation rule with conditional formulaRequired field setting is binary — cannot express conditional requirements
Integration needs to bypass validation rulesCustom Permission + NOT($Permission.X) wrapProfile-based or user-based bypasses are too broad and non-auditable
New org, need to block duplicates from day oneDuplicate Rule in Block mode + standard Matching RuleAlert mode creates false confidence and duplicate backlog immediately
Large-volume deduplication of existing recordsThird-party dedupe app (Cloudingo, DemandTools)Standard Duplicate Rules do not run retroactively
Need 10-year audit trailShield Field Audit TrailStandard field history only retains 18 months
GDPR erasure request receivedAnonymize PII fields (tokenize), do not hard deleteHard delete breaks referential integrity on related records
Want to discover PII fields in existing orgEinstein Data Detect (requires Shield)Manual field inventory misses fields; automated scanning is more complete
Need to encrypt PII at restShield Platform EncryptionField-level masking does not encrypt stored values
Compliance audit requires field classificationSet Data Sensitivity Level + Compliance Categorization on field metadataNative metadata attributes; no custom objects required

Step-by-step instructions for an AI agent or practitioner activating this skill:

  1. Gather context — confirm the org edition, relevant objects, and current configuration state
  2. Review official sources — check the references in this skill's well-architected.md before making changes
  3. Implement or advise — apply the patterns from Core Concepts and Common Patterns sections above
  4. Validate — run the skill's checker script and verify against the Review Checklist below
  5. Document — record any deviations from standard patterns and update the template if needed

Review Checklist

Run through these before marking data quality or governance work complete:

Salesforce-Specific Gotchas

Non-obvious platform behaviors that cause real production problems:

  1. Validation rules fire on every API save — There is no "UI only" mode for a validation rule. Every REST API call, Apex DML operation, Bulk API job, and Flow-initiated save that modifies the fields in scope will trigger the rule. Data migration jobs that skip bypass setup fail at scale.
  2. Field history does not capture the initial record creation value — The first entry in XxxHistory appears only after the field is edited after creation. If a record is created with Status = New and never edited, there is no history row. For compliance use cases where the initial value is audit-critical, a trigger on insert must write to a custom audit object.
  3. Duplicate Rules are skipped in Apex DML by default — Inserting or updating records via Apex bypasses Duplicate Rules unless Database.DMLOptions.DuplicateRuleHeader.allowSave = false is set before the operation. This is not a widely known default and causes silent data quality failures in orgs that rely on Duplicate Rules for integrity.
  4. Bulk API 2.0 cannot trigger Duplicate Rules at all — Unlike Bulk API v1 (which has some limited Duplicate Rule support), Bulk API 2.0 does not support DMLOptions and will not check for duplicates regardless of configuration. Any Bulk API 2.0 load must use a pre-load deduplication step.
  5. Shield Platform Encryption breaks SOQL LIKE searches — Fields encrypted with probabilistic encryption (the default) cannot be searched using LIKE, wildcards, or full-text search (SOSL). Switching to deterministic encryption enables exact-match SOQL but has its own security trade-offs (deterministic ciphertext is vulnerable to frequency analysis). Plan encryption choices before enabling, not after.
  6. Cross-object validation rules are limited to one relationship level — A validation rule on Opportunity can reference Account.Type but cannot reference Account.Parent.Type. Attempts to traverse two levels result in a compile error. For deeper checks, use Apex triggers.

Output Artifacts

ArtifactDescription
Validation rule audit reportList of all validation rules with bypass status, formula complexity rating, and API-safe flag
Duplicate rule coverage matrixObject-by-object table of Matching Rule and Duplicate Rule configuration, mode (alert/block/report), and API DML coverage
Field history configuration listPer-object table of history-tracked fields (up to 20), retention tier (18-month vs Shield 10-year), and first-value gap flag
PII field registerField-level inventory with Data Sensitivity Level, Compliance Categorization, and encryption status
GDPR erasure runbookStep-by-step process for receiving, executing, and logging a right-to-erasure request with tokenization map
Data retention enforcement designBatch job or Flow design for enforcing retention cutoffs, with Retention_Expires__c field configuration
check_data_quality.py audit outputMachine-readable issue list from static metadata analysis

Package files