Stage C

Conform

SP_CORE_CONFORM

Conform means: land each source’s evidence into the programme’s locked canonical model — one shared entity shape, not each system’s native schema. That is the first moment enterprise meaning is persisted. Native keys remain; lane-unique payload sits in extensions. Every assertion carries three control axes — valid, system, and warehouse — not one “load timestamp” blob.

Where Conform sits in CRISP

Rendering diagram…

Highlighted: Conform — first persistence of programme facts.

What is the canonical model?

Choosing the canonical model

You cannot conform without a target. The canonical model is that target: the overlap you deliberately lock when multiple vendors / systems / sources of the same product type describe the same subjects — typed columns in SP_CORE_CONFORM — while lane-only fields stay in extensions. It is not “copy one source’s schema.” Which entities belong is up to the implementor. Best practice: forensic lane inventories, AI promote / map / extensions drafts, then human lock. Industry standards are optional bias — not a wholesale copy into CORE.

What “canonical” means

Same product type, multiple vendors / systems / sources → one programme model. Example: native labels name, customer, preferred collapse to locked name in the ∩; each lobe keeps that system’s other variance (see Extensions).

Lobes — native labels

Common ∩ — locked name

A · B · C — discrete sources

Canonical is the overlap: what multiple vendors / systems / sources of the same product type agree on enough to lock as typed Conform columns. Everything else stays lane-native.

For how to shape that canonical (headers, links, why not FACT_*) and optional industry-model lenses (ACORD, FHIR, …), see ER modelling.

How do you load data from sources? Learn more about ER modelling

Adaptors — landed source → canonical Conform

An adaptor is how a source conforms: the deterministic map from a landed datasource (adaptor input / SRC — still native-shaped) onto the locked canonical entity contract in Conform. It keeps native keys inspectable, lifts temporal axes, and parks lane-unique fields in extensions. Cleaning a lake is not adapting; rewriting meaning in BI is not adapting.

Adaptor map · native → canon

System A

name

System B

customer

name

System C

preferred

name

Conform · typed PARTY.name (canon ∩)

Each adaptor conforms its native label to the locked canonical column. Lane-only fields stay in extensions — not shown here.

CRISP is tool-agnostic: the adaptor contract is fixed; delivery tooling is a team choice. Common helpers:

Category	Examples	How they help
SQL modelling	dbt, SQLMesh, Coalesce.io	Versioned SELECT models from landed sources into adaptor views / Conform-bound intermediates.
Native warehouse SQL	Stored procedures, Tasks, Dynamic Tables (where SELECT-only fits)	Same adaptor contract as pure SQL in Snowflake, BigQuery, Databricks SQL, Postgres, …
Spark / lakehouse jobs	AWS Glue, EMR, Databricks jobs, Dataflow / Dataproc	Batch or streaming transforms when landed data lives in object storage or Delta/Iceberg.
Orchestration (schedule only)	Airflow, Dagster, Step Functions, ADF / Fabric pipelines, Snowflake Tasks	Order Conform → Identity → Span → Publish; does not replace adaptor semantics.

Landing / ELT products (Fivetran, Airbyte, warehouse COPY, …) typically feed adaptor input. Adaptors then shape that land into Conform — do not conflate ingest with conformity.

Why three time axes? Learn more about Extensions

Three axes — valid, system, warehouse

Split by what the interval means, not by entity. Restatement, source-assertion history, and platform ingest are different questions — they must not share one clock.

AXIS · Valid time

Valid time

valid_from · valid_to

When was this true in the business?

Business-effective interval. Restatement (AS_WAS) slices on this axis — not on load day.

policy cover 2024-01-01 → 2024-12-31

Retroactive endorsement moves valid_* even if the warehouse only learns later.

AXIS · System time

System time

system_from · system_to

When did the source assert or correct this?

Source-recorded assertion interval as lifted into CORE. Corrections close the prior open row.

PAS posts correction system_from = 2024-03-12

Prefer native transaction time when the source has it; extract time is a documented proxy.

AXIS · Warehouse time

Warehouse time

warehouse_from · warehouse_to

When did the trusted load accept this row?

Platform ingest control — stamped by the Conform load, not by adaptor views. Does not propagate to Publish.

batch load warehouse_from = 2024-03-13 02:14Z

Audit of what CORE held uses system_* + Identity warehouse_*; warehouse_* alone is governance cut.

Adaptor input (SRC / L2) keeps per-source shapes until adaptors run — cleaning a lake is not conforming. Promotion from extensions to typed columns is a governed decision. Conform rows stay stable after load; Identity rebuilds separately so lag is visible.

What does the load look like in code? Learn more about Tritemporal

Pseudocode

Adaptor merge into Conform

function conform_load(lane, batch):
  src = read_adaptor_input(lane, batch)     # native keys intact

  for entity in programme_entities:         # locked canonical model
    mapped = VW_CORE_{entity}(src)          # adaptor view
    # Stamp axes — adaptors supply valid/system; trusted load stamps warehouse_*
    mapped.valid_from, mapped.valid_to = lift_business_time(mapped)
    mapped.system_from, mapped.system_to = lift_assertion_time(mapped)
    mapped.warehouse_from = load_now()
    mapped.warehouse_to   = NULL

    MERGE INTO SP_CORE_CONFORM.{entity}
      ON natural_or_surrogate_key
      WHEN new_assertion THEN INSERT
      WHEN correction THEN close prior (warehouse_to) + INSERT successor

  # Programme canon columns may still be unset — Rules apply next
  return conform_batch_id

Shape Conform as an assertion model (headers, roles/links — not FACT_*): ER modelling. Lane payload and promotion: Extensions.

Next: Rules Learn more about Immutability