Data Transformation¶

The Silver and Gold layers are built by transformation pipelines that cleanse, validate, enrich, and aggregate data from the Bronze layer.

Layer responsibilities¶

Layer	Purpose	Owner
Bronze	Raw copy of source data	Data Engineering
Silver	Cleansed, deduplicated, typed, validated	Data Engineering
Gold	Business aggregates and domain models	Analytics Engineering

Delta Live Tables (DLT)¶

Delta Live Tables is the primary pipeline framework for Silver layer transformations.

Example DLT pipeline¶

import dlt
from pyspark.sql.functions import col, to_date

@dlt.table(
    comment="Cleansed orders — nulls removed, types corrected",
    table_properties={"quality": "silver"},
)
@dlt.expect_or_drop("valid_order_id", "order_id IS NOT NULL")
@dlt.expect_or_drop("valid_amount", "amount > 0")
def orders_cleansed():
    return (
        dlt.read("orders_raw")
        .withColumn("order_date", to_date(col("order_date_str"), "yyyy-MM-dd"))
        .drop("order_date_str", "_source_file")
    )

Pipeline configuration¶

Each DLT pipeline is defined in a JSON configuration file committed to the repository:

{
  "name": "silver_orders_pipeline",
  "target": "silver_catalog.orders",
  "continuous": false,
  "clusters": [{"label": "default", "autoscale": {"min_workers": 1, "max_workers": 4}}],
  "libraries": [{"notebook": {"path": "/pipelines/silver/orders"}}],
  "configuration": {
    "pipelines.tableManagedByExternalMetastore": "true"
  }
}

dbt (Gold layer)¶

The Gold layer uses dbt with the Databricks adapter to build dimension and fact tables.

Project structure¶

dbt/
├── dbt_project.yml
├── profiles.yml          # local only — not committed
├── models/
│   ├── staging/          # thin wrappers over Silver tables
│   └── marts/
│       ├── finance/
│       └── operations/
└── tests/

Running dbt¶

cd dbt
dbt run --select marts/finance
dbt test --select marts/finance

Materialization strategy¶

Layer	dbt materialisation	Partition
Staging	`view`	n/a
Marts — large tables	`incremental`	`event_date`
Marts — small dimensions	`table`	n/a

Data quality checks¶

All Silver tables are validated with DLT expectations (hard or soft constraints). Gold models are tested with dbt tests:

not_null on primary keys
unique on primary keys
accepted_values on status/enum columns
relationships between fact and dimension tables

Scheduling¶

Pipeline	Schedule	Cluster type
Silver DLT (batch)	Every 2 hours	Autoscaling, `i3.2xlarge`
Silver DLT (streaming)	Continuous	Fixed 2-node, `m5.xlarge`
Gold dbt run	Daily at 04:00 UTC	Serverless SQL Warehouse