Architecture Overview¶
High-level design¶
The platform follows the Medallion Architecture (Bronze → Silver → Gold) on top of Delta Lake, deployed entirely within AWS.
┌──────────────────────────────────────────────────────────────┐
│ Data Sources │
│ RDS / Aurora │ Kinesis │ S3 files │ SaaS APIs │
└────────┬───────┴─────┬─────┴──────┬─────┴──────┬────────────┘
│ │ │ │
▼ ▼ ▼ ▼
┌──────────────────────────────────────────────────────────────┐
│ Ingestion Layer │
│ AWS DMS │ Kinesis Firehose │ Autoloader │ APIs │
└─────────────────────────┬────────────────────────────────────┘
│
▼
┌──────────────────────────────────────────────────────────────┐
│ Amazon S3 (Delta Lake) │
│ │
│ ┌──────────┐ ┌──────────┐ ┌──────────────────────┐ │
│ │ Bronze │───▶│ Silver │───▶│ Gold │ │
│ │ (raw) │ │(cleansed)│ │ (aggregated/served) │ │
│ └──────────┘ └──────────┘ └──────────────────────┘ │
└─────────────────────────┬────────────────────────────────────┘
│
▼
┌──────────────────────────────────────────────────────────────┐
│ Databricks Workspace │
│ │
│ Delta Live Tables │ Notebooks │ Jobs │ SQL Warehouse │
│ │
│ Unity Catalog (Metastore, Access Control, Lineage) │
└─────────────────────────┬────────────────────────────────────┘
│
▼
┌──────────────────────────────────────────────────────────────┐
│ Consumption Layer │
│ BI Tools (Tableau/Power BI) │ ML Models │ APIs │
└──────────────────────────────────────────────────────────────┘
Component descriptions¶
Amazon S3¶
All data at rest lives in versioned S3 buckets, organised by layer:
| Bucket prefix | Layer | Retention |
|---|---|---|
s3://…/bronze/ |
Raw ingested data | 7 years |
s3://…/silver/ |
Cleansed & validated | 7 years |
s3://…/gold/ |
Aggregated, query-ready | 7 years |
s3://…/checkpoints/ |
Streaming checkpoints | 90 days |
Databricks Workspace¶
- Deployed in a customer-managed VPC (no public IPs on clusters).
- Workspace is linked to a Unity Catalog metastore scoped to the AWS region.
- Cluster policies enforce node types and autoscaling limits.
Unity Catalog¶
Unity Catalog is the governance layer:
- Metastore — one per region, shared across workspaces.
- Catalogs —
bronze_catalog,silver_catalog,gold_catalog. - Schemas / Tables — Delta tables registered with column-level permissions.
Networking¶
| Resource | Configuration |
|---|---|
| VPC | Dedicated VPC with private subnets |
| NAT Gateway | Outbound internet access for clusters |
| VPC Endpoints | S3, STS, Kinesis via PrivateLink |
| Security Groups | Least-privilege inbound/outbound rules |
Key design decisions¶
ADR-001: Delta Lake as the storage format¶
Decision: Use Delta Lake (Parquet + transaction log) for all layers.
Rationale: ACID transactions, schema enforcement, time-travel, and Z-order optimisation make Delta Lake the best fit for a mixed batch/streaming workload.
ADR-002: Medallion Architecture¶
Decision: Three-layer Bronze/Silver/Gold pattern.
Rationale: Separates raw from curated data, simplifies debugging, and gives consumers a stable Gold layer regardless of upstream schema changes.
ADR-003: Unity Catalog for governance¶
Decision: All table access goes through Unity Catalog; direct S3 access is denied for end users.
Rationale: Centralised lineage, audit logs, and fine-grained access control without per-cluster configuration.
ADR-004: Security and Networking¶
Decision: Deploy the Databricks workspace in a customer-managed VPC with private subnets, use NAT Gateway for outbound internet access, and enforce least-privilege security group rules.
Rationale: Ensures network isolation, controlled internet access, and adherence to security best practices for cloud deployments.