Security & Governance¶

Security is enforced at every layer of the Lakehouse — from AWS IAM at the infrastructure level to Unity Catalog at the data level.

AWS IAM¶

Roles¶

Role	Purpose	Assigned to
`DatabricksClusterRole`	EC2 instance profile for cluster nodes; allows S3 read/write	Databricks clusters
`DatabricksAdminRole`	Workspace administration	Platform team
`LakehouseReadOnlyRole`	Read-only access to Gold S3 prefix	BI tools, external consumers
`LakehouseIngestionRole`	Write access to Bronze S3 prefix	DMS, Lambda ingestion functions

Least-privilege S3 bucket policy¶

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Sid": "AllowClusterReadWrite",
      "Effect": "Allow",
      "Principal": {"AWS": "arn:aws:iam::<account>:role/DatabricksClusterRole"},
      "Action": ["s3:GetObject", "s3:PutObject", "s3:DeleteObject", "s3:ListBucket"],
      "Resource": [
        "arn:aws:s3:::my-lakehouse-bucket",
        "arn:aws:s3:::my-lakehouse-bucket/*"
      ]
    }
  ]
}

Encryption¶

Resource	Encryption
S3 buckets	SSE-KMS with a customer-managed key
EBS volumes (cluster nodes)	AWS-managed KMS key
Databricks Secret Scope	Backed by AWS Secrets Manager
Data in transit	TLS 1.2+ enforced on all endpoints

Unity Catalog¶

Unity Catalog is the single governance layer for all data assets in the Lakehouse.

Privilege hierarchy¶

Metastore
└── Catalog  (e.g. gold_catalog)
    └── Schema  (e.g. finance)
        └── Table  (e.g. fact_revenue)
            └── Column-level permissions (mask / filter)

Granting access¶

-- Grant read access to an analytics group on the Gold catalog
GRANT USE CATALOG ON CATALOG gold_catalog TO `analytics-team`;
GRANT USE SCHEMA  ON SCHEMA gold_catalog.finance TO `analytics-team`;
GRANT SELECT      ON TABLE  gold_catalog.finance.fact_revenue TO `analytics-team`;

Row and column filters¶

-- Mask PII column for non-privileged users
CREATE OR REPLACE FUNCTION gold_catalog.security.mask_email(email STRING)
RETURN IF(is_member('pii-access'), email, regexp_replace(email, '(.).+(@.+)', '$1***$2'));

ALTER TABLE gold_catalog.customers.dim_customer
  ALTER COLUMN email SET MASK gold_catalog.security.mask_email;

Data lineage¶

Unity Catalog automatically captures lineage for all reads and writes performed through Databricks. View lineage in the Catalog Explorer UI or query the system tables:

SELECT *
FROM system.access.table_lineage
WHERE target_table_full_name = 'gold_catalog.finance.fact_revenue'
ORDER BY event_time DESC
LIMIT 100;

Audit logging¶

All data access events are written to the Unity Catalog system tables and forwarded to Amazon CloudWatch Logs for SIEM integration:

system.access.audit — workspace audit events
system.access.table_lineage — read/write lineage
system.billing.usage — DBU consumption

Compliance¶

Standard	Status
SOC 2 Type II	Databricks holds certification; AWS environment in scope
GDPR	PII columns masked via Unity Catalog column masks; right-to-erasure via Delta `DELETE`
HIPAA	PHI stored in dedicated, additional-encrypted schemas with strict role-based access

Note

Contact the Security team before storing data classified as PHI or PCI in the Lakehouse.