Operations & Monitoring¶

This page covers day-to-day operations, cluster management, cost controls, and the observability stack for the AWS Databricks Lakehouse.

Cluster management¶

Cluster policies¶

All clusters must use a policy that enforces:

Maximum cluster size (e.g. 8 workers for interactive clusters)
Autoscaling enabled
Auto-termination after 30 minutes of inactivity (interactive) or job completion (job clusters)
Approved instance types only (m5, i3, r5 families)

Interactive vs. job clusters¶

Type	Use case	Termination
Interactive (All-Purpose)	Notebooks, ad-hoc exploration	Auto-terminate after 30 min
Job cluster	Scheduled pipeline runs	Terminates on job completion
SQL Warehouse (Serverless)	SQL analytics, BI tools	Auto-suspend after 10 min

Restarting a cluster¶

databricks clusters restart --cluster-id <cluster-id>

Delta table maintenance¶

OPTIMIZE and VACUUM¶

Run periodically to compact small files and remove deleted data:

-- Compact files and apply Z-order on frequently filtered columns
OPTIMIZE silver_catalog.orders.orders_cleansed
ZORDER BY (order_date, customer_id);

-- Remove files older than 7 days (default retention)
VACUUM silver_catalog.orders.orders_cleansed;

A scheduled Databricks Job runs these commands nightly for all registered tables.

Table statistics¶

Keep statistics up to date for query planning:

ANALYZE TABLE gold_catalog.finance.fact_revenue COMPUTE STATISTICS FOR ALL COLUMNS;

Monitoring & alerting¶

Metrics¶

Metric	Source	Alert threshold
Job run duration	Databricks Jobs API → CloudWatch	>2× historical p95
Streaming lag	Databricks Structured Streaming UI	>15 minutes
Cluster CPU utilization	CloudWatch EC2 metrics	>90% sustained 5 min
DBU spend	Databricks billing API	>110% of weekly budget
S3 storage growth	CloudWatch S3 metrics	>20% week-over-week

CloudWatch dashboards¶

Two dashboards are maintained in CloudWatch:

Lakehouse — Pipelines: job success rate, streaming lag, ingestion throughput.
Lakehouse — Cost: daily DBU spend by workspace, S3 storage cost, EC2 spend.

Alerts¶

All alerts route to:

PagerDuty (high-severity, production pipelines)
#lakehouse-alerts Slack channel (all environments)

Incident runbooks¶

Pipeline failure¶

Open the Databricks Jobs UI and check the run details.
Review the driver logs for the error message.
If the error is transient (network timeout, spot interruption), re-trigger the job manually.
If data quality expectations fail, investigate the quarantine table: bronze_catalog.<domain>.dlt_expectations_quarantine.
Escalate to the Data Engineering on-call if the issue is not resolved within 30 minutes.

Streaming job stuck / lagging¶

Check the Structured Streaming tab on the cluster UI.
Verify Kinesis shard iterator age in CloudWatch.
Restart the streaming job via the Databricks Jobs UI.
If lag persists, increase worker count on the cluster.

Cost management¶

Tagging strategy¶

All AWS resources are tagged:

Tag key	Example value
`Environment`	`production`
`Team`	`data-engineering`
`CostCenter`	`12345`
`Project`	`aws-lakehouse`

Spot instances¶

Job clusters use Spot instances with an On-Demand fallback to reduce EC2 costs by ~60%. Interactive clusters use On-Demand to avoid interruptions during development.

Photon¶

Photon is enabled on all SQL Warehouses and job clusters running SQL/DLT workloads. Photon reduces query time and therefore DBU consumption for scan-heavy workloads.

Backup & disaster recovery¶

Asset	Backup mechanism	RPO	RTO
Delta tables	S3 versioning + cross-region replication	1 hour	4 hours
Unity Catalog metadata	Databricks-managed backup	24 hours	8 hours
Notebooks	Git-backed via Databricks Repos	On every commit	1 hour
Secrets	AWS Secrets Manager with cross-region replication	1 hour	1 hour