1. Executive Summary¶
This document defines a recommended AWS multi-account and multi-OU architecture for Databricks using AWS Transit Gateway (TGW) as the centralized network connectivity layer.
The design separates Databricks workloads from other application workloads by placing them in dedicated AWS Organizational Units (OUs), while keeping shared networking services in a centralized Infrastructure/Network account.
The proposed structure is:
AWS Organization
│
├── Security OU
│ ├── Security Account
│ └── Log Archive Account
│
├── Infrastructure OU
│ └── Network Account
│ └── Transit Gateway
│
├── Databricks OU
│ ├── Databricks-Prod Account
│ ├── Databricks-NonProd Account
│ └── Databricks-Shared Account
│
└── Applications OU
├── Applications-Prod Account
├── Applications-NonProd Account
└── Shared-Applications Account
AWS recommends organizing OUs around functions and common security controls rather than organizational reporting structures, and recommends foundational Security and Infrastructure OUs for centralized governance and infrastructure. AWS Organizations best practices
2. Architecture Goals¶
The architecture is designed to provide:
- Strong separation between Databricks and application workloads
- Centralized network connectivity
- Centralized Transit Gateway management
- Multi-account isolation
- Production/non-production separation
- Controlled east-west traffic
- Private connectivity between Databricks and application services
- Centralized network security and monitoring
- Scalable connectivity for future VPCs
- Clear ownership and separation of duties
3. AWS Organizational Structure¶
3.1 Recommended OU Structure¶
Root
│
├── Security OU
│ ├── Security Account
│ └── Log Archive Account
│
├── Infrastructure OU
│ └── Network Account
│ ├── Transit Gateway
│ ├── Network Firewall
│ ├── Route 53 Resolver
│ └── IPAM
│
├── Databricks OU
│ ├── Databricks Production
│ ├── Databricks Non-Production
│ └── Databricks Shared Services
│
├── Applications OU
│ ├── Application Production
│ ├── Application Non-Production
│ └── Application Shared Services
│
└── Sandbox OU
└── Developer/Sandbox Accounts
The Databricks OU is intentionally separate from the Applications OU. This allows different Service Control Policies (SCPs), IAM boundaries, security controls, budgets, and operational ownership to be applied to each workload group.
AWS recommends applying controls at the OU level where practical and separating production from non-production workloads. AWS multi-account design principles
4. Account Responsibilities¶
4.1 Network Account¶
The Network Account resides in the Infrastructure OU.
Primary responsibilities:
- AWS Transit Gateway
- TGW route tables
- VPC attachments
- AWS RAM sharing
- Network Firewall
- Centralized egress, if required
- Route 53 Resolver
- AWS VPC IPAM
- Network monitoring
- Network flow logs
Example:
Infrastructure OU
└── Network Account
├── Transit Gateway
├── Network Firewall
├── Route 53 Resolver
├── IPAM
└── Network monitoring
AWS Prescriptive Guidance recommends a centralized network account for shared network services and identifies Transit Gateway as a preferred hub-and-spoke connectivity option for multi-VPC environments. AWS Network Account guidance
5. Databricks OU¶
The Databricks OU contains AWS accounts dedicated to Databricks workloads.
5.1 Production Account¶
Production workspaces should be isolated from development and testing environments.
5.2 Non-Production Account¶
This account can contain:
- Development workspace
- QA workspace
- Test workspace
- Development data pipelines
5.3 Shared Databricks Account¶
Optional shared account:
Databricks OU
└── Databricks-Shared Account
├── Shared platform services
├── CI/CD services
├── Automation
└── Platform management
6. Applications OU¶
The Applications OU contains application and data services that Databricks may need to access.
Example:
Applications OU
│
├── Applications-Prod Account
│ └── Application/Data VPC
│ CIDR: 10.20.0.0/16
│
├── Applications-NonProd Account
│ └── Application/Data VPC
│ CIDR: 10.21.0.0/16
│
└── Shared-Applications Account
└── Shared Services VPC
CIDR: 10.22.0.0/16
Potential services include:
- Amazon RDS
- Amazon Aurora
- Internal APIs
- ECS/EKS applications
- Application Load Balancers
- Data services
- Internal microservices
- Shared application platforms
7. High-Level Network Architecture¶
AWS ORGANIZATION
│
┌────────────────────────┼────────────────────────┐
│ │ │
▼ ▼ ▼
Security OU Infrastructure OU Workload OUs
│ │
▼ │
Network Account │
│ │
┌───────▼───────┐ │
│ Transit │ │
│ Gateway │ │
└───────┬───────┘ │
│ │
┌──────────────────┼──────────────────┐ │
│ │ │ │
▼ ▼ ▼ │
Databricks Databricks Applications
Prod VPC NonProd VPC VPC
10.10/16 10.11/16 10.20/16
│ │ │
▼ ▼ ▼
Databricks Databricks RDS/Aurora
Workspaces Workspaces APIs/Apps
8. Databricks AWS Network¶
For classic Databricks compute, compute resources are deployed in the customer's AWS VPC. Databricks documentation supports a hub-and-spoke architecture in which workspace VPCs connect to a hub through Transit Gateway. Databricks AWS network architecture
Example:
Databricks Production VPC
10.10.0.0/16
│
├── AZ-1
│ └── Databricks Private Subnets
│
├── AZ-2
│ └── Databricks Private Subnets
│
├── VPC Endpoints
├── NAT Gateway (if required)
├── Network Firewall (if required)
└── TGW Attachment
9. Transit Gateway¶
The Network Account owns the Transit Gateway.
Network Account
│
▼
┌───────────────┐
│ Transit │
│ Gateway │
└───────┬───────┘
│
┌─────────────────┼─────────────────┐
│ │ │
▼ ▼ ▼
Databricks Prod Databricks NonProd Applications
VPC VPC VPC
10.10/16 10.11/16 10.20/16
Transit Gateway supports VPC attachments across AWS accounts, and AWS RAM can be used to share the TGW with accounts in the organization. AWS Transit Gateway
10. Cross-Account Connectivity¶
The Network Account shares the Transit Gateway with the organization or selected OUs using AWS Resource Access Manager (RAM).
Network Account
│
└── Transit Gateway
│
│ AWS RAM
│
├───────────────► Databricks-Prod Account
│
├───────────────► Databricks-NonProd Account
│
├───────────────► Applications-Prod Account
│
└───────────────► Applications-NonProd Account
Each workload account creates its own VPC attachment to the shared TGW.
11. CIDR Allocation¶
A centralized IP address plan should be used to prevent overlapping CIDRs.
Environment Account VPC CIDR
Databricks Production Databricks-Prod 10.10.0.0/16 Databricks Non-Production Databricks-NonProd 10.11.0.0/16 Applications Production Applications-Prod 10.20.0.0/16 Applications Non-Production Applications-NonProd 10.21.0.0/16 Shared Services Shared-Applications 10.22.0.0/16 Network Network Account 10.30.0.0/16
Use AWS VPC IPAM to centrally allocate and monitor CIDR ranges as the environment grows.
12. Transit Gateway Route Tables¶
A single TGW route table can work for simple environments, but production architectures should generally use separate TGW route tables when traffic isolation is required.
12.1 Databricks Production Route Table¶
TGW Route Table: Databricks-Prod
Destination Attachment
10.20.0.0/16 Applications-Prod
10.30.0.0/16 Shared Services
12.2 Databricks Non-Production Route Table¶
TGW Route Table: Databricks-NonProd
Destination Attachment
10.21.0.0/16 Applications-NonProd
10.30.0.0/16 Shared Services
12.3 Applications Production Route Table¶
TGW Route Table: Applications-Prod
Destination Attachment
10.10.0.0/16 Databricks-Prod
10.30.0.0/16 Shared Services
This prevents development traffic from automatically reaching production.
AWS specifically recommends using separate TGW route tables when traffic must be isolated between environments. AWS network connectivity guidance
13. Recommended Segmentation¶
Transit Gateway
│
┌──────────────────┼──────────────────┐
│ │ │
▼ ▼ ▼
DBX PROD RT DBX NONPROD RT APP PROD RT
│ │ │
▼ ▼ ▼
DBX PROD VPC DBX NONPROD VPC APP PROD VPC
│ │ │
│ │ │
└───────────X──────┴──────X──────────┘
No direct cross-environment
communication
The desired result is:
Databricks Prod ─────► Applications Prod
│
└──────────────► Shared Services
Databricks NonProd ──► Applications NonProd
│
└──────────────► Shared Services
Databricks NonProd ──X─► Applications Prod
Databricks Prod ─────X─► Applications NonProd
14. Security Groups¶
Security Groups should implement least-privilege access at the workload level.
Example:
Aurora Security Group:
For an internal API:
Only the required source CIDRs and ports should be allowed.
15. DNS¶
Private DNS should be centrally designed where applications are accessed using internal names.
Example:
Recommended components:
- Amazon Route 53 Private Hosted Zones
- Route 53 Resolver
- Resolver forwarding rules
- Central DNS services where required
DNS architecture should be independent of the Databricks OU so that shared services can be consumed by both Databricks and application accounts.
16. Network Security¶
Recommended centralized network security:
Internet
│
▼
Centralized Egress / Inspection
│
▼
Network Firewall
│
▼
Transit Gateway
│
├── Databricks
├── Applications
└── Shared Services
Potential controls:
- AWS Network Firewall
- VPC Flow Logs
- TGW Flow Logs
- AWS CloudTrail
- AWS Config
- GuardDuty
- Security Hub
- Network Access Analyzer
17. Databricks Egress¶
Databricks classic compute may require outbound connectivity for AWS and Databricks services.
A production design may use:
For high availability, use NAT Gateway and associated networking across the required Availability Zones.
Databricks recommends customer-managed VPC architectures when organizations require stronger control over networking, routing, and egress. Databricks AWS network design
18. S3 Connectivity¶
Where appropriate, use Amazon S3 VPC endpoints for private connectivity to S3.
Example:
This can reduce unnecessary NAT Gateway traffic and keep S3 access on AWS private networking.
19. Application Access from Databricks¶
Example: Databricks Production accessing Aurora in Applications Production.
┌───────────────────────────────┐
│ Databricks Prod Account │
│ │
│ Databricks Cluster │
│ 10.10.x.x │
└───────────────┬───────────────┘
│
│ TCP 5432
▼
Databricks VPC
│
▼
TGW Attachment
│
▼
Transit Gateway
│
▼
Applications-Prod
│
▼
Aurora PostgreSQL
10.20.x.x
No public IP is required for this application-to-database traffic.
20. Recommended OU-Level Controls¶
Databricks OU¶
Potential controls:
- Restrict creation of unauthorized network gateways
- Require approved regions
- Restrict public S3 access
- Restrict public database exposure
- Require encryption
- Require CloudTrail/Config integration
- Restrict IAM privilege escalation
- Require approved VPC configurations
Applications OU¶
Potential controls:
- Restrict public exposure of databases
- Require encryption
- Require centralized logging
- Restrict unsupported regions
- Enforce security monitoring
- Require approved network patterns
SCPs should be designed according to organizational requirements and tested in a policy-staging environment before broad deployment.
21. Ownership Model¶
Area Primary Owner
AWS Organizations Cloud Platform Security OU Security Team Infrastructure OU Network/Platform Team Transit Gateway Network Team Databricks OU Databricks Platform Team Databricks Workspace Databricks Platform Admin Applications OU Application Teams VPCs Account/Application Owners TGW Route Tables Network Team Security Groups Workload Owners DNS Network/Platform Team Network Firewall Security/Network Team
This provides separation of duties while allowing application teams to own their workloads.
22. Deployment Flow¶
1. Create AWS Organization
│
▼
2. Create Security and Infrastructure OUs
│
▼
3. Create Network Account
│
▼
4. Create Transit Gateway
│
▼
5. Create Databricks OU
│
▼
6. Create Databricks Accounts
│
▼
7. Create Applications OU
│
▼
8. Create Application Accounts
│
▼
9. Share TGW using AWS RAM
│
▼
10. Create VPC Attachments
│
▼
11. Configure TGW Route Tables
│
▼
12. Configure VPC Route Tables
│
▼
13. Configure Security Groups/NACLs
│
▼
14. Configure DNS
│
▼
15. Test Connectivity
23. Connectivity Testing¶
Test the following from Databricks:
DNS¶
TCP Connectivity¶
HTTPS¶
Validate:
- DNS resolution
- VPC route
- TGW route
- TGW attachment
- Security Group
- NACL
- Target service
- Firewall policy
24. Terraform Architecture¶
A Terraform implementation can be separated into modules:
terraform/
│
├── modules/
│ ├── organization/
│ ├── vpc/
│ ├── transit-gateway/
│ ├── tgw-route-table/
│ ├── security-groups/
│ ├── network-firewall/
│ └── dns/
│
├── environments/
│ ├── prod/
│ │ ├── network/
│ │ ├── databricks/
│ │ └── applications/
│ │
│ └── nonprod/
│ ├── network/
│ ├── databricks/
│ └── applications/
│
└── policies/
├── security-scp/
├── databricks-scp/
└── applications-scp/
Typical AWS resources include:
aws_organizations_organizational_unit
aws_organizations_account
aws_ec2_transit_gateway
aws_ec2_transit_gateway_vpc_attachment
aws_ec2_transit_gateway_route_table
aws_ec2_transit_gateway_route
aws_ec2_transit_gateway_route_table_association
aws_ec2_transit_gateway_route_table_propagation
aws_ram_resource_share
aws_vpc
aws_route_table
aws_route
aws_security_group
aws_network_acl
25. Recommended Final Architecture¶
AWS ORGANIZATION
│
┌──────────────────────────┼──────────────────────────┐
│ │ │
▼ ▼ ▼
Security OU Infrastructure OU Workload OUs
│ │ │
│ ▼ │
│ Network Account │
│ │ │
│ ┌───────▼───────┐ │
│ │ Transit │ │
│ │ Gateway │ │
│ └───────┬───────┘ │
│ │ │
│ ┌─────────────┼─────────────┐ │
│ │ │ │ │
│ ▼ ▼ ▼ │
│ Databricks Databricks Applications │
│ Prod NonProd Prod │
│ Account Account Account │
│ │ │ │ │
│ ▼ ▼ ▼ │
│ VPC VPC VPC │
│ 10.10/16 10.11/16 10.20/16 │
│ │ │ │ │
│ └─────────────┼─────────────┘ │
│ │ │
│ Controlled TGW │
│ Route Tables │
│ │ │
│ ▼ │
│ Shared Services │
│ DNS / Security │
│ Monitoring / Egress │
│ │
└─────────────────────────────────────────────────────┘
26. Key Design Decisions¶
Decision 1 --- Separate Databricks OU¶
Databricks accounts are isolated from general application accounts.
Benefit: independent governance, security controls, budgets, access policies, and platform ownership.
Decision 2 --- Central Network Account¶
The Transit Gateway is owned by the Network Account under the Infrastructure OU.
Benefit: centralized routing and separation of network administration from workload administration.
Decision 3 --- TGW Instead of Full-Mesh Peering¶
Transit Gateway provides scalable hub-and-spoke connectivity.
Benefit: adding another VPC does not require creating multiple VPC peering relationships.
Decision 4 --- Separate TGW Route Tables¶
Production and non-production traffic are separated.
Benefit: reduces unintended lateral movement and supports environment isolation.
Decision 5 --- Non-Overlapping CIDRs¶
All VPC CIDRs are centrally managed.
Benefit: prevents routing conflicts and simplifies future expansion.
Decision 6 --- Private Connectivity¶
Databricks accesses application services using private IP connectivity through the TGW.
Benefit: reduces exposure to the public Internet.
27. Reference Architecture Summary¶
The recommended architecture is:
AWS Organizations
│
┌────────────────────┼────────────────────┐
│ │ │
▼ ▼ ▼
Security OU Infrastructure OU Databricks OU
│ │
▼ │
Network Account │
│ │
▼ ▼
Transit Gateway Databricks Accounts
│ │
┌─────────────┼─────────────┐ │
│ │ │ │
▼ ▼ ▼ ▼
DBX Prod DBX NonProd Apps VPCs
VPC VPC VPC
│ │ │
└─────────────┼─────────────┘
│
Controlled Routing
│
▼
Shared Services / DNS /
Security / Monitoring
This design provides a scalable foundation for an enterprise Databricks platform while keeping Databricks workloads logically and administratively separated from other AWS applications.
28. Reference Documentation¶
- AWS Organizations best practices: https://docs.aws.amazon.com/organizations/latest/userguide/orgs_manage_ous_best_practices.html
- AWS Transit Gateway: https://docs.aws.amazon.com/vpc/latest/tgw/tgw-transit-gateways.html
- AWS centralized Transit Gateway architecture: https://docs.aws.amazon.com/prescriptive-guidance/latest/patterns/centralize-network-connectivity-using-aws-transit-gateway.html
- AWS network connectivity guidance: https://docs.aws.amazon.com/prescriptive-guidance/latest/transitioning-to-multiple-aws-accounts/network-connectivity.html
- Databricks AWS network architecture: https://docs.databricks.com/aws/en/lakehouse-architecture/deployment-guide/network