We are seeking an experienced AWS Databricks Platform Administrator to design operate secure and optimize enterprise Databricks environments on AWS The role is responsible for platform availability governance performance cost management automation and production support The ideal candidate combines strong Databricks administration expertise with handson AWS infrastructureascode security networking and DevOps experience
Key Responsibilities
- Administer Databricks accounts workspaces users groups service principals entitlements permissions and accesscontrol policies
- Provision configure and maintain Databricks workspaces on AWS using Terraform CloudFormation or equivalent automation
- Manage Unity Catalog metastores catalogs schemas external locations storage credentials lineage and finegrained data access
- Configure and support clusters SQL warehouses instance pools cluster policies libraries jobs notebooks and workspace objects
- Integrate Databricks with AWS services including IAM S3 VPC KMS CloudWatch CloudTrail Secrets Manager PrivateLink and Route 53
- Implement secure network architecture including VPC peering private subnets security groups endpoints DNS firewall rules and egress controls
- Monitor platform health capacity audit logs job performance cluster utilization and service availability proactively resolve risks and bottlenecks
- Drive cost optimization through tagging budgets chargebackshowback cluster rightsizing autotermination spot instances and usage analysis
- Automate platform deployment configuration policy enforcement monitoring and operational tasks through CICD pipelines and scripting
- Lead incident problem change release and servicerequest management for production and nonproduction environments
- Define backup disaster recovery business continuity patching upgrade and environmentmanagement procedures
- Partner with data engineering analytics security cloud and application teams to onboard workloads and enforce platform standards
- Create and maintain runbooks architecture diagrams configuration standards operating procedures and knowledge articles
- Provide technical leadership conduct rootcause analysis mentor junior administrators and support audits and compliance reviews
Required Skills and Experience
- 6 years of experience in cloud data platform infrastructure DevOps or production administration roles
- Strong hands on experience administering Azure Databricks or AWS Databricks with demonstrable expertise in AWS hosted deployments
- Deep knowledge of Databricks workspace architecture compute jobs SQL warehouses cluster policies account console and Unity Catalog
- Strong AWS expertise across IAM S3 EC2 VPC KMS CloudWatch CloudTrail Secrets Manager networking and private connectivity
- Experience with Terraform and CICD platforms such as GitHub Actions Jenkins GitLab CI or AWS Code Pipeline
- Proficiency in Python Bash PowerShell or similar scripting languages for platform automation and troubleshooting
- Working knowledge of Spark Delta Lake SQL data lakehouse architecture and common data engineering workloads
- Solid understanding of identity RBAC encryption secrets audit logging data governance security controls and compliance requirements
- Experience with monitoring and observability tools and with troubleshooting distributed systems jobs clusters and cloud infrastructure
- Strong IT service management experience covering incidents problems changes requests SLAs root cause analysis and oncall support
Education and Certifications
- Bachelors degree in Computer Science Information Technology Engineering or a related discipline equivalent practical experience may be considered
- Databricks Certified Data Engineer Associate Developer for Apache Spark or Databricks platform related certification is preferred
- AWS Certified Solutions Architect SysOps Administrator DevOps Engineer or Security certification is preferred
Disclaimer: This job posting has been aggregated from external source. Role details, content, and availability are subject to change. Applicants are advised to confirm the latest information directly on the company website before applying.