Job Purpose The Senior Databricks Platform & Data Engineer will design, build, deploy, administer, and optimize enterprise data solutions using the Databricks Data Intelligence Platform on Amazon Web Services. The role combines advanced data engineering with platform administration and requires hands-on ownership of batch and realtime pipelines, platform security, Unity Catalog governance, compute optimization, deployment automation, monitoring, and production support. The engineer will work closely with data engineering, architecture, DevOps, cloud infrastructure, cybersecurity, networking, and firewall teams to deliver secure, scalable, reliable, and cost-effective data platforms.

Key Responsibilities
• Design, build, and maintain production-grade batch and real-time data pipelines using Databricks, Python, SQL, PySpark, Delta Lake, Apache Kafka, and Spark Structured Streaming.
• Administer, configure, deploy, monitor, and support Databricks environments on AWS, including workspaces, compute resources, permissions, jobs, and environment-specific configurations.
• Tune and scale Spark workloads, SQL queries, Delta tables, streaming jobs, and compute resources to meet performance, reliability, latency, and cost objectives.
• Evaluate serverless, classic, and hybrid compute models, recommending the appropriate approach based on workload characteristics, security, cost, scalability, performance, and operational complexity.
• Build fault-tolerant streaming pipelines using Kafka and Structured Streaming, including checkpointing, schema evolution, failure recovery, latency monitoring, and streaming-performance optimization.
• Implement Unity Catalog for centralized governance, metadata management, lineage, data discovery, access control,



auditing, and secure data sharing.
• Automate data ingestion, transformation, testing, deployment, monitoring, and recovery using Databricks workflows, Lakeflow Jobs, Auto Loader, Git-based CI/CD, and Databricks Asset Bundles where applicable.
• Establish secure connectivity between source systems, AWS services, Kafka, and Databricks, working with infrastructure, network, cybersecurity, and firewall teams.
• Configure and troubleshoot AWS networking and security, including VPCs, subnets, routing, security groups, private connectivity, IAM roles and policies, encryption keys, and secrets.
• Integrate Databricks with AWS services such as Amazon S3, EC2, CloudWatch, KMS, and Secrets Manager, with exposure to Kubernetes and Amazon EKS where applicable.
• Diagnose and resolve production issues involving source availability, data ingestion, streaming latency, Spark performance, compute capacity, permissions, networking, firewalls, and platform security.
• Document architecture decisions and technical trade-offs, clearly explaining the impact of design choices on cost, security, performance, availability, scalability, and maintainability. Key competencies Technical Competencies Databricks Platform Engineering: Advanced hands-on capability in administering, configuring, deploying, monitoring, tuning, scaling, and supporting Databricks on AWS.



Python, SQL and PySpark Engineering: Solid programming capability for developing, testing, optimizing, and troubleshooting production-grade pipelines. Kafka and Structured Streaming: Expertise in real-time ingestion, event-driven processing, fault tolerance, checkpointing, schema evolution, latency management, and streaming recovery. Unity Catalog, Security and Governance: Experience implementing data lineage, metadata governance, permissions, auditing, data isolation, secure sharing, and least-privilege access. AWS Infrastructure and Networking: Working knowledge of IAM, VPCs, subnets, routing, security groups, private connectivity, S3, EC2, CloudWatch, KMS, Secrets Manager, and firewall dependencies. Performance, Scalability and Cost Optimization: Ability to tune Spark jobs, SQL queries, streaming pipelines, Delta tables, clusters, and serverless compute while managing cost and reliability. Deployment and Automation: Experience with Git, CI/CD, Databricks Asset Bundles, automated testing, deployment automation, workflow orchestration, and operational monitoring. Architecture and Trade-off Analysis: Ability to assess serverless, classic, and hybrid approaches based on security, cost, performance, scalability, supportability, and operational risk. Behavioral Competencies Production Ownership and Problem-Solving: Ability to independently diagnose and resolve complex data, platform, networking, security, streaming, and performance issues. Cross-functional Collaboration and Communication: Ability to work effectively with data engineering, architecture, DevOps, cloud infrastructure, cybersecurity, networking, firewall, and source-system teams.

📌 Assistant Director-Databricks Platform and Data Engineer (Bengaluru)
🏢 Acuity Analytics
📍 Bengaluru