Role Overview:
We are looking for a skilled and passionate Databricks Engineer to design, build, and optimize enterprise-scale data lakehouse solutions on the Databricks platform. The successful candidate will be responsible for creating Databricks pipeline delivering Financial Crime platforms covering Anti-Money Laundering (AML), Know Your Customer (KYC), Customer Risk Assessment (CRA), Sanctions Screening, Transaction Monitoring, Fraud Detection, and Regulatory Reporting
Responsibilities for Internal Candidates
- Design, build, and maintain Databricks workspaces, clusters, and compute pools across development, testing, and production environments.
- Configure and manage Unity Catalog for data governance, fine-grained access control, permissions, metadata management, and data lineage.
- Optimize Databricks cluster configurations, including instance types, auto-scaling, spot/preemptible nodes, and compute pools to improve performance and reduce costs.
- Implement workspace best practices, including folder structures, access controls, secret management using Databricks Secrets,
Azure Key Vault, or AWS Secrets Manager.
- Create, schedule, and manage Databricks Jobs, Workflows, and multi-task job orchestration with dependency management.
- Design and implement Delta Lake tables using partitioning, Z-Ordering, OPTIMIZE, VACUUM, and file compaction techniques.
- Build and maintain Medallion Architecture (Bronze, Silver, and Gold layers) for scalable and governed data lakehouse solutions.
- Develop Delta Live Tables (DLT) pipelines with built-in data quality expectations for reliable ETL/ELT processing.
- Manage schema evolution, table versioning, Time Travel, and Change Data Feed (CDF) to support incremental data processing.
- Design and implement lakehouse architectures integrating Delta Lake with cloud storage and external systems such as Azure Data Lake Storage (ADLS), Kafka, Event Hubs, and Kinesis.
- Develop scalable batch and real-time data pipelines using PySpar
📌 Databricks Engineer (Bengaluru)
🏢 EXL
📍 Bengaluru