Role Overview:
We are looking for a skilled and passionate Databricks Engineer to design, build, and optimize enterprise-scale data lakehouse solutions on the Databricks platform. The successful candidate will be responsible for creating Databricks pipeline delivering Financial Crime platforms covering Anti-Money Laundering (AML), Know Your Customer (KYC), Customer Risk Assessment (CRA), Sanctions Screening, Transaction Monitoring, Fraud Detection, and Regulatory Reporting
Responsibilities for Internal Candidates
Design, build, and maintain Databricks workspaces, clusters, and compute pools across development, testing, and production environments.
Configure and manage Unity Catalog for data governance, fine-grained access control, permissions, metadata management, and data lineage.
Optimize Databricks cluster configurations, including instance types, auto-scaling, spot/preemptible nodes, and compute pools to improve performance and reduce costs.
Implement workspace best practices, including folder structures, access controls, secret management using Databricks Secrets,
Azure Key Vault, or AWS Secrets Manager.
Create, schedule, and manage Databricks Jobs, Workflows, and multi-task job orchestration with dependency management.
Design and implement Delta Lake tables using partitioning, Z-Ordering, OPTIMIZE, VACUUM, and file compaction techniques.
Build and maintain Medallion Architecture (Bronze, Silver, and Gold layers) for scalable and governed data lakehouse solutions.
Develop Delta Live Tables (DLT) pipelines with built-in data quality expectations for reliable ETL/ELT processing.
Manage schema evolution, table versioning, Time Travel, and Change Data Feed (CDF) to support incremental data processing.
Design and implement lakehouse architectures integrating Delta Lake with cloud storage and external systems such as Azure Data Lake Storage (ADLS), Kafka, Event Hubs, and Kinesis.
Develop scalable batch and real-time data pipelines using PySpark, Spark SQL, Structur
📌 Databricks Engineer (India)
🏢 EXL
📍 India