Role Overview:
We are looking for a skilled and passionate Databricks Engineer to design,
build, and optimize enterprise-scale data lakehouse solutions on the Databricks
platform. The successful candidate will be responsible for creating Databricks
pipeline delivering Financial Crime platforms covering Anti-Money Laundering
(AML), Know Your Customer (KYC), Customer Risk Assessment (CRA), Sanctions
Screening, Transaction Monitoring, Fraud Detection, and Regulatory Reporting
Responsibilities for Internal Candidates:
• Design, build, and maintain Databricks workspaces, clusters, and compute pools
across development, testing, and production environments.
• Configure and manage Unity Catalog for data governance, fine-grained access
control, permissions, metadata management, and data lineage.
• Optimize Databricks cluster configurations, including instance types,
auto-scaling, spot/preemptible nodes, and compute pools to improve performance
and reduce costs.
• Implement workspace best practices, including folder structures, access
controls,
secret management using Databricks Secrets, Azure Key Vault, or AWS
Secrets Manager.
• Create, schedule, and manage Databricks Jobs, Workflows, and multi-task job
orchestration with dependency management.
• Design and implement Delta Lake tables using partitioning, Z-Ordering,
OPTIMIZE, VACUUM, and file compaction techniques.
• Build and maintain Medallion Architecture (Bronze, Silver, and Gold layers)
for scalable and governed data lakehouse solutions.
• Develop Delta Live Tables (DLT) pipelines with built-in data quality
expectations for reliable ETL/ELT processing.
• Manage schema evolution, table versioning, Time Travel, and Change Data Feed
(CDF) to support incremental data processing.
• Design and implement lakehouse architectures integrating Delta Lake with cloud
storage and external systems such as Azure Data Lake Storage (ADLS), Kafka,
Event Hubs, and Kinesis.
• Develop scalable batch and real-time data pipelines
📌 Lead Assistant Manager (Bengaluru)
🏢 EXL
📍 Bengaluru