Databricks Platform Engineering ● Design, build, and maintain Databricks workspaces, clusters, and compute pools across dev/test/prod environments. ● Configure and manage Databricks Unity Catalog for data governance, access control, fine-grained permissions, and data lineage. ● Optimize cluster configurations — instance types, auto-scaling policies, spot/preemptible nodes — for cost and performance. ● Implement workspace-level best practices: folder structures, access controls, secret management (Databricks Secrets / Azure Key Vault / AWS Secrets Manager). ● Manage Databricks jobs, workflows, and multi-task job orchestration with dependency management.

Delta

Lake &

• Lakehouse Architecture ● Design and implement Delta Lake tables with appropriate partitioning, Z-ordering, and file compaction (OPTIMIZE / VACUUM). ● Build Medallion Architecture (Bronze / Silver / Gold) layers for structured data lake organization. ● Implement Delta Live Tables (DLT) pipelines for declarative, reliable ETL/ELT with built-in data quality expectations. ● Manage schema evolution, table versioning, time travel, and Change Data Feed (CDF) for incremental processing. ● Design data lakehouse patterns integrating Delta Lake with external systems (Kafka, ADLS, S3, GCS). EXL Service 10 Exchange Place, Suite 2200, Jersey City, NJ T: +1.(phone hidden) www.exlservice.com Data Pipeline Development (PySpark / SQL) ● Develop scalable batch and streaming data pipelines using PySpark, Spark SQL, and Delta Lake. ● Build structured streaming pipelines for real-time ingestion from Kafka, Event Hubs, and Kinesis into Delta tables. ● Write optimized PySpark transformations leveraging broadcast joins, adaptive query execution (AQE), and dynamic partition pruning. ● Create reusable transformation libraries, utility frameworks, and pipeline templates for team productivity. ● Implement robust error handling, retry logic, and dead-letter queue patterns in production pipelines. MLflow &

- AI/ML Workloads ● Set up and manage MLflow tracking servers, experiment registries, and model lifecycle management on Databricks. ● Support data scientists and ML engineers in deploying model training and inference workloads on Databricks clusters and GPU instances. ● Build feature engineering pipelines using Databricks Feature Store for reusable, versioned ML features. ● Enable GenAI workloads — LLM fine-tuning, RAG pipeline development, and vector search (Databricks Vector Search / Mosaic AI). ● Implement MLOps practices: model versioning, A/B testing, model serving via Databricks Model Serving endpoints.

Cloud

Integration &

• DevOps ● Integrate Databricks with cloud-native services: Azure Data Lake Storage (ADLS). ● Build and maintain CI/CD pipelines for Databricks notebooks and jobs using Azure DevOps, GitHub Actions, or GitLab CI. ● Implement Databricks Asset Bundles (DABs) or Terraform for infrastructure-as-code (IaC) deployment of Databricks resources. ● Manage data ingestion using Auto Loader, COPY INTO, and partner integrations (Fivetran, dbt, Airbyte). ● Monitor pipeline health, cluster utilization, and costs using Databricks system tables and cloud cost management tools. Governance, Security &
- Optimization ● Implement row-level security, column masking, and dynamic data views using Unity Catalog policies. ● Ensure data quality enforcement using Delta Live Tables expectations and Great Expectations integrations. ● Conduct performance tuning — query plan analysis, caching strategies, Photon engine enablement. ● Maintain data cataloging, metadata management, and data lineage tracking within Unity Catalog. ● Document architecture decisions, runbooks, and operational guides for Databricks workloads.

Responsibilities

Databricks Platform Engineering ● Design, build, and maintain Databricks workspaces, clusters, and compute pools across dev/test/prod environments. ● Configure and manage Databricks Unity Catalog for data governance, access control, fine-grained permissions, and data lineage. ● Optimize cluster configurations — instance types, auto-scaling policies, spot/preemptible nodes — for cost and performance. ● Implement workspace-level best practices: folder structures, access controls, secret management (Databricks Secrets / Azure Key Vault / AWS Secrets Manager). ● Manage Databricks jobs, workflows, and multi-task job orchestration with dependency management. Delta Lake & Lakehouse Architecture ● Design and implement Delta Lake tables with appropriate partitioning, Z-ordering, and file compaction (OPTIMIZE / VACUUM). ● Build Medallion Architecture (Bronze / Silver / Gold) layers for structured data lake organization. ● Implement Delta Live Tables (DLT) pipelines for declarative,



reliable ETL/ELT with built-in data quality expectations. ● Manage schema evolution, table versioning, time travel, and Change Data Feed (CDF) for incremental processing. ● Design data lakehouse patterns integrating Delta Lake with external systems (Kafka, ADLS, S3, GCS). EXL Service 10 Exchange Place, Suite 2200, Jersey City, NJ T: +1.(phone hidden) www.exlservice.com Data Pipeline Development (PySpark / SQL) ● Develop scalable batch and streaming data pipelines using PySpark, Spark SQL, and Delta Lake. ● Build structured streaming pipelines for real-time ingestion from Kafka, Event Hubs, and Kinesis into Delta tables. ● Write optimized PySpark transformations leveraging broadcast joins, adaptive query execution (AQE), and dynamic partition pruning. ● Create reusable transformation libraries, utility frameworks, and pipeline templates for team productivity. ● Implement robust error handling, retry logic, and dead-letter queue patterns in production pipelines. MLflow & AI/ML Workloads ● Set up and manage MLflow tracking servers, experiment registries, and model lifecycle management on Databricks. ● Support data scientists and ML engineers in deploying model training and inference workloads on Databricks clusters and GPU instances. ● Build feature engineering pipelines using Databricks Feature Store for reusable, versioned ML features. ● Enable GenAI workloads — LLM fine-tuning, RAG pipeline development, and vector search (Databricks Vector Search / Mosaic AI). ● Implement MLOps practices: model versioning, A/B testing, model serving via Databricks Model Serving endpoints. Cloud Integration & DevOps ● Integrate Databricks with cloud-native services: Azure Data Lake Storage (ADLS). ● Build and maintain CI/CD pipelines for Databricks notebooks and jobs using Azure DevOps, GitHub Actions, or GitLab CI. ● Implement Databricks Asset Bundles (DABs) or Terraform for infrastructure-as-code (IaC) deployment of Databricks resources. ● Manage data ingestion using Auto Loader, COPY INTO, and partner integrations (Fivetran, dbt, Airbyte). ● Monitor pipeline health, cluster utilization, and costs using Databricks system tables and cloud cost management tools. Governance, Security & Optimization ● Implement row-level security, column masking, and dynamic data views using Unity Catalog policies. ● Ensure data quality enforcement using Delta Live Tables expectations and Great Expectations integrations. ● Conduct performance tuning — query plan analysis, caching strategies, Photon engine enablement. ● Maintain data cataloging, metadata management, and data lineage tracking within Unity Catalog. ● Document architecture decisions, runbooks, and operational guides for Databricks workloads.

Qualifications

Databricks Platform Engineering ● Design, build, and maintain Databricks workspaces, clusters, and compute pools across dev/test/prod environments. ● Configure and manage Databricks Unity Catalog for data governance, access control, fine-grained permissions, and data lineage. ● Optimize cluster configurations — instance types, auto-scaling policies, spot/preemptible nodes — for cost and performance. ● Implement workspace-level best practices: folder structures, access controls, secret management (Databricks Secrets / Azure Key Vault / AWS Secrets Manager). ● Manage Databricks jobs, workflows, and multi-task job orchestration with dependency management. Delta Lake & Lakehouse Architecture ● Design and implement Delta Lake tables with appropriate partitioning, Z-ordering, and file compaction (OPTIMIZE / VACUUM). ● Build Medallion Architecture (Bronze / Silver / Gold) layers for structured data lake organization. ● Implement Delta Live Tables (DLT) pipelines for declarative, reliable ETL/ELT with built-in data quality expectations. ● Manage schema evolution, table versioning, time travel, and Change Data Feed (CDF) for incremental processing. ● Design data lakehouse patterns integrating Delta Lake with external systems (Kafka, ADLS, S3, GCS). EXL Service 10 Exchange Place, Suite 2200, Jersey City, NJ T: +1.(phone hidden) www.exlservice.com Data Pipeline Development (PySpark / SQL) ● Develop scalable batch and streaming data pipelines using PySpark, Spark SQL, and Delta Lake. ● Build structured streaming pipelines for real-time ingestion from Kafka, Event Hubs, and Kinesis into Delta tables.



● Write optimized PySpark transformations leveraging broadcast joins, adaptive query execution (AQE), and agile partition pruning. ● Create reusable transformation libraries, utility frameworks, and pipeline templates for team productivity. ● Implement robust error handling, retry logic, and dead-letter queue patterns in production pipelines. MLflow & AI/ML Workloads ● Set up and manage MLflow tracking servers, experiment registries, and model lifecycle management on Databricks. ● Support data scientists and ML engineers in deploying model training and inference workloads on Databricks clusters and GPU instances. ● Build feature engineering pipelines using Databricks Feature Store for reusable, versioned ML features. ● Enable GenAI workloads — LLM fine-tuning, RAG pipeline development, and vector search (Databricks Vector Search / Mosaic AI). ● Implement MLOps practices: model versioning, A/B testing, model serving via Databricks Model Serving endpoints. Cloud Integration & DevOps ● Integrate Databricks with cloud-native services: Azure Data Lake Storage (ADLS). ● Build and maintain CI/CD pipelines for Databricks notebooks and jobs using Azure DevOps, GitHub Actions, or GitLab CI. ● Implement Databricks Asset Bundles (DABs) or Terraform for infrastructure-as-code (IaC) deployment of Databricks resources. ● Manage data ingestion using Auto Loader, COPY INTO, and partner integrations (Fivetran, dbt, Airbyte). ● Monitor pipeline health, cluster utilization, and costs using Databricks system tables and cloud cost management tools. Governance, Security & Optimization ● Implement row-level security, column masking, and dynamic data views using Unity Catalog policies. ● Ensure data quality enforcement using Delta Live Tables expectations and Great Expectations integrations. ● Conduct performance tuning — query plan analysis, caching strategies, Photon engine enablement. ● Maintain data cataloging, metadata management, and data lineage tracking within Unity Catalog. ● Document architecture decisions, runbooks, and operational guides for Databricks workloads. EXL (NASDAQ: EXLS) is a leading data analytics and digital operations and solutions company. We partner with clients using a data and AI-led approach to reinvent business models, drive better business outcomes and unlock growth with speed. EXL harnesses the power of data, analytics, AI, and deep industry knowledge to transform operations for the world’s leading corporations in industries including insurance, healthcare, banking and financial services, media and retail, among others.

EXL was founded in 1999 with the core values of innovation, collaboration, excellence, integrity and respect. We are headquartered in New York and have more than 54,000 employees spanning six continents. For more information, visit www.exlservice.com.

EXL never requires or asks for fees/payments or credit card or bank details during any phase of the recruitment or hiring process and has not authorized any agencies or partners to collect any fee or payment from prospective candidates. EXL will only extend a job offer after a candidate has gone through a formal interview process with members of EXL’s Human Resources team, as well as our hiring managers.

EXL is the indispensable partner for leading businesses in data-led industries such as insurance, banking and financial services, healthcare, retail and logistics. We bring a unique combination of data, advanced analytics, digital technology and industry expertise to help our clients turn data into insights, streamline operations, improve customer experience, and transform their business. Our partnerships with clients are built on a foundation of collaboration – and we’ve been chosen as a partner by nine of the top ten leading US insurance companies, nine of the top 20 global banks, and six of the top ten US health care payers.

We function as one team to make your goals our goals, whether that’s unlocking the value of generative AI or embedding analytics into workflows that reduce risk or power your growth. Clients choose EXL as their transformation partner for many reasons. Our geographic diversity make talent all over the world instantly accessible.

Digital accelerators enable unmatched speed-to-value, letting you realize results fast. It’s our people that truly set us apart, though, including the 1,500 data scientists we have dedicated to our generative AI practice. And our more than twenty years of experience in delivering business services, garnering stellar client references, and maintaining a solid balance sheet are reassuring to our C-suite clients.

Find out for yourself why clients, employees, and analysts think we’re some of the best in the business. Contact us to see how we can help you achieve your goals.

📌 Databricks Tech Lead (Bengaluru)
🏢 EXL
📍 Bengaluru