Location

Jaipur

Experience

6-8 years

Industry

AI/Technology

Job Summary

Design, build, and optimize large-scale, production-grade data pipelines and analytics platforms on Azure, leveraging Databricks, Synapse, and the broader Microsoft data ecosystem. Deliver business-critical data assets for analytics, BI, and AI/ML initiatives.

Key Technical Responsibilities

Architect contemporary data lakes using Azure Data Lake Storage Gen2 for batch and streaming workloads.
Build and maintain scalable ETL/ELT pipelines using Azure Data Factory and Databricks (PySpark, Scala, SQL).
Orchestrate data workflows across ADF, Databricks, and Synapse Pipelines; implement modular and reusable data pipeline components.
Develop advanced notebooks and production jobs in Azure Databricks (PySpark, SparkSQL, Delta Lake).
Optimize Spark jobs by tuning partitioning, caching, cluster configuration, and autoscaling for performance and cost.
Implement Delta Lake for ACID-compliant data lakes and enable time travel and audit features.
Engineer real-time data ingestion from Event Hubs, IoT Hub,



and Kafka into Databricks and Synapse.
Transform and enrich raw data, building robust data models and marts for analytics and AI use cases.
Integrate structured, semi-structured, and unstructured data sources, including APIs, logs, and files.
Implement data validation, schema enforcement, and quality checks using Databricks, PySpark, and tools like Great Expectations.
Manage access controls using Azure AD, Databricks workspace permissions, RBAC, and Azure Key Vault integration.
Enable end-to-end data lineage and cataloging via Microsoft Purview (or Unity Catalog in multi-cloud environments).
Automate deployment of Databricks assets (notebooks, jobs, clusters) using Databricks CLI/REST API, ARM/Bicep, or Terraform.
Build and manage CI/CD pipelines in Azure DevOps for data pipelines and infrastructure as code.
Containerize and deploy custom code using Azure Kubernetes Service (AKS) or Databricks Jobs as re

📌 Senior Data Engineer (Jaipur)
🏢 ThoughtsWin
📍 Jaipur