Job Description:
Responsibilities:
Develop & Optimize Data Pipelines
Build, test, and maintain ETL/ELT data pipelines using Azure Databricks & Apache Spark (PySpark).
Optimize performance and cost-efficiency of Spark jobs.
Ensure data quality through validation, monitoring, and alerting mechanisms.
Understand cluster types, configuration, and use-case for serverless
Implement Unity Catalog for Data Governance
Design and enforce access control policies using Unity Catalog.
Manage data lineage, auditing, and metadata governance.
Enable secure data sharing across teams and external stakeholders.
Integrate with Cloud Data Platforms
Work with Azure Data Lake Storage / Azure Blob Storage/ Azure Event Hub to integrate Databricks with cloud-based data lakes, data warehouses, and event streams.
Implement Delta Lake for scalable, ACID-compliant storage.
Automate & Orchestrate Workflows
Develop CI/CD pipelines for data workflows using Azure Databricks Workflows or Azure Data Factory.
Monitor and troubleshoot failures in job execution and cluster performance.




Collaborate with Stakeholders
Work with Data Analysts, Scientists, and Business Teams to understand requirements.
Translate business needs into scalable data engineering solutions.
API expertise
Ability to pull data from a wide variety of APIs using different strategies and methods

Required Skills & Experience:
Azure Databricks & Apache Spark (PySpark) – Solid experience in building distributed data pipelines.
Python – Proficiency in writing optimized and maintainable Python code for data engineering.
Unity Catalog – Hands-on experience implementing data governance, access controls, and lineage tracking.
SQL – Solid knowledge of SQL for data transformations and optimizations.
Delta Lake – Understanding of time travel, schema evolution, and performance tuning.
Workflow Orchestration – Experience with Azure Databricks Jobs or Azure Data Factory.
CI/CD & Infrastructure

📌 Senior Data Engineer Chennai
🏢 Toppan Merrill
📍 Chennai