Job Description:

Responsibilities:

- Develop & Optimize Data Pipelines
- Build, test, and maintain ETL/ELT data pipelines using Azure Databricks & Apache Spark (PySpark).
- Optimize performance and cost-efficiency of Spark jobs.
- Ensure data quality through validation, monitoring, and alerting mechanisms.
- Understand cluster types, configuration, and use-case for serverless

- Implement Unity Catalog for Data Governance

- Design and enforce access control policies using Unity Catalog.
- Manage data lineage, auditing, and metadata governance.
- Enable secure data sharing across teams and external stakeholders.

- Integrate with Cloud Data Platforms

- Work with Azure Data Lake Storage / Azure Blob Storage/ Azure Event Hub to integrate Databricks with cloud-based data lakes, data warehouses, and event streams.
- Implement Delta Lake for scalable, ACID-compliant storage.

- Automate & Orchestrate Workflows

- Develop CI/CD pipelines for data workflows using Azure Databricks Workflows or Azure Data Factory.
- Monitor and troubleshoot failures in job execution and cluster performance.

- Collaborate with Stakeholders

- Work with Data Analysts, Scientists, and Business Teams to understand requirements.
- Translate business needs into scalable data engineering solutions.

- API expertise

- Ability to pull data from a wide variety of APIs using different strategies and methods

Required Skills & Experience:

- Azure Databricks & Apache Spark (PySpark)



– Strong experience in building distributed data pipelines.
- Python – Proficiency in writing optimized and maintainable Python code for data engineering.
- Unity Catalog – Hands-on experience implementing data governance, access controls, and lineage tracking.
- SQL – Strong knowledge of SQL for data transformations and optimizations.
- Delta Lake – Understanding of time travel, schema evolution, and performance tuning.
- Workflow Orchestration – Experience with Azure Databricks Jobs or Azure Data Factory.
- CI/CD & Infrastructure as Code (IaC) – Familiarity with Databricks CLI, Databricks DABs, and DevOps principles.
- Security & Compliance – Knowledge of IAM, role-based access control (RBAC), and encryption.

Preferred Qualifications:

- Experience with MLflow for model tracking & deployment in Databricks.
- Familiarity with streaming technologies (Kafka, Delta Live Tables, Azure Event Hub, Azure Event Grid).
- Hands-on experience with dbt (Data Build Tool) for modular ETL development.
- Certification in Databricks, Azure is a plus.
- Experience with Azure Databricks Lakehouse connectors for SalesForce and SQL Server
- Experience with Azure Synapse Link for Dynamics, dataverse
- Familiarity with other data pipeline strategies, like Azure Functions, Fabric, ADF, etc

Soft Skills:

- Robust problem-solving and debugging skills.
- Ability to work independently and in teams.
- Excellent communication and documentation skills.

📌 Senior Data Engineer (Chennai)
🏢 Toppan Merrill
📍 Chennai