Key Responsibilities:

- Design and develop scalable data pipelines using Azure Data Factory (ADF) and Azure Databricks.
- Build and optimize large-scale ETL/ELT processes using PySpark and Python.
- Ingest data from various structured and unstructured sources into Azure Data Lake.
- Develop batch and real-time data processing solutions.
- Create reusable data transformation frameworks and data quality checks.
- Implement Infrastructure as Code (IaC) using Terraform.
- Manage Azure resources including Data Lake Storage, Key Vault, Databricks, ADF, and Monitoring services.
- Optimize Spark jobs for performance, scalability, and cost efficiency.
- Develop CI/CD pipelines for deployment automation.
- Collaborate with business stakeholders, architects, and development teams to define data requirements and solutions.
- Support production deployments and troubleshoot data pipeline issues.
- Follow data governance, security, and compliance standards.

Required Technical Skills: 1) Azure Services: Azure Databricks,



Azure Data Factory (ADF), Azure Data Lake Storage Gen2, Azure Key Vault, Azure Monitor, Azure DevOps

2) Data Engineering: Robust expertise in ETL/ELT development, Data Warehousing concepts, Data Modeling (Star Schema, Snowflake Schema). Batch and Streaming Data Processing

3) Programming: Python, PySpark, SQL

4) Infrastructure as Code: Terraform, Azure Resource Management Concepts

5) Databases: Azure SQL Database, SQL Server, PostgreSQL, Cosmos DB (Preferred)

6)DevOps: Azure DevOps, Git, CI/CD Pipeline Implementation

Mandatory Skills:

- Azure Databricks
- Azure Data Factory (ADF)
- PySpark
- Python
- Terraform
- SQL
- Azure Data Lake Storage (ADLS)
- Azure DevOps

Preferred Skills:

- Delta Lake
- Unity Catalog
- Spark Optimization
- Data Governance
- Real-time Streaming (Kafka/Event Hub)
- Synapse Analytics
- Machine Learning Integration

📌 Azure Databricks, ADF, Pyspark, Python, Terraform (Chennai)
🏢 Tata Consultancy Services
📍 Chennai