Overall Experience: 6 to 12 Years

Job Location: Bangalore/Hyderabad/Chennai

Job Requirements*

- Azure data factory, Databricks, synapse, delta lake development with hands-on coding experience
- Implement ETL solution to integrate, transform and load data from various sources into data lake and data warehouse
- Hands-on expertise in python, pyspark and sql for large scale data processing
- Optimize and tune data pipelines for performance and scalability
- Ability to write complex SQL queries
- Collaborate with business analysts and business stakeholders to gather requirements and ensure data quality and availability.
- Good understanding of Agile Methodologies and DevOps Culture
- Solid Problem-solving skills

.

Key Responsibilities*

- Design, develop, and deploy scalable data pipelines using Databricks (PySpark, Spark SQL), Azure Synapse, Azure Data Factory, and other Azure data services.
- Implement ETL/ELT processes to ingest, transform, and load data from various sources into data lakes and data warehouses.
- Optimize and tune data pipelines for performance and scalability.




- Write and optimize complex SQL queries for data extraction, transformation, and analysis.
- Use PySpark for large-scale data processing and analytics.
- Implement data partitioning, bucketing, z-ordering, liquid clustering and indexing strategies for efficient data retrieval.
- Integrate data from multiple sources, including structured, semi-structured, and unstructured data.
- Work with APIs, streaming data, and batch processing to ensure seamless data integration.
- Implement data governance practices to ensure data quality, consistency, and security.
- Monitor and troubleshoot data pipelines to ensure data accuracy and availability.
- Collaborate with data scientists, analysts, and other stakeholders to understand data requirements and deliver solutions.
- Work closely with DevOps teams to deploy and monitor data pipelines in production environments.
- Document data pipelines, workflows, and processes for knowledge sharing and future reference.
- Maintain up-to-date documentation on data architecture and data models.

📌 Azure Data Engineer (Chennai)
🏢 Tata Consultancy Services
📍 Chennai