- Independently understand business and technical requirements and translate them into scalable data engineering solutions.
- Take end-to-end ownership of assigned projects, modules and data pipelines with minimal supervision.
- Design, develop, test and deploy robust end-to-end data pipelines on the Azure Data Platform.
- Build scalable and reusable data ingestion and transformation frameworks using Azure Data Factory and Azure Databricks.
- Develop data transformation and processing logic using Python, PySpark and SQL.
- Work with structured, semi-structured and unstructured data from databases, APIs, files and other enterprise data sources.
- Design and implement batch and, where required, near-real-time data processing solutions.
- Develop optimized Databricks workloads using Apache Spark and Delta Lake.
- Perform data analysis, profiling and validation to identify data quality, completeness and consistency issues.
- Implement appropriate data quality checks, error handling, logging and monitoring within data pipelines.
- Troubleshoot data pipeline failures,



performance issues and data discrepancies independently.
- Optimize SQL queries, Spark jobs and data pipelines for performance and scalability.
- Create reusable components and frameworks to improve development efficiency across projects.
- Participate in solution design and technical discussions and provide recommendations on implementation approaches.
- Implement CI/CD processes for data engineering components using Azure DevOps.
- Deploy and manage ADF, Databricks and related Azure data components across development, test and production environments.
- Prepare appropriate technical design, data mapping and operational documentation.
- Collaborate with solution architects, analysts, BI developers, source-system teams and other engineering teams.
- Support production deployments and troubleshoot post-production issues when required.

Requirements

Robust hands-on experience with:

Azure Data Platform
- Azur

📌 Data Engineer (Chennai)
🏢 techcarrot FZ
📍 Chennai