Independently understand business and technical requirements and translate them into scalable data engineering solutions.
Take end-to-end ownership of assigned projects, modules and data pipelines with minimal supervision.
Design, develop, test and deploy robust end-to-end data pipelines on the Azure Data Platform.
Build scalable and reusable data ingestion and transformation frameworks using Azure Data Factory and Azure Databricks.
Develop data transformation and processing logic using Python, PySpark and SQL.
Work with structured, semi-structured and unstructured data from databases, APIs, files and other enterprise data sources.
Design and implement batch and, where required, near-real-time data processing solutions.
Develop optimized Databricks workloads using Apache Spark and Delta Lake.
Perform data analysis, profiling and validation to identify data quality, completeness and consistency issues.
Implement appropriate data quality checks, error handling, logging and monitoring within data pipelines.
Troubleshoot data pipeline failures,



performance issues and data discrepancies independently.
Optimize SQL queries, Spark jobs and data pipelines for performance and scalability.
Create reusable components and frameworks to improve development efficiency across projects.
Participate in solution design and technical discussions and provide recommendations on implementation approaches.
Implement CI/CD processes for data engineering components using Azure DevOps.
Deploy and manage ADF, Databricks and related Azure data components across development, test and production environments.
Prepare appropriate technical design, data mapping and operational documentation.
Collaborate with solution architects, analysts, BI developers, source-system teams and other engineering teams.
Support production deployments and troubleshoot post-production issues when required.

Requirements
Robust hands-on experience with:
Azure Data Platform
Azure Data Factory (ADF) – pipelines, datasets

📌 Data Engineer (Hyderabad)
🏢 Techcarrot Fz
📍 Hyderabad