- Independently understand business and technical requirements and translate them into scalable data engineering solutions.
- Take end-to-end ownership of assigned projects, modules and data pipelines with minimal supervision.
- Design, develop, test and deploy robust end-to-end data pipelines on the Azure Data Platform.
- Build scalable and reusable data ingestion and transformation frameworks using Azure Data Factory and Azure Databricks.
- Develop data transformation and processing logic using Python, PySpark and SQL.
- Work with structured, semi-structured and unstructured data from databases, APIs, files and other enterprise data sources.
- Design and implement batch and, where required, near-real-time data processing solutions.
- Develop optimized Databricks workloads using Apache Spark and Delta Lake.
- Perform data analysis, profiling and validation to identify data quality, completeness and consistency issues.
- Implement appropriate data quality checks, error handling, logging and monitoring within data pipelines.
- Troubleshoot data pipeline failures, performance issues and data discrepancies independently.
- Optimize SQL queries, Spark jobs and data pipelines for performance and scalability.
- Create reusable components and frameworks to improve development efficiency across projects.
- Participate in solution design and technical discussions and provide recommendations on implementation approaches.
- Implement CI/CD processes for data engineering components using Azure DevOps.
- Deploy and manage ADF, Databricks and related Azure data components across development, test and production environments.
- Prepare appropriate technical design,



data mapping and operational documentation.
- Collaborate with solution architects, analysts, BI developers, source-system teams and other engineering teams.
- Support production deployments and troubleshoot post-production issues when required.

Requirements

Strong hands-on experience with:

Azure Data Platform

- Azure Data Factory (ADF) – pipelines, datasets, linked services, triggers, parameterization and reusable frameworks

- Azure Databricks
- Azure Data Lake Storage Gen2 (ADLS Gen2)
- Azure Synapse Analytics / Azure SQL

- Azure Key Vault
- Azure Monitor / Log Analytics or equivalent monitoring capabilities
- Azure Event Hubs – exposure/experience is preferred

Databricks & Data Processing
- Strong hands-on experience with Databricks and Apache Spark

- Strong PySpark development skills
- Experience with Delta Lake / Delta tables
- Understanding of Medallion or similar layered data architectures
- Experience implementing incremental and full-load processing patterns
- Ability to troubleshoot and optimize Spark workloads
- Experience with Databricks Workflows/Jobs
- Knowledge of Unity Catalog is preferred

SQL & Python
- Strong SQL development skills
- Ability to write and optimize complex SQL queries
- Experience with joins, CTEs,



window functions and analytical queries
- Strong working knowledge of Python
- Ability to develop reusable Python/PySpark modules and utilities

DevOps / CI-CD
- Hands-on experience with Azure DevOps
- Git-based source control
- Branching and code-management practices
- CI/CD implementation for Azure Data Factory and Databricks
- Experience deploying solutions across multiple environments

Qualifications & Experience

- 4–6 years of overall experience in Data Engineering / Data Platform development.
- Preferably 4+ years of hands-on experience with Microsoft Azure Data Platform technologies.
- Strong hands-on experience with Azure Data Factory, Databricks, SQL and Python/PySpark.
- Bachelor's degree in Computer Science, Information Technology, Engineering or a related discipline.
- Experience delivering at least one or more end-to-end Azure data engineering projects.
- Good understanding of data warehousing, data lake and modern data platform concepts.
- Robust analytical and problem-solving skills.
- Ability to work independently and manage multiple technical activities.
- Good written and verbal communication skills.

​Preferred Skills

- Microsoft Azure / Databricks certifications.
- Experience with Unity Catalog and Databricks governance.
- Experience with REST API-based data ingestion.
- Knowledge of dimensional modelling and data warehouse concepts.
- Experience working with Power BI or downstream analytics platforms.
- Exposure to streaming/event-driven data processing.
- Experience working in enterprise-scale Azure environments.

📌 Data Engineer (Hyderabad)
🏢 Techcarrot Fz
📍 Hyderabad