We are looking for a skilled and experienced Senior Data Engineer with strong expertise in the Azure Data Engineering ecosystem. The ideal candidate should have hands-on experience in designing and developing scalable data pipelines, batch and real-time data processing, and working with Azure data services.
The candidate will be responsible for building robust data solutions, collaborating with cross-functional teams, and contributing to end-to-end data engineering projects.
Key Responsibilities
- Design, develop, and maintain scalable data pipelines and data processing solutions on Microsoft Azure.
- Develop and manage ETL/ELT pipelines using Azure Data Factory (ADF).
- Work extensively with Azure Data Lake Storage (ADLS) and Azure Databricks.
- Implement batch and real-time data ingestion and processing solutions.
- Develop data transformation solutions using PySpark / Spark / Scala.
- Work with Delta Lake and Spark Structured Streaming.
- Design and work with Azure Synapse Analytics / Synapse Workspace.
- Work with Azure SQL Data Warehouse / Azure Synapse and Cosmos DB.
- Develop and maintain data models and optimize data processing workflows.
- Use Python and Shell scripting for automation and data engineering tasks.
- Work with Azure DevOps for source control, CI/CD, and deployment processes.
- Understand and work with basic Azure infrastructure and cloud concepts.
- Participate in end-to-end implementation of data engineering projects.
- Troubleshoot performance, data quality, and pipeline-related issues.
- Collaborate with technical teams, stakeholders, and clients to understand requirements and deliver effective data solutions.
Required Skills
- Minimum 2.5 years of relevant experience in Data Engineering.
- Strong hands-on experience with:
- Azure Data Factory
- Azure Data Lake / ADLS
- Azure Databricks
- Azure Synapse Analytics
- PySpark / Spark / Scala
- Delta Lake
- Spark Streaming
- Strong understanding of batch and real-time data processing.
- Good knowledge of Azure SQL / SQL Data Warehouse and Cosmos DB.
- Good programming experience in Python.
- Knowledge of Shell scripting.
- Understanding of Azure DevOps, CI/CD, and Azure infrastructure concepts.
- Strong SQL and data manipulation skills.
- Ability to work independently as well as collaboratively in a team environment.
- Solid analytical, problem-solving, and communication skills.
Good to Have
- Azure or other relevant cloud certifications.
- Experience working on an end-to-end Azure Data Engineering project.
- Experience interacting directly with clients or stakeholders.
- Knowledge of data architecture and scalable cloud-based data solutions.
📌 Senior Data Engineer (Noida)
🏢 ThoughtSol Infotech
📍 Noida
Reply to this offer
Impress this employer describing Your skills and abilities, fill out the form below and leave Your personal touch in the presentation letter.