Senior Data Engineer with 5+ years of experience
Mandatory Skills:
-Extensive experience in developing ETL pipelines,
- Databricks
-scheduling using Azure Data Factory (ADF) and/or Apache Airflow,
- performance tuning using SQL.
- SQL, Python, Spark, PySpark, and Azure.
- Fabric experience preferred.
Key Responsibilities:
• ETL Pipeline Development:
• Design, develop, and maintain ETL pipelines to extract, transform, and load data from various sources into Azure data platforms.
• Optimize ETL processes for performance and scalability.
• Implement data quality and validation checks within ETL processes.
• Scheduling and Orchestration:
• Develop and manage data integration workflows using Azure Data Factory (ADF) and/or Apache Airflow.
• Schedule and monitor ETL jobs to ensure timely data processing and availability.
• Performance Tuning:
• Optimize SQL queries and data processing tasks for performance.
• Utilize notebooks for development and performance tuning of ETL processes.
• Data Integration and Management:
• Develop and maintain data models and schemas.




• Ensure data integrity, consistency, and security across data platforms.
• Collaborate with data engineers, solution architects, and other stakeholders to understand business requirements and translate them into technical solutions.

Qualifications:
• Bachelor’s degree in computer science, Information Technology, or a related field.
• 5-6 years of experience as a Data Engineer in developing ETL pipelines and scheduling using ADF and/or Apache Airflow.
• Strong technical skills in SQL, Python, Spark, PySpark, and Azure.
• Proven experience in performance tuning and optimization of SQL and ETL processes.
• Robust understanding of Data Architecture and Data Modeling.
• Experience in healthcare and pharmaceutical industries is preferred.
• Excellent problem-solving and analytical skills.
• Robust communication and collaboration skills.

Preferred Skills:
• Knowledge of big data technologies (Hadoop, Hive, Kafka).
•

📌 Senior Data Engineer (Pune)
🏢 NewVision Software
📍 Pune