Senior Data Engineer with 5+ years of experience

Mandatory Skills

-Extensive experience in developing ETL pipelines,

- Databricks

-scheduling using Azure Data Factory (ADF) and/or Apache Airflow,

- performance tuning using SQL.
- SQL, Python, Spark, PySpark, and Azure.
- Fabric experience preferred.

Key Responsibilities

- ETL Pipeline Development:

- Design, develop, and maintain ETL pipelines to extract, transform, and load data from various sources into Azure data platforms.
- Optimize ETL processes for performance and scalability.
- Implement data quality and validation checks within ETL processes.

- Scheduling and Orchestration:

- Develop and manage data integration workflows using Azure Data Factory (ADF) and/or Apache Airflow.
- Schedule and monitor ETL jobs to ensure timely data processing and availability.

- Performance Tuning:

- Optimize SQL queries and data processing tasks for performance.
- Utilize notebooks for development and performance tuning of ETL processes.

- Data Integration and Management:

- Develop and maintain data models and schemas.
- Ensure data integrity, consistency,



and security across data platforms.
- Collaborate with data engineers, solution architects, and other stakeholders to understand business requirements and translate them into technical solutions.

Qualifications:
- Bachelor’s degree in computer science, Information Technology, or a related field.
- 5-6 years of experience as a Data Engineer in developing ETL pipelines and scheduling using ADF and/or Apache Airflow.
- Strong technical skills in SQL, Python, Spark, PySpark, and Azure.
- Proven experience in performance tuning and optimization of SQL and ETL processes.
- Strong understanding of Data Architecture and Data Modeling.
- Experience in healthcare and pharmaceutical industries is preferred.
- Excellent problem-solving and analytical skills.
- Robust communication and collaboration skills.

Preferred Skills:
- Knowledge of big data technologies (Hadoop, Hive, Kafka).
- Familiarity with data visualization tools (Power BI, Tableau).
- Microsoft Fabric experience.

📌 Senior Data Engineer (Pune)
🏢 NewVison
📍 Pune