Job Summary
Senior Consultant-National-Tech Consulting-CNS - TC - AI and Data - Noida
Responsibilities
Design, develop, and maintain scalable Python-based microservices for data ingestion and processing.
Build and optimize microservices to support end-to-end ETL pipelines for large-scale batch and streaming data.
Develop and manage distributed data processing workflows using Apache Spark and PySpark.
Create, schedule, and monitor workflows using Apache Airflow.
Design and deploy data pipelines on AWS EMR or similar big data platforms using Python.
Implement real-time data ingestion and streaming pipelines.
Ensure data quality, reliability, performance, and scalability of data systems.
Apply best practices for code quality, testing, CI/CD, and production monitoring.
Troubleshoot production issues and perform root-cause analysis.
68 years of experience in Data Engineering or backend data platform roles.
Robust proficiency in Python with experience building production-grade microservices.
Hands-on expertise with Apache Spark and PySpark.
Experience working with AWS EMR or equivalent cloud-based big data services. (Must have knowledge on AWS ecosystem)
Robust experience with Apache Airflow for workflow orchestration.
Deep understanding of ETL pipeline design and data modeling.
Hands-on experience with Kafka for inter-domain communication and real-time or streaming data pipelines.
Robust experience with PostgreSQL or similar RDBMS.
Experience working in Agile/Scrum environments.
Qualifications
Bachelor of Technology in Computer Science
Master of Computer Applications
Experience
Data Engineering (4+ years)
Skills and Competencies
Python
Apache Spark
PySpark
Apache Airflow
AWS EMR
Kafka
PostgreSQL
data modeling
Agile/Scrum
📌 Senior Consultant Tech Consulting National Cns Tc Ai And Data Noida
🏢 EY
📍 Noida