Job Summary
Senior Consultant-National-Tech Consulting-CNS - TC - AI and Data - Noida
Responsibilities
- Design, develop, and maintain scalable Python-based microservices for data ingestion and processing.
- Build and optimize microservices to support end-to-end ETL pipelines for large-scale batch and streaming data.
- Develop and manage distributed data processing workflows using Apache Spark and PySpark.
- Create, schedule, and monitor workflows using Apache Airflow.
- Design and deploy data pipelines on AWS EMR or similar big data platforms using Python.
- Implement real-time data ingestion and streaming pipelines.
- Ensure data quality, reliability, performance, and scalability of data systems.
- Apply best practices for code quality, testing, CI/CD, and production monitoring.
- Troubleshoot production issues and perform root-cause analysis.
- 68 years of experience in Data Engineering or backend data platform roles.
- Strong proficiency in Python with experience building production-grade microservices.
- Hands-on expertise with Apache Spark and PySpark.
- Experience working with AWS EMR or equivalent cloud-based big data services. (Must have knowledge on AWS ecosystem)
- Robust experience with Apache Airflow for workflow orchestration.
- Deep understanding of ETL pipeline design and data modeling.
- Hands-on experience with Kafka for inter-domain communication and real-time or streaming data pipelines.
- Strong experience with PostgreSQL or similar RDBMS.
- Experience working in Agile/Scrum environments.
Qualifications
- Bachelor of Technology in Computer Science
- Master of Computer Applications
Experience
- Data Engineering (4+ years)
Skills and Competencies
- Python
- Apache Spark
- PySpark
- Apache Airflow
- AWS EMR
- Kafka
- PostgreSQL
- data modeling
- Agile/Scrum
📌 Senior Consultant-Tech Consulting-National-CNS-TC-AI and Data (Noida)
🏢 EY
📍 Noida