Description
We are looking for an experienced Data Engineer to join our team in India. The ideal candidate will be responsible for building and maintaining data pipelines and architectures, ensuring data quality, and collaborating with data teams to meet their data needs.
To be successful in this role, you should meet the following requirements:
Should have at least 5+ years experience working in data engineering.
(Note: Must have Requirements)
Scala and Python development and understanding requirements and come up solutions.
Experience using scheduling tools such as Airflow.
Experience with most of the following technologies (Apache Hadoop, Pyspark, Apache Spark, YARN, Hive, Python, ETL frameworks, Map Reduce, SQL, RESTful services).
Sound knowledge of working in Unix/Linux Platform
Hands-on experience building data pipelines using Hadoop components - Hive, Spark, Spark SQL.
Experience with industry standard version control tools (Git, GitHub), automated deployment tools (Ansible & Jenkins) and requirement management in JIRA.




Understanding of big data modelling techniques using relational and non-relational techniques
Experience on debugging code issues and then publishing the highlighted differences to the development team.
Adaptable to adapt to new tooling.

Job responsibilities
Software design, Scala and Python - Spark development experience , automated testing of current and existing components in an Agile, DevOps and agile environment.
Minimum 1 year experience in Scala
Promoting development standards, code reviews, mentoring, knowledge sharing
Production support & troubleshooting.
Implement the tools and processes, handling performance, scale, availability, accuracy and monitoring
Liaison with BAs to ensure that requirements are correctly interpreted and implemented.
Participation in regular planning and status meetings. Input to the development process through the involvement in Sprint reviews and retrospectives. Input into syste

📌 Data Engineer Pune
🏢 Purview India Consulting And Services
📍 Pune