Job Summary

We are seeking an experienced Big Data Engineer (EL3) to design, develop, and optimize scalable big data solutions and data validation platforms. The ideal candidate will have solid expertise in Scala, PySpark, AWS, AWS Glue, and Parquet, with hands-on experience building high-performance data pipelines. This role involves developing rule-based data validation engines, ensuring data quality, and leveraging AI-assisted development tools to improve productivity and innovation.

Requirements
- Bachelor's degree in Computer Science, Engineering, or a related field.
- 4+ years of software engineering experience developing big data pipelines.
- 3+ years of hands-on programming experience with Scala and PySpark.
- 3+ years of experience building solutions on AWS, including AWS Glue.
- 3+ years of experience working with big data file formats, especially Parquet.
- 1+ year of experience in data validation, data profiling, or developing rule-based data engines.
- 1+ year of experience using AI coding assistants or code-generation tools (e.g., GitHub Copilot, Codex)



as part of software development.

Key Responsibilities
- Design, build, and optimize a highly scalable, rule-based data validation engine capable of validating data across multiple data pipelines.
- Develop robust, high-performance big data pipelines using Scala, PySpark, and AWS Glue.
- Write clean, efficient, and well-tested code utilizing optimized data formats such as Parquet for high-performance storage and retrieval.
- Perform data validation and profiling to ensure data quality, consistency, and integrity across diverse data systems.
- Utilize enterprise-approved AI development tools to streamline software development, automate repetitive tasks, and improve engineering productivity.
- Collaborate with cross-functional teams to translate business requirements into scalable technical solutions.
- Evaluate emerging AI technologies and big data trends to enhance data validation frameworks and solu

📌 Big Data Engineer (Noida)
🏢 WebSenor InfoTech
📍 Noida