Job Title
Big Data Engineer (EL3)
Experience Required
4+ Years
Employment Type
Full-Time
Job Summary
We are looking for an experienced Big Data Engineer to design, develop, and optimize scalable big data pipelines and data validation solutions. The ideal candidate should have strong hands-on experience in Scala, PySpark, AWS, and AWS Glue, along with expertise in data validation, profiling, and modern big data technologies. The role also involves leveraging AI-assisted development tools to improve engineering productivity and software quality.
Required Skills & Experience
- 4+ years of software engineering experience developing Big Data pipelines.
- 3+ years of hands-on programming experience using Scala and PySpark.
- 3+ years of experience building cloud-based data solutions on AWS, including AWS Glue.
- 3+ years of experience working with Big Data file formats such as Parquet.
- 1+ years of experience in data validation, data profiling, or developing rule-based data validation engines.
- 1+ years of experience using AI-powered coding assistants or code-generation tools such as GitHub Copilot, OpenAI Codex, or similar tools in daily software development.
Key Responsibilities
- Design, develop, and optimize scalable, rule-based data validation engines capable of validating data across diverse data pipelines.
- Build robust, high-performance Big Data pipelines and processing platforms using Scala, PySpark, AWS, and AWS Glue.
- Develop clean, maintainable, and well-tested code while utilizing optimized storage formats such as Parquet for effective data storage and retrieval.
- Perform comprehensive data validation, data profiling, and quality assurance to ensure data accuracy, consistency, and reliability across multiple data sources.
- Utilize enterprise-approved AI development tools to streamline software development, automate repetitive tasks, and improve engineering productivity.
- Collaborate with cross-functional teams including Business Analysts, Data
📌 Big Data Engineer (Noida)
🏢 UIDM
📍 Noida