Senior Data Engineer / Data Engineer (PySpark)

The Senior Data Engineer / Data Engineer is responsible for designing, developing, deploying, and supporting secure, scalable, and reliable data processing pipelines using PySpark, Apache Oozie, Hive, SQL, and distributed data platforms.

The role collaborates with business stakeholders, product owners, data architects, platform teams, and downstream consumers to translate business requirements into reusable, governed, and production-ready data solutions.

Senior-level professionals are expected to provide technical leadership, define engineering standards, review solution designs, mentor team members, and drive automation and continuous improvement initiatives.

Key Responsibilities

Data Pipeline Design & Development

- Design, develop, test, and deploy large-scale batch data pipelines using PySpark, Hive, SQL, and Apache Oozie.
- Build reusable ingestion, transformation, validation, reconciliation, and publishing frameworks.
- Analyze data requirements including source systems, transformation rules, volume, latency, and retention needs.
- Participate in backlog refinement, sprint planning, testing, release management, and production validation.
- Ensure high reliability, scalability, and maintainability of data pipelines.

Data Modeling & Performance Optimization
- Design logical and physical data models aligned with enterprise architecture standards.
- Develop optimized PySpark applications and Hive/SQL transformations.
- Improve performance through partitioning, bucketing, caching, parallelism, and resource optimization.
- Manage schema evolution, metadata, data lineage, and business rules documentation.
- Deliver highly performant and cost-productive data solutions.





Data Governance, Security & Quality
- Implement security controls including authentication, authorization, encryption, and data masking.
- Apply data governance standards covering metadata, lineage, retention, and data classification.
- Develop automated unit tests, integration tests, and data quality validations.
- Monitor workflows, Spark jobs, and platform services.
- Conduct troubleshooting, root-cause analysis, and production support activities.

Stakeholder Collaboration
- Work closely with business teams to understand and prioritize data requirements.
- Translate requirements into source-to-target mappings and technical specifications.
- Collaborate with architects, engineers, governance teams, and application teams.
- Maintain design documents, operational runbooks, release notes, and knowledge transfer materials.

Technical Leadership & Continuous Improvement (For Senior Data Engineers)

- Define engineering standards, architecture guidelines, and reusable frameworks.
- Conduct design reviews and code reviews.
- Drive CI/CD, GitOps, and automation best practices.
- Mentor junior engineers and support capability development.
- Promote adoption of AI-assisted development tools while ensuring security and governance compliance.

Required Qualifications

Education

- Bachelor's or Master's degree in Computer Science,



Information Technology, Information Systems, or a related field.

Experience

Senior Data Engineer

- 6+ years of data engineering experience.
- 4+ years of hands-on experience with PySpark, Hive, SQL, workflow orchestration, and distributed data platforms.
- Experience leading technical designs, code reviews, and complex project deliveries.

Data Engineer
- 3+ years of data engineering or software development experience.
- Hands-on experience building data pipelines using PySpark, Hive, Oozie, and SQL.
- Experience working on distributed data platforms.

Preferred:

- Banking or Financial Services domain experience.
- Agile development methodologies.
- Git-based development practices.
- Production support experience.

Technical Skills

Must Have

- Python
- PySpark
- Apache Oozie
- Hive / HiveQL
- SQL
- Distributed Data Platforms

Additional Expertise
- Data Modeling & Data Warehousing
- ETL/ELT Development
- Data Lakes
- Performance Tuning & Optimization
- Schema Design & Evolution
- Metadata Management & Data Lineage
- Git, CI/CD & GitOps Practices
- Automated Testing Frameworks
- Workflow Scheduling & Monitoring

Nice to Have
- Cloudera Data Platform (CDP)
- AI-assisted development tools (Claude Code/Copilot)
- Release Management & Deployment Automation

Behavioral Competencies
- Strong business acumen
- Excellent analytical and problem-solving skills
- Critical and systems thinking
- Leadership and mentoring capabilities
- Stakeholder management and influencing skills
- Organized and disciplined approach
- Adaptability and flexibility
- Performance-driven mindset
- Strong communication and collaboration skills

📌 Senior Data Engineer (Chennai)
🏢 ValueLabs
📍 Chennai