Job Purpose

We are seeking a skilled and motivated Data Engineer to design, develop, deploy, and support secure, scalable, and high-performance data pipelines on the Cloudera Data Platform (CDP). The role involves building robust data solutions using PySpark, Hive, SQL, and Apache Oozie, while ensuring data quality, governance, security, and compliance with enterprise standards.

The successful candidate will collaborate closely with Solution Architects, Senior Data Engineers, Product Owners, and business stakeholders to deliver reliable and production-ready data assets that support strategic analytics, risk, and compliance initiatives across the organization.

Key Responsibilities

- Design, develop, and maintain scalable batch data pipelines using PySpark, Hive, SQL, and Apache Oozie.
- Build and support production-grade data processing solutions on Cloudera Data Platform (CDP).
- Analyze business requirements and convert them into technical designs, source-to-target mappings, and reusable data solutions.
- Develop and optimize PySpark applications for large-scale distributed data processing.
- Implement data quality checks, reconciliation processes, and governance controls.
- Monitor, troubleshoot, and enhance data pipeline performance, reliability, and operational efficiency.
- Collaborate with cross-functional teams to ensure successful delivery of data products and services.
- Participate in code reviews, testing, deployment activities, and release management processes.
- Maintain technical documentation and ensure adherence to enterprise engineering standards.




- Leverage AI-assisted development tools such as Claude Code in compliance with organizational policies and security requirements.

Required Skills

Data Engineering

- Solid hands-on experience with PySpark, Python, Hive, and SQL.
- Experience developing and maintaining enterprise-scale data pipelines.
- Solid understanding of distributed data processing concepts and Big Data technologies.
- Experience with Apache Oozie workflow development, scheduling, monitoring, and troubleshooting.
- Working knowledge of Cloudera Data Platform (CDP) ecosystem.

Data Warehousing & Modeling

- Strong understanding of data warehousing concepts and best practices.
- Experience with Data Vault 2.0 and Dimensional Modeling techniques.
- Knowledge of schema design, data lineage, metadata management, and data governance.

DevOps & CI/CD

- Experience with Git, Jenkins, and CI/CD implementation.
- Familiarity with version control, code reviews, and release management processes.
- Exposure to GitHub Actions is desirable.

Data Quality & Governance

- Experience implementing data quality frameworks and controls.
- Understanding of metadata management and governance practices.
- Ability to ensure secure and compliant handling of enterprise data.

Preferred Skills

- Banking or Financial Services domain experience.
- Informatica BDM.
- Cloud-native data platform experience.
- Exploratory Data Analysis (EDA).
- Power BI reporting and visualization.
- Airflow orchestration.
- Feature Engineering for analytics or machine learning use cases.
- Exposure to AI-assisted coding tools such as Claude Code.

Educational Qualification

- Bachelor's or Master's Degree in:
- Computer Science
- Information Technology
- Information Systems
- Engineering
- Related Technical Discipline

📌 Data Engineer (Chennai)
🏢 ValueLabs
📍 Chennai