? Key Responsibilities

- Data Pipeline Development & Engineering

● PySpark Mastery (Mandatory): Design, develop, and implement complex ETL/ELT data pipelines using PySpark and Python, focusing on efficiency and scalability.

● Data Transformation: Deep experience in developing data processing tasks, including reading data from various external sources, merging datasets, performing advanced data enrichment, and loading data into target data destinations (Data Warehousing/Data Lake).

● Data Integrity: Hands-on experience with data validation, writing comprehensive unit test cases, and implementing logic for handling Change Data Capture (CDC) operations for high-volume data sets.

● Cloud Operations: Utilize hands-on experience with the AWS ecosystem (e.g., EC2, Lambda) and possess specific expertise in AWS S3 Filesystem operations.

- Optimization, Deployment, and Quality Assurance

● Performance Tuning: Demonstrate expertise in fine-tuning data processes and proactively troubleshooting and resolving performance issues within Spark jobs.

● Deployment & Scheduling: Experience in deployment and operationalizing code, with knowledge of up-to-date scheduling and workflow orchestration tools such as Apache Airflow or NiFi.

● Quality & Standards: Ensure all solutions adhere to development best practices, including robust code quality and adherence to version control.





- Technical Leadership & Documentation

● Design Ownership: Lead the development of comprehensive design documents, including High-Level Design (HLD) and Low-Level Design (LLD) specifications.

● Solution Architecture: Lead requirements gathering sessions and develop end-to-end solution architectures for new Data Migration and Integration initiatives.

● Team & Client Interaction:

● Experience in leading a team or module within a project, providing guidance and technical mentorship.

● Expertise in handling client interactions at various phases of the project lifecycle.

● Be well-versed with the dynamics and challenges of working in an onsite/offshore delivery model.

---

? Required Skills and Qualifications

● Education: BE/ B. Tech/ME/ M. Tech/MCA (or equivalent).

● Experience: 4-6 years of experience in the design and implementation of Data Pipeline projects.

● Mandatory Technical Skills:

● Primary Skill: PySpark and strong Python programming.

● Deep experience with Apache Spark architecture and components.

● Experience with Change Data Capture (CDC) operations.

● Methodology: Operating experience with Agile delivery methodologies.

● Preferred/Added Advantage:

● Working experience on Cloud technology architecture (preferably AWS).

● Knowledge of scheduling tools (Airflow, NiFi).

📌 Data Engineer (Pune)
🏢 CJET Tech Solutions
📍 Pune