Description
Responsibilities
Direct Responsibilities
· Design & develop robust ingestion, transformation, and enrichment pipelines with Python, PySpark, and SQL.
- Write and optimize complex SQL queries, analytical UDFs, and window functions for data aggregation and reporting.
- Collaborate with CEFS data architects, data scientists, and business analysts to translate functional requirements into technical specifications.
- Unit‑test, integrate‑test, and review code.
- Maintain CI/CD pipelines (Git, Jenkins, Docker) for automated build, test, and deployment of jobs.
- Monitor production workloads and troubleshoot performance bottlenecks, memory issues, and job failures.
- Document data lineage, pipeline design, and operational run‑books in Confluence/SharePoint.

· Keep up to date with latest technologies and trends and provide input, expertise and recommendations.
Contributing Responsibilities
· Contribute towards innovation (e.g. AI/ML); suggest new technical practices for efficiency improvement.
- Participate in Agile ceremonies (sprint planning, daily standups, retrospectives)



and help groom the backlogs.
- Mentor junior engineers and champion best practices in Python coding, Spark optimization, and data‑engineering patterns.
- Evaluate emerging technologies and deliver proof‑of‑concepts for CEFS.

Technical & Behavioral Competencies
· Resourceful to quickly understand complexities involved and provide the way forward.
· Good experience in technical analysis of n-tier applications with multiple integrations using object oriented, APIs & Microservices approaches.
· Strong knowledge about design patterns and development principles.
· Inclination and prior experience of working across SQL, Python and ETL.
· Robust Hands-on experience - Python (NumPy, pandas, Python Frameworks, Restful APIs, MS-SQL or Oracle.
· PySpark - DataFrames, Spark SQL, Structured Streaming, performance tuning (partitioning, caching, broadcast joins).
· Advanced SQL – complex queries, stored

📌 Phyton with Spark Developer (Chennai)
🏢 Infotel UK
📍 Chennai