Senior Data Lake Engineer Python / PySpark

Experience: 7+ years

ROLE OVERVIEW

Build and optimize data-lake ingestion, transformation, and curation pipelines for enterprise data platforms. The engineer will work with large datasets and deliver trusted, governed, analytics-ready data using Python, PySpark, Spark, and SQL.

KEY RESPONSIBILITIES

- Develop ingestion and transformation pipelines for batch and/or streaming data using Python and PySpark.
- Process structured and semi-structured data and implement reusable data-lake processing patterns.
- Work with partitioning, file formats, schema evolution, metadata, and data organization across lake layers.
- Implement data quality rules, reconciliation, lineage, access controls, error handling, and monitoring.
- Optimize distributed Spark jobs and troubleshoot pipeline failures and performance bottlenecks.




- Partner with data architecture, platform, analytics, and governance teams to deliver dependable datasets.

REQUIRED SKILLS & EXPERIENCE

- Strong hands-on Python, PySpark, Apache Spark, and SQL skills.
- Experience with data-lake architecture, ETL/ELT, distributed processing, and large datasets.
- Knowledge of cloud storage/data platforms, data quality, and pipeline monitoring.
- Robust debugging, performance tuning, version control, and testing practices.
- Banking/financial-services data experience is preferred.

Work Arrangement: Candidates must be available to work onsite three days per week and collaborate with distributed teams.

📌 Data Lake Engineer (Chennai)
🏢 Sass Innovations
📍 Chennai