Senior Data Lake Engineer Python / PySpark

Experience: 7+ years

ROLE OVERVIEW

Build and optimize data-lake ingestion, transformation, and curation pipelines for enterprise data platforms. The engineer will work with large datasets and deliver trusted, governed, analytics-ready data using Python, PySpark, Spark, and SQL.

KEY RESPONSIBILITIES
Develop ingestion and transformation pipelines for batch and/or streaming data using Python and PySpark.
Process structured and semi-structured data and implement reusable data-lake processing patterns.
Work with partitioning, file formats, schema evolution, metadata, and data organization across lake layers.
Implement data quality rules, reconciliation, lineage, access controls, error handling, and monitoring.
Optimize distributed Spark jobs and troubleshoot pipeline failures and performance bottlenecks.




Partner with data architecture, platform, analytics, and governance teams to deliver dependable datasets.

REQUIRED SKILLS & EXPERIENCE
Solid hands-on Python, PySpark, Apache Spark, and SQL skills.
Experience with data-lake architecture, ETL/ELT, distributed processing, and large datasets.
Knowledge of cloud storage/data platforms, data quality, and pipeline monitoring.
Robust debugging, performance tuning, version control, and testing practices.
Banking/financial-services data experience is preferred.

Work Arrangement: Candidates must be available to work onsite three days per week and collaborate with distributed teams.

📌 Data Lake Engineer Chennai
🏢 Sass Innovations
📍 Chennai