Role: Senior Developer – Modern Data Platform Architect
Role Summary
We are seeking a highly skilled Senior Developer / Data Platform Engineer to design and build scalable data pipelines and modern Lakehouse solutions on Databricks-based platforms. The candidate should have strong expertise in PySpark, ETL/ELT frameworks, Data Quality, Apache Iceberg, and job orchestration, with exposure to asset management domain considered a strong advantage.

Key Responsibilities
1. Data Platform Development & Engineering
- Design and develop scalable data ingestion and transformation pipelines using PySpark and SQL.
- Build and manage batch and streaming ETL/ELT pipelines on Databricks.
- Implement data processing frameworks supporting high-volume enterprise datasets.

2. Lakehouse Architecture & Iceberg Implementation
- Design and maintain up-to-date data Lakehouse architecture (Bronze/Silver/Gold layers).
- Work on Apache Iceberg / open table formats including schema evolution, partitioning, and compaction strategies.




- Ensure interoperability across engines and optimize data storage formats for performance.

3. Data Ingestion & Pipeline Management
- Design and monitor end-to-end ingestion pipelines (batch + near real-time).
- Build reusable ingestion frameworks for structured and semi-structured data.
- Manage dependencies, data lineage, and pipeline orchestration workflows.

4. Data Transformation & dbt (Nice to Have)
- Implement transformation logic using dbt or similar ELT frameworks.
- Standardize transformation pipelines for maintainability and versioning.

5. Data Quality & Observability
- Implement robust data quality checks, validation rules, and reconciliation processes.
- Develop monitoring frameworks for pipeline failures, SLA tracking, and alerting.
- Ensure data governance and audit compliance across datasets.

6. Job Orchestration & Workflow Management
- Implement and manage orchestration tools (Airflow / Databricks Jobs / other schedulers).
- Handle pipeline schedu

📌 Senior Data Engineer (Bengaluru)
🏢 State Street
📍 Bengaluru