Role: Senior Developer – Up-to-date Data Platform Architect
Role Summary
We are seeking a highly skilled Senior Developer / Data Platform Engineer to design and build scalable data pipelines and contemporary Lakehouse solutions on Databricks-based platforms. The candidate should have strong expertise in PySpark, ETL/ELT frameworks, Data Quality, Apache Iceberg, and job orchestration, with exposure to asset management domain considered a strong advantage.

Key Responsibilities
Data Platform Development & Engineering
Design and develop scalable data ingestion and transformation pipelines using PySpark and SQL.
Build and manage batch and streaming ETL/ELT pipelines on Databricks.
Implement data processing frameworks supporting high-volume enterprise datasets.
Lakehouse Architecture & Iceberg Implementation
Design and maintain up-to-date data Lakehouse architecture (Bronze/Silver/Gold layers).
Work on Apache Iceberg / open table formats including schema evolution, partitioning, and compaction strategies.




Ensure interoperability across engines and optimize data storage formats for performance.
Data Ingestion & Pipeline Management
Design and monitor end-to-end ingestion pipelines (batch + near real-time).
Build reusable ingestion frameworks for structured and semi-structured data.
Manage dependencies, data lineage, and pipeline orchestration workflows.
Data Transformation & dbt (Nice to Have)
Implement transformation logic using dbt or similar ELT frameworks.
Standardize transformation pipelines for maintainability and versioning.
Data Quality & Observability
Implement robust data quality checks, validation rules, and reconciliation processes.
Develop monitoring frameworks for pipeline failures, SLA tracking, and alerting.
Ensure data governance and audit compliance across datasets.
Job Orchestration & Workflow Management
Implement and manage orchestration tools (Airflow / Databricks Jobs / other schedulers).
Handle pipeline schedu

📌 Senior Data Engineer Bengaluru
🏢 State Street
📍 Bengaluru