- Develop scalable pipelines using Databricks, Apache Spark, PySpark, SQL and Delta Lake.
- Build Bronze, Silver and Gold pipelines following the approved architecture.
- Implement source ingestion, cleansing, transformation, enrichment and conformance.
- Develop reusable pipeline components for multiple sources.
- Implement incremental, historical and schema-evolution processing.
- Build portfolio accounting/custody data products covering holdings, tax lots, transactions, GL, cash, accruals and corporate actions.
- Process benchmark/index data and integrate Security Master and Account Master data.
- Implement enterprise identifier matching and cross-references.
- Implement data-quality validation, reconciliation, exception handling and certification.
- Support up to 10 years of historical backfill with reconciliation and auditability.
- Optimize Spark jobs, Delta tables and SQL workloads.
- Develop unit/integration tests and support SIT, UAT, production deployment and stabilization.
- Contribute to documentation and knowledge transfer.

Required Experience
- 712+ years in data engineering.
- 4+ years hands-on Databricks/Spark experience.
- Strong Databricks, Apache Spark, PySpark, SQL and Delta Lake skills.




- Enterprise-scale batch and/or streaming pipeline experience.
- Strong medallion architecture knowledge.
- Data transformation, cleansing, enrichment and conformance experience.
- Data-quality and reconciliation framework experience.
- Large historical dataset experience.
- Solid debugging and performance-tuning skills.

Financial Services Experience Strong preference for investment management, asset management, portfolio accounting, custody, ABOR/IBOR, securities, benchmarks/indexes, Security Master or Account Master.

Preferred Databricks Skills

Databricks Jobs/Workflows, Lakeflow/declarative pipelines, Unity Catalog, Databricks SQL, Delta optimization, schema evolution, SCD, lineage, data-quality frameworks, metadata-driven ingestion and large-scale historical backfill.

Out of Scope

Azure infrastructure, VNet/Private Link/networking, Terraform, Azure landing zone, CI/CD platform engineering, Databricks workspace provisioning and infrastructure operations.

Education

Bachelor's or Master's degree in Computer Science, Engineering, Information Technology, Data Engineering or related discipline.

📌 Databricks Engineer/Lead (Bengaluru)
🏢 Impetus Technologies
📍 Bengaluru