Role - Data Engineer
Location - Remote(Pan India)
Position Summary
Deloitte is seeking a Data Engineer to support the Omnia Data platform, a fintech data-processing solution that enables users to upload datasets through a web application and receive processed data products and Excel-based reports.
The Data Engineer will design, develop, operate, troubleshoot, and optimize Databricks data pipelines that ingest user-uploaded datasets, apply data transformations and validations, publish data to Unity Catalog, and generate downstream reporting outputs. The role primarily supports Databricks jobs running on serverless compute, while also requiring working knowledge of classic clusters for workload-specific processing, compatibility, troubleshooting, and performance analysis.
The successful candidate will combine solid PySpark and SQL skills with practical experience managing production job runs, diagnosing failures, improving pipeline performance, and working with financial-services data.
Key Responsibilities
Develop scalable data pipelines
- Design and maintain data pipelines using Python, PySpark, Apache Spark, and SQL.
- Build ingestion and transformation processes for datasets uploaded through the Omnia Data web application.
- Implement medallion architecture patterns across Bronze, Silver, and Gold data layers.
- Create reusable, modular, and testable transformation logic for structured and semi-structured data.
- Apply schema validation, data standardization, deduplication, enrichment, and business-rule processing.
- Build curated datasets that support downstream analysis, reporting, and financial-services use cases.
- Ingest processed data into governed tables and objects within Databricks Unity Catalog.
Manage and support Databricks job runs
- Configure, monitor, and maintain Databricks workflows and scheduled job runs.
- Manage task dependencies, parameters, retries, alerts, execution order, and operational handoffs.
- Investigate failed or incomplete job run
📌 Data Engineer (Pune)
🏢 NewVison
📍 Pune