19 Nov
|
Atlas Systems
|
India
19 Nov
Atlas Systems
India
Job Description
Senior Data Engineer / Data Architect
About the role
We are hiring a Senior Data Engineer to build and run the pipelines that bring portfolio company data into one business-aligned model on Databricks. The data comes from a third-party portfolio monitoring platform, with history carried over once from a legacy portfolio system. You will take it through Bronze, Silver and Gold layers and serve it to Power BI reporting and internal applications. You will own pipelines end to end: ingestion, transformation, data quality, performance and day-to-day operation.
What you'll do
• Ingestion: build incremental loads from the source platform into Bronze using change timestamps and watermarks, with safe handling of late-arriving rows and full retention of raw history.
• Silver: model source data close to its native structure, with deterministic deduplication, type-aware unit conversion, sign rules and quarantine of suspect rows rather than silent fixes.
• Gold: implement a source-agnostic business model of dimensions, fact tables with different history rules and reference data, built as performant Delta tables for Power BI and application use.
• Migration: support a single cutover from the legacy system, and load and freeze the history of companies that will not move.
• Data modeling: work with the data model owner on the item catalog, dimensions (period, currency, fund and entity), cash transactions and derived measures such as multiples and FX conversion.
• Data quality and reconciliation: build checks and quality flags, reconcile new figures against the legacy system before the cutover and run regression checks when the model changes.
• Serving: publish Gold to downstream consumers, including Lakebase (Postgres) synced tables for applications and direct Delta access for BI.
• Operations: orchestrate and monitor the pipelines, manage access in Unity Catalog, and keep documentation,
lineage and runbooks current.
• Collaboration: work closely with the vendor's data team, reporting and application teams, and finance and portfolio operations users on definitions and issues. What you'll bring
• 5+ years building production data pipelines, including 2+ years on Databricks with Delta Lake.
• Solid SQL (Spark SQL and T-SQL) and Python or PySpark.
• Hands-on experience with a medallion (Bronze, Silver, Gold) architecture and incremental processing: watermarks, MERGE, change data feed.
• Dimensional modeling: facts at a clear grain, dimensions, surrogate keys, slowly changing dimensions.
• A track record of data quality work: deduplication, reconciliation, and tracing a wrong number back to its source.
• Experience with Unity Catalog or a comparable governance layer: permissions, lineage, documentation.
• Performance tuning for BI workloads, such as file layout, clustering and query plans.
• Clear written communication, including explaining data issues and trade-offs to non-technical users. Nice to have
• Private equity, asset management or fund administration data: portfolio monitoring, valuations, cash flows, fund and co-invest structures.
• Financial reporting concepts: LTM and YTD periods, fiscal year ends, budgets and forecasts, restatements, currency conversion.
• Databricks Lakebase or PostgreSQL, including serving data to applications.
• Lakeflow (Delta Live Tables) or Databricks Workflows orchestration.
• Azure SQL and migrating SQL Server logic to Databricks.
• Power BI data modeling, or working closely with BI developers.
• Infrastructure as code and CI/CD for data pipelines, for example Databricks Asset Bundles. Our environment Area Tools Lakehouse Databricks, Delta Lake, Unity Catalog Languages Spark SQL, Python, PySpark, T-SQL Application serving Databricks Lakebase (Postgres) synced tables Legacy and transition Azure SQL Database Reporting Power BI Cloud Microsoft Azure
📌 GP - Senior Data Engineer / Data Architect (India)
🏢 Atlas Systems
📍 India