Job Description: Senior Data Engineer – Data Ingestion & Platforms

Role Overview

We are seeking a seasoned Senior Data Engineer with a robust software engineering mindset to design,

build, and optimize our next-generation Ingest Factory and Data Processing Frameworks. In this role,

you will go beyond traditional ETL scripting to build scalable, metadata-driven pipelines and reusable

data frameworks.

The ideal candidate possesses deep expertise in Python, PySpark, and the Databricks Lakehouse

ecosystem (including LakeFlow and Delta Lake), combined with rigorous software engineering

discipline (SOLID, CI/CD, and infrastructure as code). You will work both independently and

collaboratively within an Agile environment to build production-grade software that ensures data quality,

governance, and seamless orchestration.

Key Responsibilities

Architecture & Pipeline Engineering

• Ingest Factory Design: Design, develop, and maintain robust data ingestion frameworks

leveraging Databricks LakeFlow, managed connectors, and declarative pipelines.

• Data Lakehouse Patterns:



Implement repeatable ETL/ELT patterns within a Delta Lake

architecture, ensuring optimized storage, table design, and strict data lineage enforcement.

• Metadata-Driven Orchestration: Build parameterized notebooks and end-to-end orchestration

flows to automate ingestion across diverse source system patterns.

Software Craftsmanship & Automation

• Advanced Python Development: Write clean, modular, and maintainable Python code applying

SOLID and DRY principles. Move beyond basic PySpark scripting to contribute to and publish

reusable internal packages (e.g., PyPI).

• Framework Creation: Define reusable functions and framework-level abstractions to

dramatically improve development efficiency across the data team.

• Testing & Quality Assurance: Implement rigorous data quality checks, monitoring, and alerting

frameworks. Lead test practices including Unit, Integration, and End-to-End (E2E)

📌 Senior Data Engineer (Nagpur)
🏢 Headsup Corporation
📍 Nagpur