Responsibilities

* Understand the values and vision of the organization
* Protect the Intellectual Property
* Adhere to all the policies and procedures
* Design, develop, and maintain scalable data pipelines for data ingestion,
processing and storage.
* Build and optimize data architectures and data models (Lakehouse / medallion,
dimensional) for effective data storage and retrieval.
* Develop ETL/ELT processes to transform and load data from various sources
into data warehouses and data lakes.
* Build and orchestrate pipelines on Azure Databricks using PySpark, Spark SQL
and Delta Lake, orchestrated with Databricks Workflows.
* Integrate data from enterprise source systems including SAP (ABAP/CDS
extracts, RPA/CSV or connectors) and load into Snowflake and Databricks.
* Own end-to-end CI/CD for data pipelines using Databricks Asset Bundles (DAB)
and Azure DevOps (Git repositories, YAML build and release pipelines),
promoting code across dev, QA and production.
* Implement data quality, validation, freshness and reconciliation checks with
pipeline observability.




* Ensure data integrity, quality, and security across all data systems.
* Collaborate with data scientists, analysts, and other stakeholders to
understand data requirements and deliver solutions that meet business needs.
* Monitor and troubleshoot data pipelines and workflows to ensure high
availability and performance.
* Document data processes, architectures, and data flow diagrams.

Essential Skills

Job

* 7 - 8 years of hands-on data engineering experience building and running
production data pipelines at scale.
* Strong expertise in Azure and Azure data services (ADLS Gen2, Azure
Databricks, Azure DevOps).
* Deep hands-on experience with Databricks: PySpark, Spark SQL, Delta Lake,
Lakehouse / medallion architecture and Databricks Workflows.
* CI/CD for data engineering using Databricks Asset Bundles (DAB) and Azure
DevOps (Git, YAML build/release pipelines, multi-environment promotion).
(Must-have)
*

📌 Senior Data Engineer- Remote, India
🏢 The Hackett Group
📍 India