Title: Senior Data Engineer / Data Architect Azure Databricks
Location: Pan India
Work Arrangement: Hybrid
Experience: 10+ Years
Joining Preference: Immediate Joiners / Candidates Serving Notice Period

Job Summary

Innova Software Services Pvt. Ltd. is seeking an experienced Senior Data Engineer / Data Architect with strong expertise in Azure Databricks, data lakehouse architecture, PySpark, Delta Lake, and enterprise data engineering.

The successful candidate will lead the modernization of an existing Azure Blob Storage-based reporting platform into a scalable Databricks Lakehouse solution.

This role involves designing enterprise data architecture, developing reliable ETL/ELT pipelines, improving data quality, and optimizing curated datasets for Power BI reporting and future analytics requirements.

Key Responsibilities

Databricks Lakehouse Architecture

- Design and implement scalable Databricks Lakehouse architecture using Bronze, Silver, and Gold layers.

- Define enterprise data architecture standards and transformation patterns.

- Design reliable and maintainable data ingestion and processing frameworks.

- Establish appropriate data modeling and warehousing practices.

Data Migration and Modernization

- Assess existing Azure Blob Storage data processes and reporting architecture.

- Develop migration strategies for transitioning data workloads to Azure Databricks.

- Design and implement scalable data ingestion and transformation pipelines.

- Support end-to-end migration planning, execution, and validation.

Data Engineering and ETL/ELT Development

- Build and optimize ETL/ELT pipelines using PySpark, Python, SQL, and Delta Lake.

- Develop reliable data transformation and processing workflows.

- Use Databricks Workflows for pipeline orchestration.

- Optimize data processing performance, scalability, and reliability.





Power BI and Analytics Enablement

- Develop curated Silver and Gold layer datasets for business reporting.

- Optimize data structures to support efficient Power BI consumption.

- Collaborate with reporting teams to understand analytical requirements.

- Improve data availability, consistency, and reporting performance.

Data Governance and Quality

- Implement data quality checks, monitoring, and validation processes.

- Apply data governance, metadata management, and data lineage practices.

- Work with Unity Catalog for data governance and access management.

- Establish appropriate security and monitoring controls.

- Maintain technical documentation and data architecture standards.

Technical Leadership and Collaboration

- Provide architectural guidance to engineering and reporting teams.

- Collaborate with business stakeholders to translate requirements into technical solutions.

- Lead cloud data modernization activities from design through production.

- Mentor engineers and support technical design reviews.

- Communicate project progress, architectural decisions, and technical risks.

Required Qualifications
- 10+ years of experience in data engineering, data architecture, or related enterprise data platform roles.

- Strong hands-on experience with Azure Databricks.

- Expertise in Delta Lake, Unity Catalog, and Databricks Workflows.

- Robust development experience with PySpark, Python, and SQL.





- Proven experience designing ETL/ELT pipelines.

- Strong understanding of Lakehouse and Medallion Architecture.

- Experience with enterprise data modeling and data warehousing.

- Knowledge of data governance, data lineage, and metadata management.

- Experience with cloud data platform modernization and migration.

- Strong stakeholder management, communication, and technical leadership skills.

Preferred Qualifications
- Experience supporting Power BI reporting and analytics.

- Experience migrating Azure Blob Storage-based workloads into Databricks.

- Experience optimizing reporting datasets and analytical performance.

- Experience delivering enterprise data transformation initiatives from architecture through production.

Expected Deliverables
- Current-state data architecture assessment.

- Target-state Databricks Lakehouse solution design.

- Data migration strategy and implementation plan.

- Scalable ingestion and transformation pipelines.

- Curated Silver and Gold layer reporting datasets.

- Data quality and monitoring framework.

- Performance optimization, governance, and technical documentation.

Core Technical Skills

Azure Databricks | PySpark | Python | SQL | Delta Lake | Unity Catalog | Databricks Workflows | ETL | ELT | Azure Blob Storage | Lakehouse Architecture | Medallion Architecture | Bronze Silver Gold | Data Modeling | Data Warehousing | Data Governance | Data Lineage | Metadata Management | Power BI | Cloud Data Migration

Application Details

Interested candidates may share their updated resumes at:

Email: [email protected]

Subject: Senior Data Engineer / Data Architect Databricks India

Please highlight your experience with Azure Databricks, PySpark, Lakehouse architecture, cloud data migration, and your availability to join.

📌 Data Engineer / Data Architect (Hyderabad)
🏢 Innova Software Services
📍 Hyderabad