Job Summary
We are looking for an experienced Senior Data Engineer with strong expertise in Azure Databricks, PySpark, Azure Data Factory (ADF), and T-SQL to design and deliver enterprise-scale data engineering solutions. The ideal candidate will have extensive experience modernizing legacy SQL-based data transformation logic into scalable Databricks implementations, building high-performance ETL/ELT pipelines, and optimizing large-scale data processing workloads. This role requires strong technical leadership, the ability to mentor engineering teams, and close collaboration with architects, business stakeholders, and global delivery teams to implement enterprise data transformation programs.

Key Responsibilities

- Design, develop, and maintain enterprise-scale data engineering solutions using Azure Databricks and PySpark.
- Analyze, modernize, and migrate complex T-SQL transformation logic into optimized Databricks implementations.
- Design and develop scalable data ingestion pipelines using Azure Data Factory (ADF) to ingest data from multiple enterprise source systems into RAW and curated data layers.
- Build, maintain, and optimize enterprise ETL/ELT pipelines ensuring high availability, scalability, and performance.
- Monitor pipeline execution, analyze performance metrics, troubleshoot bottlenecks, and implement optimization strategies.
- Optimize Databricks workloads using partitioning, caching, Delta Lake optimization, file format tuning, and Spark performance best practices.
- Collaborate with Solution Architects, Business Analysts, Product Owners, and cross-functional engineering teams.




- Participate in technical solution design, architecture discussions, and engineering best practices.
- Lead technical initiatives and provide guidance to junior and mid-level Data Engineers.
- Conduct code reviews and ensure adherence to coding standards and data engineering best practices.
- Prepare technical documentation and support production deployments.
- Work closely with global teams on enterprise-scale cloud data transformation initiatives.

Required Skills

- Minimum 6+ years of hands-on experience with Azure Databricks.
- Strong expertise in Core SQL, Complex SQL, and T-SQL (Mandatory).
- Strong hands-on experience with PySpark.
- Experience designing enterprise-scale data ingestion pipelines using Azure Data Factory (ADF).
- Strong understanding of ETL/ELT architecture and enterprise data engineering principles.
- Experience migrating complex SQL transformation logic into Databricks using PySpark.
- Expertise in Databricks performance optimization and Spark tuning.
- Experience working with enterprise cloud data platforms.
- Strong analytical, debugging, troubleshooting, and problem-solving skills.
- Excellent communication and stakeholder management skills.
- Ability to mentor engineering teams and provide technical leadership.

Positive to Have

- .NET knowledge
- Delta Lake
- Lakehouse Architecture
- Azure DevOps
- CI/CD Pipelines
- Agile/Scrum methodology

Educational Qualification

- Bachelor's Degree in Computer Science, Information Technology, Engineering, or a related discipline.

Pay: ₹500,000.00 - ₹1,000,000.00 per year

Benefits:

- Provident Fund

Work Location: In person

📌 Senior Data Engineer (India)
🏢 AVISOFT
📍 India