JD

Position Summary

We are seeking an Azure Databricks Senior Developer to design, build, and optimize cloud-native data engineering solutions on Azure. This role will focus on developing scalable batch and streaming pipelines,

implementing robust data integration and transformation patterns, and contributing to data governance and quality frameworks. The candidate will also lead and mentor engineers while supporting QA/UAT and production implementation activities.

Required Skills

Technical (Must Have)

- Azure Databricks
- Azure Data Lake Storage (ADLS)
- Apache Spark (Core concepts and performance tuning)
- Scala
- SQL
- Data Streaming (e.g., Spark Structured Streaming and streaming design patterns)

Domain (Preferred)

- Healthcare domain experience (claims, clinical, eligibility, provider, member, HIPAA-aware handling preferred)

Nice to Have

- Python / PySpark
- Azure (general platform knowledge)

Classification

Internal

- MongoDB

Required Experience

- 5+ years of hands-on experience in Data Engineering / Data Warehousing development and operations.
- Proven experience designing and building enterprise-grade ETL/ELT pipelines on contemporary data platforms.
- Strong experience with:

o Spark/Databricks development (Scala preferred)

o SQL-based transformations and performance tuning o Streaming concepts and implementation patterns

- Experience working in Agile delivery models and collaborating with architects, analysts,



and upstream/downstream application teams.

Roles & Responsibilities

- Design and develop Azure cloud-native enterprise data solutions with emphasis on data integration,

transformation, and governance.

- Build solution designs and technical designs for data pipelines based on business and technical requirements.
- Develop and standardize reusable plug-and-play components that can be orchestrated into:

o data ingestion patterns (batch/streaming)

o data flows across multiple zones (raw/curated/consumption as applicable)

o data quality frameworks and validation rules

- Implement and maintain pipelines leveraging Databricks + ADLS, ensuring reliability, scalability, and maintainability.
- Identify opportunities to enhance/streamline existing codebase for:

o automation o performance improvements o scalability and reliability o operational efficiency and reduced run-cost

- Support backlog execution by helping prioritize pipeline development, estimate effort, and create implementation roadmaps.
- Lead a team of data engineers by providing technical direction, code reviews, and design guidance.
- Provide QA, UAT, and implementation support, including deployment readiness, operational handoffs,

and production troubleshooting.

- Ensure adherence to engineering standards (documentation, logging/monitoring expectations, and

CI/CD practices where applicable).

📌 Data Engineer- Databricks and SQL (Shruti) (Hyderabad)
🏢 HCL INDIA
📍 Hyderabad