JDData Engineer- Databricks and SQL

Position Summary We are seeking an Azure Databricks Senior Developer to design, build, and optimize cloud-native data engineering solutions on Azure. This role will focus on developing scalable batch and streaming pipelines, implementing robust data integration and transformation patterns, and contributing to data governance and quality frameworks. The candidate will also lead and mentor engineers while supporting QA/UAT and production implementation activities. Required Skills Technical (Must Have) Azure Databricks Azure Data Lake Storage (ADLS) Apache Spark (Core concepts and performance tuning) Scala SQL Data Streaming (e.g., Spark Structured Streaming and streaming design patterns) Domain (Preferred) Healthcare domain experience (claims, clinical, eligibility, provider, member, HIPAA-aware handling preferred) Nice to Have Python / PySpark Azure (general platform knowledge) MongoDB Required Experience 5+ years of hands-on experience in Data Engineering / Data Warehousing development and operations. Proven experience designing and building enterprise-grade ETL/ELT pipelines on up-to-date data platforms. Strong experience with: o Spark/Databricks development (Scala preferred) o SQL-based transformations and performance tuning o Streaming concepts and implementation patterns Experience working in Agile delivery models and collaborating with architects, analysts, and upstream/downstream application teams. Roles & Responsibilities Design and develop Azure cloud-native enterprise data solutions with emphasis on data integration, transformation, and governance.



Build solution designs and technical designs for data pipelines based on business and technical requirements. Develop and standardize reusable plug-and-play components that can be orchestrated into: o data ingestion patterns (batch/streaming) o data flows across multiple zones (raw/curated/consumption as applicable) o data quality frameworks and validation rules Implement and maintain pipelines leveraging Databricks + ADLS, ensuring reliability, scalability, and maintainability. Identify opportunities to enhance/streamline existing codebase for: o automation o performance improvements o scalability and reliability o operational efficiency and reduced run-cost Support backlog execution by helping prioritize pipeline development, estimate effort, and create implementation roadmaps. Lead a team of data engineers by providing technical direction, code reviews, and design guidance. Provide QA, UAT, and implementation support, including deployment readiness, operational handoffs, and production troubleshooting. Ensure adherence to engineering standards (documentation, logging/monitoring expectations, and CI/CD practices where applicable).

Background Verification

- Mandatory background verification checks include:
- 1. Education Check
- 1. Employment Check

- 1. Database Check

- 1. Criminal Check

- 1. Address Check

- 1. CIBIL Check

- 1. Reference Check

- 1. ID Check

- 1. CV Validation

- 1. Gap Check

- 1. Credit Check

- 1. Drug test

Pay: ₹508,642.91 - ₹1,869,405.03 per year

Work Location: In person

📌 Data Engineer- Databricks and SQL (Hyderabad)
🏢 Opulent Vidya Private
📍 Hyderabad