Role: Databricks Engineer

Exp:8 to 16 Years,

Location: Kolkata,

Type: Full Time.

NP: Immediate to 7 days.

Must have experience: DataBricks, AWS, Python/ Py-spark, SQL, AI Coding(GitHub Copilot), Healthcare domain.

Build scalable data pipelines on a contemporary Databricks Lakehouse platform on AWS, integrating multi-source data and enabling analytics.

Key Responsibilities

Develop and maintain ingestion and transformation pipelines using Databricks (Jobs/DLT) and PySpark.

Implement Delta Lake tables with ACID, schema evolution, and efficient MERGE/CDC patterns.

Ingest data from Amazon S3 and APIs; standardize data into Bronze/Silver/Gold layers.

Design event-driven triggers with EventBridge to orchestrate pipelines.

Implement data quality checks, reconciliation, and performance tuning.

Collaborate with Architects, Data Modelers, and Business Analysts.

Leverage AI copilots (GitHub Copilot, ChatGPT) to accelerate development and optimization.

Must-Have Skills

Hands-on experience with Databricks Lakehouse,



(Delta Lake, Delta Live Tables (DLT), and Databricks Workflows), PySpark/Spark, Python, and SQL, AWS services/ S3 and AWS Glue, CI/CD practices.

Experience in PySpark/Spark transformations and performance optimization.

Strong SQL and Python skills for ETL/ELT development.

Experience building modern data pipelines on Databricks.

Experience with AI-Coding Copilots tools (GitHub Copilot and ChatGPT) for code generation, debugging, optimization, and documentation.

Good-to-Have Skills

Healthcare domain experience, including: Quality Measures, Risk Adjustment (HCC/RAF)

HEDIS

Exposure to FHIR/HL7 data models and patient/member/provider entities.

Experience with Event-Driven patterns using AWS EventBridge for Ingestion/Orchestration.

Delta Live Tables expectations and data quality frameworks.

Lakehouse integration patterns.

📌 Databricks Engineer (West Bengal)
🏢 Shell Infotech
📍 West Bengal