Role & responsibilities :

Databricks (Spark SQL, Delta Lake, Unity Catalog)

SQL & LargeScale Data Processing Performance

Data Pipeline Design & Lakehouse Architecture

Data Modeling (Fact/Dimension, Analytics Readiness)

Git, CI/CD, Governance & Security

dbt (Models, Incremental Logic Working Knowledge)
Design, develop, and maintain largescale batch data pipelines using Databricks and Delta Lake.
Build and optimize Spark SQLbased transformations for highvolume, highperformance data processing.
Implement Delta Lake best practices, including partitioning, OPTIMIZE, ZORDER, VACUUM, and timetravel.
Design and manage curated fact and dimension tables aligned with Lakehouse architecture principles.
Apply data modeling techniques to support analytical, BI, and reporting use cases.
Ensure data reliability, consistency, and correctness across endtoend pipelines.
Implement data governance, access control, and security policies using Unity Catalog.




Build and manage CI/CD pipelines for Databricks notebooks, jobs, and SQL artifacts using Gitbased workflows.
Monitor, troubleshoot, and optimize production pipelines for performance, cost, and scalability.
Collaborate with analytics teams and support dbt integrations where required (valuable to have).

Preferred candidate profile
Own endtoend data engineering pipelines from raw ingestion to curated datasets.
Ensure productiongrade reliability, monitoring, and operational support of Databricks workloads.
Participate in architecture design and platform decisionmaking.
Optimize Spark and SQL workloads to meet SLA and cost targets.
Mentor junior engineers and promote data engineering best practices.
Work closely with analytics engineers and business stakeholders to deliver trusted data products.

📌 Azure Data Engineer Kolkata (India)
🏢 Tata Consultancy Services
📍 India