Role & responsibilities :

Databricks (Spark SQL, Delta Lake, Unity Catalog)

SQL & LargeScale Data Processing Performance

Data Pipeline Design & Lakehouse Architecture

Data Modeling (Fact/Dimension, Analytics Readiness)

Git, CI/CD, Governance & Security

dbt (Models, Incremental Logic Working Knowledge)

- Design, develop, and maintain largescale batch data pipelines using Databricks and Delta Lake.
- Build and optimize Spark SQLbased transformations for highvolume, highperformance data processing.
- Implement Delta Lake best practices, including partitioning, OPTIMIZE, ZORDER, VACUUM, and timetravel.
- Design and manage curated fact and dimension tables aligned with Lakehouse architecture principles.
- Apply data modeling techniques to support analytical, BI, and reporting use cases.
- Ensure data reliability, consistency, and correctness across endtoend pipelines.
- Implement data governance, access control, and security policies using Unity Catalog.




- Build and manage CI/CD pipelines for Databricks notebooks, jobs, and SQL artifacts using Gitbased workflows.
- Monitor, troubleshoot, and optimize production pipelines for performance, cost, and scalability.
- Collaborate with analytics teams and support dbt integrations where required (valuable to have).

Preferred candidate profile

- Own endtoend data engineering pipelines from raw ingestion to curated datasets.
- Ensure productiongrade reliability, monitoring, and operational support of Databricks workloads.
- Participate in architecture design and platform decisionmaking.
- Optimize Spark and SQL workloads to meet SLA and cost targets.
- Mentor junior engineers and promote data engineering best practices.
- Work closely with analytics engineers and business stakeholders to deliver trusted data products.

📌 Azure Data Engineer (Kolkata)
🏢 Tata Consultancy Services
📍 Kolkata