Key Responsibilities

Data Pipeline Development
Design, develop, and optimize scalable batch and streaming data pipelines using Databricks.
Build robust ETL/ELT frameworks using PySpark and Spark SQL.
Develop ingestion pipelines from:
APIs
Kafka
Cloud Storage
Databases
Event-driven sources

Lakehouse Architecture
Design and maintain Medallion Architecture:
Bronze Layer
Silver Layer
Gold Layer
Implement Delta Lake features including:
ACID Transactions
Schema Evolution
MERGE
UPSERT
Time Travel

Data Engineering & Modeling
Build dimensional and Lakehouse data models.
Support analytical and reporting workloads.
Ensure data quality, reconciliation, lineage, and governance.

Performance Optimization
Optimize Spark workloads using:
Partitioning
Caching
Cluster Tuning
Query Optimization
Improve scalability and reduce processing costs.

Governance & Security
Implement governance practices using:
Unity Catalog
Metadata Management
Access Controls




Compliance Standards

Orchestration & Automation
Integrate pipelines using:
Airflow
Cloud Composer
Azure Data Factory
Support CI/CD and deployment automation.

Must-Have Skills

Databricks
Solid hands-on experience with:
Databricks
Databricks Workflows
Cluster Management
Performance Optimization

Big Data
Expertise in:
Apache Spark
PySpark
Spark SQL
Delta Lake

Programming
Strong proficiency in:
Python
SQL

Data Engineering
Experience with:
ETL / ELT Development
Data Modeling
Batch Processing
Streaming Data Processing

Cloud Platform
Solid experience with GCP services such as:
BigQuery
Dataflow
Cloud Composer

Positive-to-Have Skills
Kafka
Event Hub
Airflow
Terraform
Docker
Kubernetes
GitHub Actions
Azure DevOps
Power BI
Tableau

📌 Senior Data Engineer Databricks, Pyspark & Lakehouse Pune
🏢 Tata Consultancy Services
📍 Pune