Key Responsibilities

Data Pipeline Development

- Design, develop, and optimize scalable batch and streaming data pipelines using Databricks.
- Build robust ETL/ELT frameworks using PySpark and Spark SQL.
- Develop ingestion pipelines from:
- APIs
- Kafka
- Cloud Storage
- Databases
- Event-driven sources

Lakehouse Architecture

- Design and maintain Medallion Architecture:
- Bronze Layer
- Silver Layer
- Gold Layer

- Implement Delta Lake features including:

- ACID Transactions
- Schema Evolution
- MERGE
- UPSERT
- Time Travel

Data Engineering & Modeling

- Build dimensional and Lakehouse data models.
- Support analytical and reporting workloads.
- Ensure data quality, reconciliation, lineage, and governance.

Performance Optimization

- Optimize Spark workloads using:
- Partitioning
- Caching
- Cluster Tuning
- Query Optimization

- Improve scalability and reduce processing costs.

Governance & Security

- Implement governance practices using:
- Unity Catalog
- Metadata Management
- Access Controls




- Compliance Standards

Orchestration & Automation

- Integrate pipelines using:
- Airflow
- Cloud Composer
- Azure Data Factory

- Support CI/CD and deployment automation.

Must-Have Skills

Databricks

- Strong hands-on experience with:
- Databricks
- Databricks Workflows
- Cluster Management
- Performance Optimization

Big Data

- Expertise in:
- Apache Spark
- PySpark
- Spark SQL
- Delta Lake

Programming

- Strong proficiency in:
- Python
- SQL

Data Engineering

- Experience with:
- ETL / ELT Development
- Data Modeling
- Batch Processing
- Streaming Data Processing

Cloud Platform

- Solid experience with GCP services such as:
- BigQuery
- Dataflow
- Cloud Composer

Good-to-Have Skills

- Kafka
- Event Hub
- Airflow
- Terraform
- Docker
- Kubernetes
- GitHub Actions
- Azure DevOps
- Power BI
- Tableau

📌 Senior Data Engineer - Databricks, PySpark & Lakehouse (Pune)
🏢 Tata Consultancy Services
📍 Pune