Key Responsibilities

Data Pipeline Development
- Design, develop, and optimize scalable batch and streaming data pipelines using Databricks.
- Build robust ETL/ELT frameworks using PySpark and Spark SQL.
- Develop ingestion pipelines from:
- APIs
- Kafka
- Cloud Storage
- Databases
- Event-driven sources

Lakehouse Architecture
- Design and maintain Medallion Architecture:
- Bronze Layer
- Silver Layer
- Gold Layer
- Implement Delta Lake features including:
- ACID Transactions
- Schema Evolution
- MERGE
- UPSERT
- Time Travel

Data Engineering & Modeling
- Build dimensional and Lakehouse data models.
- Support analytical and reporting workloads.
- Ensure data quality, reconciliation, lineage, and governance.

Performance Optimization
- Optimize Spark workloads using:
- Partitioning
- Caching
- Cluster Tuning
- Query Optimization
- Improve scalability and reduce processing costs.

Governance & Security
- Implement governance practices using:
- Unity Catalog
- Metadata Management
- Access Controls




- Compliance Standards

Orchestration & Automation
- Integrate pipelines using:
- Airflow
- Cloud Composer
- Azure Data Factory
- Support CI/CD and deployment automation.

Must-Have Skills

Databricks
- Strong hands-on experience with:
- Databricks
- Databricks Workflows
- Cluster Management
- Performance Optimization

Big Data
- Expertise in:
- Apache Spark
- PySpark
- Spark SQL
- Delta Lake

Programming
- Strong proficiency in:
- Python
- SQL

Data Engineering
- Experience with:
- ETL / ELT Development
- Data Modeling
- Batch Processing
- Streaming Data Processing

Cloud Platform
- Strong experience with GCP services such as:
- BigQuery
- Dataflow
- Cloud Composer

Valuable-to-Have Skills
- Kafka
- Event Hub
- Airflow
- Terraform
- Docker
- Kubernetes
- GitHub Actions
- Azure DevOps
- Power BI
- Tableau

📌 Senior Data Engineer (Bengaluru)
🏢 Tata Consultancy Services
📍 Bengaluru