Key Responsibilities

- Design, develop, and optimize data solutions using Microsoft Fabric and PySpark.

- Analyze and tune Spark workloads for performance, scalability, and cost efficiency.

- Develop and maintain Lakehouse architectures for enterprise analytics platforms.

- Optimize data ingestion, transformation, and processing pipelines.

- Perform Spark code reviews and recommend performance improvements.

- Implement partitioning, caching, indexing, and file optimization techniques.

- Monitor and troubleshoot Fabric workloads, Spark jobs, and resource utilization.

- Collaborate with data architects, data scientists, and business stakeholders.

- Design robust ETL/ELT frameworks using Fabric Data Factory and Notebooks.

- Ensure data quality, governance, security, and compliance standards.

- Implement best practices for distributed computing and big data processing.

- Create technical documentation, optimization reports, and performance metrics dashboards.

Required Skills

Microsoft Fabric

- Microsoft Fabric

- Fabric Data Engineering

- Fabric Lakehouse

- Fabric Data Factory





- Fabric Notebooks

- Fabric Real-Time Analytics

- Fabric OneLake

- Fabric Warehouse

PySpark &

- Spark Optimization

- PySpark Development

- Apache Spark Architecture

- Spark SQL

- Spark DataFrames

- Spark Structured Streaming

- DAG Optimization

- Catalyst Optimizer

- Adaptive Query Execution (AQE)

- Broadcast Joins

- Partitioning &
- Bucketing

- Shuffle Optimization

- Caching &

- Persistence Techniques

Data Engineering

- ETL/ELT Development

- Data Modeling

- Data Lake Architecture

- Medallion Architecture

- Delta Lake

- Data Quality Frameworks

Programming

- Python

- PySpark

- SQL

- Scala (Preferred)

Databases

- SQL Server

- Azure SQL

- PostgreSQL

- Snowflake (Preferred)

Cloud &

- Analytics

- Azure Data Services

- Azure Data Lake Storage (ADLS)

- Power BI

- Azure Monitor

- Log Analytics

DevOps &

- Automation

- Git

- Azure DevOps

- CI/CD Pipelines

- Infrastructure as Code (Preferred)

📌 MS Fabric + Pyspark Tuning - S (Bengaluru)
🏢 Tata Consultancy Services
📍 Bengaluru