Key Responsibilities

- Design, develop, and optimize data solutions using Microsoft Fabric and PySpark.
- Analyze and tune Spark workloads for performance, scalability, and cost efficiency.
- Develop and maintain Lakehouse architectures for enterprise analytics platforms.
- Optimize data ingestion, transformation, and processing pipelines.
- Perform Spark code reviews and recommend performance improvements.
- Implement partitioning, caching, indexing, and file optimization techniques.
- Monitor and troubleshoot Fabric workloads, Spark jobs, and resource utilization.
- Collaborate with data architects, data scientists, and business stakeholders.
- Design robust ETL/ELT frameworks using Fabric Data Factory and Notebooks.
- Ensure data quality, governance, security, and compliance standards.
- Implement best practices for distributed computing and big data processing.
- Create technical documentation, optimization reports, and performance metrics dashboards.

Required Skills

Microsoft Fabric

- Microsoft Fabric
- Fabric Data Engineering
- Fabric Lakehouse
- Fabric Data Factory




- Fabric Notebooks
- Fabric Real-Time Analytics
- Fabric OneLake
- Fabric Warehouse

PySpark & Spark Optimization

- PySpark Development
- Apache Spark Architecture
- Spark SQL
- Spark DataFrames
- Spark Structured Streaming
- DAG Optimization
- Catalyst Optimizer
- Adaptive Query Execution (AQE)
- Broadcast Joins
- Partitioning & Bucketing
- Shuffle Optimization
- Caching & Persistence Techniques

Data Engineering

- ETL/ELT Development
- Data Modeling
- Data Lake Architecture
- Medallion Architecture
- Delta Lake
- Data Quality Frameworks

Programming

- Python
- PySpark
- SQL
- Scala (Preferred)

Databases

- SQL Server
- Azure SQL
- PostgreSQL
- Snowflake (Preferred)

Cloud & Analytics

- Azure Data Services
- Azure Data Lake Storage (ADLS)
- Power BI
- Azure Monitor
- Log Analytics

DevOps & Automation

- Git
- Azure DevOps
- CI/CD Pipelines
- Infrastructure as Code (Preferred)

📌 MS Fabric + Pyspark Tuning - S (Hyderabad)
🏢 Tata Consultancy Services
📍 Hyderabad