Key Responsibilities

- Design and develop robust ETL/ELT pipelines using Databricks platform and PySpark
- Implement and maintain data workflows using Databricks Asset Bundles (DABs)
- Build and optimize data processing solutions using Delta Lake architecture
- Develop complex SQL queries and stored procedures for data transformation and analysis
- Ensure data quality through comprehensive testing and validation frameworks
- Collaborate with data scientists, analysts, and other stakeholders to understand data requirements
- Monitor and optimize pipeline performance and data processing efficiency
- Maintain code quality through version control best practices and code reviews
- Document technical processes and maintain data pipeline documentation

Required Skills & Qualifications

Must-Have Technical Skills:

1. Databricks Platform - DABS

- Proficiency in Databricks Asset Bundles (DABs) for CI/CD workflows
- Experience with Databricks workspace management and configuration

- PySpark Development





- Solid programming skills in PySpark for large-scale data processing
- Experience with DataFrame API, RDD operations, and Spark SQL

- SQL

- Advanced SQL skills including complex joins, window functions, and performance optimization
- Experience with data modeling and database design principles

- Delta Lake Fundamentals

- Understanding of Delta Lake architecture and ACID transactions
- Experience with time travel, data versioning, and schema evolution

- ETL Development

- Proven experience in designing and implementing ETL/ELT pipelines
- Knowledge of data integration patterns and best practices

- Data Quality Assurance

- Experience implementing data validation and quality checks
- Knowledge of data profiling and anomaly detection techniques

- Version Control

- Proficiency with Git for code management and collaboration
- Experience with branching strategies and code review processes

📌 Data Engineer - Databricks (Hyderabad)
🏢 PwC
📍 Hyderabad