- Design, develop, and maintain scalable data pipelines using PySpark on Databricks
- Build ETL/ELT workflows to process structured and unstructured data
- Optimize Spark jobs for performance, cost, and scalability
- Work with Delta Lake, implement data versioning, and optimize storage
- Integrate data from multiple sources such as APIs, databases, and streaming platforms
- Develop and manage workflows using Databricks Workflows / Apache Airflow
- Ensure data quality through validation, monitoring, and logging
- Collaborate with data scientists, analysts, and stakeholders
- Implement best practices for data governance, security, and compliance
- Handle batch and real-time data processing (streaming).