- Design, develop, and maintain scalable data pipelines using Azure Databricks and PySpark.
- Build and optimize large-scale data processing solutions using Python and Spark.
- Develop robust ETL/ELT pipelines for batch and near-real-time data processing.
- Work with Azure Data Lake Storage (ADLS) and other Azure data services.
- Implement data transformation, cleansing, validation, and enrichment processes.
- Optimize Spark jobs, Databricks notebooks, clusters, and SQL queries for performance and cost efficiency.
- Develop reusable and maintainable data engineering frameworks and components.
- Implement data quality, governance, security, and monitoring practices.
- Work with structured and unstructured datasets across multiple data sources.
- Collaborate with solution architects and engineering teams to translate business requirements into technical solutions.
- Troubleshoot production issues and perform root-cause analysis.
- Implement CI/CD processes for Databricks code and data pipelines.
- Follow engineering best practices around version control, testing, documentation, and deployment.
- Mentor junior and mid-level data engineers and contribute to technical decision-making.
Required Skills &
Experience :
- 8 years of experience in Data Engineering, Big Data, or related technologies.
- Strong hands-on experience with Azure Databricks.
- Excellent knowledge of PySpark and Apache Spark.
- Strong programming experience in Python.
- Solid experience developing ETL/ELT data pipelines.
- Hands-on experience with Azure Data Lake Storage (ADLS Gen2).
- Good knowledge of Azure data services and cloud architecture.
- Strong SQL skills and experience working with relational and analytical databases.
- Experience with Delta Lake and Delta Tables.
- Solid understanding of data modeling, data warehousing, and distributed data processing.
- Experience with performance tuning and optimization of Spark/Databricks workloads.