- Design, develop, and maintain scalable ETL/ELT pipelines using Databricks.
- Build and optimize data workflows using Apache Spark (PySpark/Scala).
- Develop and manage data lakes and lakehouse architectures using Delta Lake.
- Ingest, transform, and process structured and unstructured data from multiple sources.
- Implement data quality, governance, and security best practices.
- Optimize Spark jobs and Databricks workloads for performance and cost efficiency.
- Collaborate with data architects, analysts, and business stakeholders to deliver data solutions.
- Integrate Databricks with cloud-native services across Azure, AWS, or GCP.
- Monitor, troubleshoot, and resolve data pipeline and platform issues.
- Participate in architecture reviews and provide technical recommendations.
Preferred candidate profile
- 4+ years of Data Engineering experience.
- Hands-on experience with Databricks Platform.
- Robust expertise in PySpark/Spark SQL.
- Proficiency in Python and SQL.
- Experience with Delta Lake, Databricks Workflows, and Unity Catalog.
- Experience in ETL/ELT pipeline development.
- Knowledge of data modeling concepts.
- Experience with Git, CI/CD, and Agile methodologies.
- Understanding of data warehousing concepts.