Roles and Responsibilities:
Design, develop, and optimize data pipelines using
Apache Spark on
Databricks
.
Implement ETL/ELT workflows to ingest, transform, and load data from various sources.
Collaborate with data scientists, analysts, and business stakeholders to deliver high-quality data solutions.
Ensure data quality, integrity, and governance across all pipelines.
Monitor and troubleshoot performance issues in Databricks notebooks and jobs.
Integrate Databricks with cloud services (Azure Data Lake) and enterprise data platforms.
Participate in code reviews, documentation, and agile ceremonies.
Qualification and Experience:
5+ years of experience in data engineering or big data development.
Solid proficiency in
Apache Spark
,
PySpark
, and
SQL
.
Hands-on experience with
Databricks platform (including notebooks, jobs, clusters).
Experience with cloud platforms
(Azure) and cloud-native data services.
Familiarity with
Delta Lake
,
MLflow
, and
Databricks Workflows
.
Knowledge of
CI/CD practices and version control (Git).
Solid problem-solving and communication skills.