- Design Python-based Databricks pipelines for ingestion, transformation, enrichment, validation and publication of DER/VPP data.
- Build scalable batch and near-real-time workloads using Spark, Delta Lake, notebooks, jobs and workflows.
- Implement data-quality, reconciliation, lineage, auditability and observability controls.
- Optimise partitioning, throughput, query performance, reliability, maintainability and cloud cost.
- Support feature engineering and productionisation of telemetry, dispatch and platform analytics.
- Deliver Python packages, tests, validation checks, workflow definitions and dashboard-ready datasets.
Must have skills:
Python; Databricks; PySpark/Apache Spark; Delta Lake; SQL; ETL/ELT; batch and streaming pipelines; data quality; lineage; performance tuning; Azure; Git/CI/CD; PostgreSQL