- Should know how to develop, deploy , monitor and optimize Etl workflows for both Batch and Streaming with Scala, Spark, Airflow, BQ , Kafka , Dataproc and GCP environments
- Experience in Medallion architecture with proper Data quality checks and summarization activities
- Experience in performance optimizations, Observability and runbook creations for continuous optimizations.
- Positive knowledge of Airflow orchestrations including dependency handing, idempotent execution etc.
- Exposure to data governance and information security would be a plus
- Through knowledge of handing streaming data including back pressure handling etc.
- Minimum Experience – 6+ years