Design, build, and optimize Azure data pipelines and lakehouse solutions. Deliver secure, reliable datasets with strong governance, automation, and documentation. Collaborate across teams and contribute to standards in an Agile setting.
Must-Have (Day 1)
- Experience: 4–6 years in data engineering
- Core Platform: Databricks with Python, Spark, Pandas (notebooks and modular code)
- Orchestration: Azure Data Factory (pipelines, integration runtimes); ingest from diverse sources
- Lakehouse: Delta Lake fundamentals; Medallion architecture (bronze/silver/gold) in production
- Storage/SQL/Performance: Azure Data Lake Storage (ADLS); solid SQL; performance-aware design
- Data Patterns: ETL/ELT; data modeling (e.g., dimensional/star schema)
- DevOps & Security: CI/CD for data projects (Azure DevOps or GitHub Enterprise); familiarity with Azure Entra ID for SSO/RBAC; secure workspace/data access
- Quality & Observability: Data validation/testing, code reviews, and basic monitoring/alerting for jobs/pipelines
- Ways of Working: Agile/Scrum (Jira/Confluence); clear pipeline and data contract documentation
- Collaboration:
Effective stakeholder engagement; support/mentor junior team members; clear communication
- Generative AI (Day 1)
- Prompt design for data tasks (ingestion, transformations, documentation) with clear objectives and constraints
- Use of Copilot/ChatGPT to scaffold notebooks/jobs, generate tests, and optimize SQL/Spark—validates outputs before merging
Nice-to-Have (Train within 60–90 days)
- Unity Catalog migration (Hive to Unity) and permissions/governance