Role Summary
Design, build, and optimize Azure data pipelines and lakehouse solutions. Deliver secure, reliable datasets with strong governance, automation, and documentation. Collaborate across teams and contribute to standards in an Agile setting.
Must-Have (Day 1)
- Experience: 4–6 years in data engineering
- Core Platform: Databricks with Python, Spark, Pandas (notebooks and modular code)
- Orchestration: Azure Data Factory (pipelines, integration runtimes); ingest from diverse sources
- Lakehouse: Delta Lake fundamentals; Medallion architecture (bronze/silver/gold) in production
- Storage/SQL/Performance: Azure Data Lake Storage (ADLS); solid SQL; performance-aware design
- Data Patterns: ETL/ELT; data modeling (e.g., dimensional/star schema)
- DevOps & Security: CI/CD for data projects (Azure DevOps or GitHub Enterprise); familiarity with Azure Entra ID for SSO/RBAC; secure workspace/data access
- Quality & Observability: Data validation/testing, code reviews, and basic monitoring/alerting for jobs/pipelines
- Ways of Working: Agile/Scrum (Jira/Confluence); clear pipeline and data contract documentation
- Collaboration:
Effective stakeholder engagement; support/mentor junior team members; clear communication
- Generative AI (Day 1):
- Prompt design for data tasks (ingestion, transformations, documentation) with clear objectives and constraints
- Use of Copilot/ChatGPT to scaffold notebooks/jobs, generate tests, and optimize SQL/Spark—validates outputs before merging
Nice-to-Have (Train within 60–90 days)
- Unity Catalog migration (Hive to Unity) and permissions/governance