Responsibilities
Design, develop, and optimize scalable batch and real-time data pipelines.
Build and maintain robust ETL/ELT processes for ingesting, transforming, and loading data from multiple sources.
Develop and manage data lakes, data warehouses, and metadata frameworks.
Collaborate with cross-functional teams to understand data requirements and deliver data solutions.
Ensure data quality, governance, lineage, and security standards are maintained.
Monitor, troubleshoot, and optimize data workflows and pipeline performance.
Technical and Skilled Requirements:
Databricks Certification
Azure Data Engineer Associate Certification
AWS Data Analytics Certification
Experience with Delta Lake, Iceberg, or Lakehouse Architecture
Knowledge of GenAI/LLM data pipelines