Design, develop, and optimize scalable data pipelines using GCP services.
• Build and maintain batch and real-time data ingestion and processing frameworks.
• Develop enterprise-grade data warehousing solutions using BigQuery. • Implement ETL/ELT processes for structured and unstructured data.
• Integrate data from multiple sources including databases, APIs, files, and streaming platforms.
• Ensure data quality, governance, security, and compliance across data platforms. Optimize query performance, storage, and processing costs within GCP.
• Automate deployment and operational processes using CI/CD and Infrastructure as Code (IaC).
• Monitor data pipelines and troubleshoot production issues.
• Collaborate with Data Architects, Business Analysts, and Data Scientists to translate business requirements into technical solutions.
• Implement data models, metadata management, and data lineage best practices.
• Support migration of on-premise or multi-cloud data platforms to GCP.
• Mentor junior engineers and participate in technical design reviews.