1. Data Pipeline Development & Orchestration -Design, develop, and orchestrate data pipelines using Apache Airflow (Cloud Composer) to automate data ingestion, transformation, and loading (ETL/ELT) workflows.
2. Data Transformation with DBT -Implement DBT (Data Build Tool) models for transforming raw data into analytics-ready datasets in BigQuery, applying SQL-based transformations, modular modeling, and version control.
3. Stream & Batch Processing with Dataflow - Build real-time (Streaming) and batch data processing pipelines using Apache Beam on Dataflow, ensuring scalable and effective data processing.
4. BigQuery Optimization & Performance Tuning - Design optimized BigQuery schemas, implement partitioning, clustering, and tune queries for cost-efficiency and performance.
5. Monitoring, Logging, and CI/CD for Data Pipelines - Set up monitoring with Stackdriver, logging, error handling, and CI/CD pipelines for automated testing and deployment of DBT, Airflow DAGs, and Dataflow jobs.