GCP Data Architect (Bengaluru)

GCP Data Architect (Bengaluru)

14 Nov
|
Persistent Systems
|
Bengaluru

14 Nov

Persistent Systems

Bengaluru

Experience Data Modernization migration of an on premises SQL data warehouse to a target state Data Lake on Google Cloud (GCP) enabling metrics & reporting advanced analytics and GenAI use cases (natural language querying accelerated summarization cross domain trend analysis) leveraging PySpark based processing cloud native DevOps CI/CD pipelines and containerized deployments on OpenShift (OCP) to deliver scalable secure and high performance data solutions. 14 years of overall IT experience with strong experience in Data Architecture Big Data Engineering Cloud Data Platforms ETL/ELT Frameworks Data Lake Architecture and Enterprise Data Modernization programs. Location & Work Mode Location: India Bengaluru Work Mode: 3 days a week Work From Client Office About Program/Project The IAM Data Modernization project involves migrating an on-premises SQL data warehouse to a target state Data Lake in GCP cloud workplace. Key highlights include: Integration Scope: 30 source system data ingestions and multiple downstream integrations Capabilities: Metrics reporting and Gen AI use cases with natural language querying advanced pattern/trend analysis faster summarizations and cross-domain metric monitoring Benefits: Scalability and access to advanced cloud functionality Highly available and performant semantic layer with historical data support Unified data strategy for executive reporting analytics and Gen AI across cyber domains This modernization establishes a single source of truth for enterprise-wide data-driven decision-making. The IAM Data Modernization program is focused on migrating and modernizing enterprise IAM data platforms from on-premises systems into a scalable metadata-driven modern data architecture.



The current phase involves building a generic metadata-driven ETL framework to ingest data from multiple source systems into OCP / S3 followed by future integration with GCP / BigQuery for advanced analytics reporting and GenAI use cases. Key Responsibilities Own the end-to-end data architecture for the IAM Data Modernization program across OCP S3 PostgreSQL metadata repository and GCP / BigQuery future state architecture. Define and guide implementation of a metadata-driven generic ETL framework supporting multiple source types including databases files APIs MongoDB Splunk VDS and Active Directory. Architect reusable metadata models using MD static metadata tables and OPS operational/runtime tables for source configuration pipeline definition run tracking audit logging checkpointing schema drift and notifications. Provide architecture direction for MVC-based orchestration including repository layer service layer wrapper controller execution plan models and standardized request/response contracts. Design scalable ingestion patterns for full load incremental load watermark-based processing CDC-style patterns batch ingestion API ingestion and file-based ingestion. Guide design of S3 data lake zones including Landing Raw Archive and Error zones with clear promotion archival replay and error-handling patterns.



Define data quality schema validation schema drift detection CLOB validation checkpoint restartability auditability and operational resilience standards. Provide technical leadership for GCP / BigQuery architecture including S3 to BigQuery integration schema mapping dataset design partitioning clustering performance optimization and secure connectivity. Review and guide CI/CD automation using GitHub Actions JFrog Harness Liquibase Helm OCP deployments and environment-specific configuration management. Provide technical guidance to team on Python PySpark SQL BigQuery PostgreSQL data modeling performance tuning and production-ready framework design. Support UAT and production readiness by defining validation scenarios reconciliation strategy performance testing approach security validation runbooks and sign-off criteria. Partner with client stakeholders to finalize architecture decisions design documents Confluence documentation implementation standards and technical roadmaps. Ensure the solution is secure scalable maintainable metadata-driven auditable and extensible for future Splunk and Grafana observability integrations. Required Skills Big Data & Processing Hands on experience with PySpark for ETL/ELT data transformation and performance optimization Solid understanding of distributed data processing concepts Data & Cloud Architecture Strong experience designing data platforms on Google Cloud Platform (GCP) Experience with Data Lakes data warehousing and large scale migration programs Data Lake Architecture & Storage Proven experience designing and implementing data lake architectures (e.g. Bronze/Silver/Gold or layered models). Strong knowledge of Cloud Storage (GCS) design including bucket layout naming conventions lifecycle polic

Required Experience:

Staff IC

📌 GCP Data Architect (Bengaluru)
🏢 Persistent Systems
📍 Bengaluru

Reply to this offer

Impress this employer describing Your skills and abilities, fill out the form below and leave Your personal touch in the presentation letter.

Subscribe to this job alert:

Get the latest job offers by email for: gcp data architect (bengaluru) / bengaluru