19 Nov
|
HCLTech
|
Bengaluru
Job Summary
seeking an experienced Data Engineer to design, develop, and optimize scalable enterprise data pipelines and transformation solutions on Google Cloud Platform (GCP).
Key Responsibilities
Data Engineering & Development
- Design and develop scalable data ingestion, transformation, and processing pipelines on GCP.
- Build reusable engineering frameworks and patterns for batch, incremental, and Change Data Capture (CDC) workloads.
- Develop and maintain data transformations using BigQuery, Dataform, SQL, and Python.
- Convert business and technical requirements into efficient data engineering solutions.
- Support high-volume enterprise data migration and modernization initiatives.
GCP Data Platform
- Design and implement solutions using GCP services such as:
- - BigQuery
- Cloud Storage
- Pub/Sub
- Dataflow
- Dataproc
- Cloud Composer
- Dataform
- Optimize BigQuery datasets, queries, partitioning, clustering, and overall workload performance.
Data Integration & CDC
- Design and implement batch and near-real-time data integration patterns.
- Build CDC pipelines from enterprise source systems into GCP.
- Handle inserts, updates, deletes, schema changes, incremental loads, and error/reprocessing scenarios.
- Experience with CDC/integration technologies such as Qlik Replicate/Qlik CDC, Datastream, Informatica, or equivalent is advantageous.
Data Quality & Reconciliation
- Implement data validation and reconciliation between source and target systems.
- Develop automated controls covering row counts, duplicates, nulls, aggregates, business rules, and transformation accuracy.
- Troubleshoot data discrepancies and work with source-system, QA, and business teams to resolve issues.
- Support SIT, UAT, migration rehearsals, production cutover, and hypercare.
DevOps & Automation
- Implement CI/CD practices for data engineering workloads.
- Maintain source code and deployment artifacts using Git/GitHub.
- Integrate automated testing and deployment into pipelines using tools such as GitHub Actions, Cloud Build, Jenkins, or equivalent.
- Promote reusable coding standards, parameterized solutions, and automated deployments across environments.
Technical Leadership
- Provide technical guidance and mentoring to junior data engineers.
- Review solution designs, code, SQL, transformation logic, and deployment artifacts.
- Define reusable patterns and data engineering best practices.
- Collaborate with Data Architects, Platform Engineers, Data Modelers, DBAs, QA teams, and business stakeholders.
- Support technical troubleshooting and root-cause analysis of complex data issues.
Skill Requirements
Core skills
- Google Cloud Platform (GCP)
- BigQuery
- Dataform
- Advanced SQL
- Python
- Data integration / ETL / ELT
- CDC patterns
- Data modeling concepts
- Git / GitHub
- CI/CD
- Data reconciliation and testing
Must Have Skill
GCP BigQuery,GCP Data Analysis,Qlik Replicate
Good to Have Skill
Other Requirements
- 5- 9 years of overall data engineering/ETL experience with strong hands-on cloud data engineering expertise.
- Strong experience working with large datasets and complex SQL transformations.
- Experience delivering enterprise data migration, modernization, or analytics solutions.
- Strong analytical, troubleshooting, communication, and stakeholder-management skills.
Preferred Certifications
- Google Cloud Qualified Data Engineer
📌 Senior Technical Consultant (Bengaluru)
🏢 HCLTech
📍 Bengaluru