Senior / Lead Data Engineer – Service Layer / QTC Integration Platform

Role Details

- Role: Senior / Lead Data Engineer
- Primary Skills: Go / Golang, Python, AWS, Kafka / MSK, Microservices, Distributed Systems, Data Engineering
- Platform Exposure: AWS Serverless, Kafka, Snowflake, PostgreSQL, ClickHouse, Event-Driven Architecture
- Domain Exposure: Quote-to-Cash, CRM, ERP, Customer Lifecycle, Enterprise Data preferred

- Experience: 8+ years

- Location/ Mode: Bengaluru/Hybrid

Role Overview

We are looking for a strong, hands-on, and highly autonomous Senior / Lead Data Engineer to design, build, and operate the Service Layer platform powering enterprise Quote-to-Cash (QTC), customer lifecycle, finance, and enterprise application integrations. This role requires deep expertise in distributed systems, event-driven architecture, microservices, data engineering, cloud-native development, and platform reliability. The candidate will also provide technical leadership to a team of engineers, driving solution design, engineering standards, delivery execution, and operational support across multiple integration initiatives.

The platform is built using AWS, Kafka / AWS MSK, Go, Python, canonical data models, asynchronous messaging, and cloud-native architecture patterns, processing high-volume, business-critical events with strong focus on reliability, security, scalability, observability, and operational readiness.

Key Responsibilities

Technical Delivery & Engineering Leadership

- Own end-to-end technical delivery of key initiatives, ensuring solutions are delivered with quality, scalability, security, and operational readiness.
- Lead and mentor a team of engineers through task planning, technical guidance, code reviews, design reviews, and delivery management.
- Drive engineering best practices across software development, CI/CD, observability, resiliency, security, and cloud-native architecture.
- Manage multiple concurrent initiatives while balancing business priorities, technical debt, and platform evolution.

Architecture & Platform Design

- Drive architecture and design decisions for event-driven integrations, canonical data models, orchestration frameworks, and platform capabilities.
- Define and enforce engineering standards, secure-by-design principles, cloud best practices, data governance, and non-functional requirements.
- Design and build scalable Service Layer foundation capabilities including messaging, orchestration, observability, reconciliation, data migration, operational tooling, and canonical data management.
- Ensure platform design supports resilience, scalability, fault tolerance, performance, and supportability.

Event-Driven Integration & Microservices

- Design, develop, and operate real-time event-driven integration services using Go / Golang and Python.
- Build and enhance the Service Layer platform to orchestrate Quote-to-Cash workflows across enterprise applications.




- Design and implement scalable Kafka / AWS MSK-based event processing with support for:
- Retries

- Idempotency

- Sequencing

- Message replay

- Fault recovery
- Dead-letter queues

- Eventual consistency

- Build secure and scalable RESTful APIs, integration services, and enterprise system interfaces.

Data Engineering, Migration & Reconciliation

- Design and implement large-scale ETL, migration, reconciliation, and data-quality solutions using:
- AWS Glue

- Step Functions

- Snowflake

- PostgreSQL

- ClickHouse

- Build and enforce canonical data models, data contracts, transformation layers, and validation frameworks.
- Support enterprise integrations by standardizing communication across multiple business systems.
- Ensure data accuracy, traceability, consistency, and operational reliability across the platform.

AWS Cloud-Native Engineering

- Design and build cloud-native applications on AWS using:
- AWS Lambda

- API Gateway
- S3

- RDS / Aurora PostgreSQL

- DynamoDB

- SQS

- SNS

- EventBridge

- IAM

- CloudWatch

- VPC

- Build high-availability, business-critical applications with strong focus on performance, security, and operational supportability.

Observability, Monitoring & Operations

- Design and implement observability, monitoring, alerting, and operational dashboards for end-to-end workflow visibility.
- Work with tools such as:
- CloudWatch

- Grafana

- Datadog

- Prometheus

- OpenTelemetry

- Support production operations including incident management, root-cause analysis, troubleshooting, and performance optimization.
- Drive metrics-driven improvements for reliability, scalability, and platform performance.

Collaboration & Stakeholder Management

- Partner with Product Managers, Architects, Enterprise Data teams, Engineering, DevOps, and business stakeholders.
- Define technical roadmaps, execution plans, dependencies, and delivery milestones.
- Communicate technical risks, trade-offs, design decisions, and mitigation plans clearly to stakeholders.
- Collaborate across multiple teams to ensure integration initiatives are delivered efficiently.

Must-Have Skills

Leadership & Delivery

- Proven experience leading and mentoring engineering teams.
- Strong experience in task planning, technical guidance, code reviews, and delivery management.
- Ability to drive architecture decisions for large-scale distributed systems and enterprise integration platforms.
- Experience managing multiple parallel technical initiatives.

Programming & Backend Engineering

- Strong hands-on experience in:
- Go / Golang

- Python





- Experience building scalable backend services, event-driven integrations, orchestration workflows, and data processing solutions.
- Solid understanding of REST API development and secure API design.

AWS Cloud & Serverless

- Strong experience designing and building cloud-native applications on AWS.
- Hands-on experience with:
- Lambda

- API Gateway
- S3

- RDS / Aurora PostgreSQL

- DynamoDB

- SQS / SNS

- EventBridge

- IAM

- CloudWatch

- VPC

- Strong understanding of AWS serverless architecture patterns.

Kafka / Event-Driven Systems

- Robust experience designing and operating event-driven and asynchronous systems using:
- Kafka

- AWS MSK
- RabbitMQ or equivalent messaging technologies

- Strong understanding of:

- Fault tolerance

- Idempotency

- Retry patterns
- Dead-letter queues

- Message replay

- Eventual consistency

- Sequencing

- Event orchestration

Databases & Data Engineering

- Robust experience with:
- PostgreSQL

- ClickHouse

- Relational databases

- DynamoDB / NoSQL databases

- Strong knowledge of:

- Data modelling

- Query optimization

- Performance tuning

- Data validation

- Reconciliation frameworks

- Experience with AWS Glue, Step Functions, Snowflake, and large-scale data workflows.

CI/CD, Security & Observability

- Experience with CI/CD pipelines, automated testing, code quality, and secure delivery using tools such as:
- GitLab CI

- Jenkins

- SonarQube

- Trivy

- Semgrep

- Strong understanding of cloud security and secure-by-design principles including:

- IAM

- OAuth2

- JWT authentication
- Encryption at rest and in transit

- Secrets Manager

- Vulnerability management

- Audit logging

- Compliance controls

Required Qualifications

- Bachelor’s degree in Computer Science, Engineering, Information Technology, or related discipline.
- 8+ years of software engineering experience.
- At least 4+ years focused on cloud-native, data engineering, integration, or distributed platforms.
- 4+ years building cloud-native solutions on AWS.
- 4+ years working with Go / Golang, Python, AWS services, and Kafka in production environments.
- Proven experience designing, building, and operating highly available enterprise integration and event-driven platforms.
- Strong experience in production support, incident management, root-cause analysis, and performance optimization.

Nice to Have

- AWS certifications such as:
- AWS Certified Data Engineer – Associate
- AWS Certified Solutions Architect – Associate
- AWS Certified Developer – Associate

- Equivalent AWS certifications

- Experience in:

- Quote-to-Cash

- CRM

- ERP

- Customer Lifecycle Management

- Enterprise Data domains

- Exposure to large-scale finance, billing, or enterprise integration transformation programmes.
- Experience building operational dashboards and platform-level reporting for engineering and business teams.

📌 Senior / Lead Data Engineer – Service Layer / QTC Integration Platform (Karnataka)
🏢 TALPRO INDIA PRIVATE
📍 Karnataka