Overview:

We are looking for a Senior Data Engineer with deep expertise in Lakehouse architecture, real-time data streaming, cloud data infrastructure, and microservices development on Azure Kubernetes Service (AKS). You will play a central role in designing and delivering next-generation data pipelines, BI solutions, AI/ML platforms, streaming APIs, and scalable microservices that power Guidepoints research and analytics products.

This is a high-impact, hands-on engineering role. You will work closely with data architects, data scientists, analysts, frontend engineers, QA, and DevOps teams to translate complex business requirements into scalable, reliable, and observable data systems.

This is a Hybrid role from our Pune office.

What Youll Do:

Data Engineering Lakehouse

- Design, build, and maintain ETL pipelines, data ingestion workflows, and table schemas on Azure Databricks to support BI, analytics, and AI/ML use cases

- Architect and optimize the Lakehouse using Delta Lake on Databricks, ensuring reliability, performance, and cost efficiency

- Build and support data pipelines from business applications such as Salesforce, NetSuite, and other enterprise systems

- Develop and maintain Knowledge Graph models, entity relationship structures, and NLP-based insight pipelines

- Maintain data governance, data privacy standards, and compliance best practices throughout the data lifecycle

- Perform root cause analysis on data and processes to identify opportunities for improvement

- Collaborate with data architects, scientists,



and business consumers to populate and optimize the data warehouse for reporting and analytics

Microservices AKS Development

- Develop and support scalable web APIs and microservices using Python and Azure Platform Services

- Build recent applications, services, and platforms; optimize existing solutions and refactor legacy components using modern, scalable architectures

- Design, implement, and deploy microservices on Azure Kubernetes Service AKS using Docker, Kubernetes, Helm, and Azure DevOps YAML pipelines

- Perform end-to-end deployments including infrastructure setup, configuration, and monitoring on AKS

- Decompose portions of legacy applications into modern microservices architecture

- Design and manage JSON payloads and payload contexts for inter-service communication

- Engage in database schema design and management, including updating tables and rows for large datasets

- Collaborate with cross-functional teams Full-Stack, QA, DevOps, and Product in agile SDLC processes

Real-Time Streaming SSE

- Design and implement robust SSE (Server-Sent Events) endpoints using Python frameworks (FastAPI, Flask, Django) for real-time event delivery to web and mobile clients

- Build and maintain asynchronous backend services using asyncio,



aiohttp, or similar libraries for non-blocking, high-concurrency streaming

- Architect streaming data pipelines integrating SSE with upstream message brokers Kafka, Redis Pub/Sub, RabbitMQ

- Optimize connection lifecycle management: reconnection logic, heartbeat signals, event ID tracking, and graceful shutdowns

- Collaborate with frontend teams to define and evolve SSE event schemas and API contracts

- Implement observability across streaming services: distributed tracing, structured logging, and metrics using Prometheus, Datadog, or OpenTelemetry

Engineering Excellence

- Write comprehensive unit, integration, and load tests for all data, streaming, and microservices components

- Write and maintain robust CI/CD pipelines using Azure DevOps YAML pipelines

- Participate in architecture reviews, code reviews, and on-call rotations

- Maintain thorough technical documentation and mentor junior engineers on best practices in data engineering, Lakehouse architecture, streaming systems, and microservices

What You Have:

Required

- Bachelors degree in Computer Science, Engineering, or a related field from an accredited university

- 7+ years of professional data engineering and/or backend software engineering experience

Disclaimer: This job posting has been aggregated from external source. Role details, content, and availability are subject to change. Applicants are advised to confirm the latest information directly on the company website before applying.

📌 Data/AI Engineer (Pune)
🏢 Guidepoint
📍 Pune