Key Responsibilities:
Design, build, and maintain CI/CD pipelines using Azure DevOps for automated testing, container image builds, and deployment
Containerise the data quality platform using Docker and manage deployment of container images across Business environments using Kubernetes
Spin up and configure new container instances when onboarding new Business areas onto the platform
Develop and maintain automated testing frameworks covering unit, integration, and end-to-end testing
Design, build, and expose RESTful APIs to integrate the platform with upstream and downstream systems
Collaborate with the Data Architecture team to ensure platform components are modular, platform-agnostic, and deployable across cloud and on-premise environments
Work with Azure services including Azure Container Registry, Azure Kubernetes Service, Azure Key Vault, Azure Blob Storage, and Azure DevOps
Develop and optimise data pipelines using PySpark and SQL Server
Support the integration of Databricks workloads into the platform architecture
Ensure platform security standards are met including secret management, image vulnerability scanning, and access controls
Contribute to technical documentation, runbooks, and onboarding guides for Business teams

Required Skills & Experience:
8–12 years of experience in data engineering or platform engineering roles
Strong hands-on experience with Azure services — AKS, ACR, Key Vault, Blob Storage, Azure DevOps




Proven experience containerising applications using Docker and orchestrating with Kubernetes
Solid experience building and maintaining CI/CD pipelines in Azure DevOps
Experience designing and building RESTful APIs
Proficiency in PySpark and SQL Server
Experience with Databricks — job orchestration, cluster management, notebook development
Strong understanding of software testing practices — unit testing, integration testing, end-to-end testing
Experience with branching strategies, versioning, and release management
Strong problem-solving skills and ability to work independently in a rapid-paced environment

Preferred Skills:
Understanding of Data Quality concepts — profiling, validation rules, DQ scoring, observability
Experience with data quality tooling such as Informatica, Great Expectations, or similar
Familiarity with Flask or similar Python web frameworks
Experience with modular, microservice-style architecture design

What We're Looking For:
Beyond technical skills, we are looking for someone who takes ownership, thinks like a platform engineer, and understands that the best infrastructure is the kind that Business teams never have to think about. The ideal candidate is equally comfortable writing a Dockerfile, debugging a failing pipeline, and discussing deployment governance with a non-technical stakeholder.

📌 Senior Data Engineer (Bengaluru)
🏢 TEKsystems
📍 Bengaluru