Job Description

Job purpose

Seeking a highly skilled Senior Data Engineer to design, build, and maintain scalable data platforms, ETL processes, and AI-enabled data solutions. The ideal candidate will have strong expertise in modern data engineering practices, cloud-native technologies, DevOps automation, and generative AI integrations. This role requires hands-on experience developing robust data pipelines, managing containerized applications, and supporting enterprise-grade data infrastructure.

Key responsibilities

- Design, develop, and maintain scalable ETL workflows and data pipelines that support business-critical applications and analytics.

- Build and manage data orchestration workflows using Apache Airflow.

- Develop high-quality, production-ready solutions using Python.

- Implement and maintain CI/CD pipelines leveraging GitHub Actions.

- Manage source control, branching strategies, and code reviews through GitHub repositories.

- Deploy, monitor, and optimize applications running on Azure Kubernetes Service (AKS).

- Develop and maintain containerized solutions using Docker.

- Integrate and support Generative AI solutions utilizing OpenAI and Anthropic Claude models.

- Collaborate with data scientists, ML engineers, and business stakeholders to deliver reliable data products.

- Ensure data quality, monitoring, observability, security, and governance standards are met.

- Troubleshoot production issues and drive continuous improvement across data engineering platforms.

- Document architecture, processes, and best practices for enterprise data solutions.

Key competencies

Essential skills

- Bachelor’s or master’s degree in computer science, Engineering, Information Technology, or a related field.





- 7–10 years of experience in data engineering, software engineering, or related technical disciplines.

- Strong hands-on experience with:

o Python

o GitHub and Git-based development workflows

o GitHub Actions CI/CD

o Azure Kubernetes Service (AKS)

o Apache Airflow

o Docker

o Building and maintaining enterprise-scale ETL processes and data pipelines

- Experience designing and operating cloud-native, distributed data platforms.

- Robust understanding of software engineering best practices, testing frameworks, and deployment automation.

- Experience working with OpenAI and Claude (Anthropic) models in production environments.

- Experience working with REST APIs and external data integrations.

- Excellent problem-solving, debugging, and performance optimization skills.

- Strong communication and stakeholder management capabilities.

Nice to Have

- Experience with S&P; Xpressfeed data platforms.

- Experience with Snowflake data warehousing and analytics solutions.

- Familiarity with data governance, data quality frameworks, and monitoring tools.

- Exposure to machine learning and AI/ML operations (MLOps) environments.

- Experience working in Agile/Scrum development teams

Skills:

- Data Engineering: ETL, Data Pipelines, Data Integration, Data Orchestration, Data Quality

- Cloud & DevOps: Azure Kubernetes Service (AKS), Docker, GitHub, GitHub Actions, CI/CD

- Programming: Python

- Workflow Orchestration: Apache Airflow

- AI & Generative AI: OpenAI, Claude, LLM Integrations

- Preferred Data Platforms: Snowflake, S&P; Xpressfeed

Job Snapshot

Job ID

JOB_7914

Department

Data and Technology Services

Location

ITPP-Kharadi, Pune, Maharashtra, India

Experience

7 - 10 Years

Employee Type

Permanent

📌 Senior Data Engineer - Delivery Lead (India)
🏢 Acuity Knowledge Partners
📍 India