Senior Data Engineer: Data Pipelines, Orchestration, System Design & AI Tooling

Location: Kolkata

Experience: 8-10 years

Employment Type: Full-time

About the Role

We are looking for a Senior Data Engineer to design, build, and operate reliable data pipelines and orchestration frameworks, and to enable AI-driven products and workflows on top of them. You will own the data flow from ingestion to consumption, set engineering standards for the team, and work closely with analytics, data science, and ML teams.

Key Responsibilities

Data pipeline design and development

- Design, build, and maintain scalable batch and streaming pipelines for structured and unstructured data.
- Develop robust ingestion, transformation, and loading (ETL/ELT) processes from diverse sources (databases, APIs, event streams, files).
- AWS services (EC2, ECS, Lambda, API Gateway, SQS etc)
- Optimize pipeline performance, cost, and reliability

Data orchestration

- Build and manage workflows using orchestration tools
- Implement scheduling, dependency management, retries, alerting, and backfill strategies.
- Establish monitoring, logging, and observability for pipeline health.

AI tools and enablement

- Build data foundations for machine learning and generative AI use cases, including feature pipelines and data preparation for model training and inference.
- Integrate AI/LLM tools and APIs into data workflows (e.g., document processing, enrichment, classification, summarization).
- Support retrieval-based AI solutions, including embedding pipelines and vector databases




- Collaborate with data scientists and ML engineers to move models from prototype to production.

Data quality, governance, and leadership

- Implement data quality checks, validation, lineage, and documentation.
- Apply security, privacy, and compliance best practices to data handling.
- Lead design reviews and code reviews, and mentor junior and mid-level engineers.
- Translate business requirements into technical solutions and communicate with stakeholders.

Required Qualifications

- 8-10 years of professional experience in data engineering or a closely related field.
- Robust proficiency in Python and advanced SQL.
- Proven experience building and maintaining production data pipelines at scale.
- Hands-on experience with at least one orchestration tool
- Experience with big data and processing frameworks
- Experience with cloud platforms and their data services.
- Solid understanding of data warehousing and lakehouse concepts, data modeling, and tools.
- Working knowledge of AI/ML workflows and tools, including LLM APIs, embeddings, and vector databases.
- Experience with version control, CI/CD, and infrastructure as code (e.g., Git, Terraform, Docker).
- Strong problem-solving, communication, and stakeholder management skills.

Preferred Qualifications

- Experience building RAG or agent-based solutions in production.
- Knowledge of data governance, cataloging, and lineage tools.
- Cloud or data platform certifications.
- Experience in healthcare industry.
- Bachelor's or Master's degree in Computer Science, Engineering, or a related field.

📌 Senior Data Engineer: Data Pipelines, Orchestration, System Design & A (Kolkata)
🏢 Applied Research Works India
📍 Kolkata