Senior Data & AI Engineer (Pune)

Senior Data & AI Engineer (Pune)

17 Nov
|
Asrah Technologies
|
Pune

17 Nov

Asrah Technologies

Pune

Role & responsibilities:

JOB SUMMARY:We are looking for a Senior Data Engineer to design, build, and operate our enterprise data platform on Azure and Databricks, with strong hands-on skills in GenAI/LLM application engineering. This role owns the full lifecycle of data pipelines and AI-powered applications from ingestion and orchestration to governance, API development, and deployment. The ideal candidate is equally comfortable writing production-grade PySpark pipelines, building LLM-powered agents and RAG systems, and managing the DevOps/IaC backbone that deploys them all.Key Responsibilities: GenAI & LLM Application Engineering:

- Design and build RAG (Retrieval-Augmented Generation) pipelines, including embedding-based and graph/metadata-grounded retrieval approaches.
- Orchestrate multi-step and multi-agent LLM workflows using LangGraph and LangChain.
- Build and integrate tools/agents using MCP (Model Context Protocol) for structured tool-calling and context management.
- Develop backend APIs for AI applications using FastAPI and Python Flask.
- Build and manage LLM/agent workflows using Azure AI Foundry, and implement semantic + vector search using Azure AI Search indexes.
- Implement prompt engineering, structured outputs, guardrails, and LLM evaluation/observability practices.

Data Engineering & Pipeline Development:

- Design and build scalable ETL/ELT pipelines using Azure Data Factory (ADF) and Databricks (PySpark, Spark SQL)
- Develop and maintain Databricks Lakeflow (Delta Live Tables) pipelines across bronze/silver/gold medallion layers, including data quality expectations and materialized views.
- Use Databricks Autoloader for incremental, schema-evolving file ingestion into Delta Lake.
- Build reusable,



config-driven ingestion frameworks supporting multiple heterogeneous source systems (databases, APIs, SFTP, cloud storage, SaaS applications).
- Implement CDC, SCD Type 1/2, and incremental loading strategies.
- Package and deploy Databricks workflows using Databricks Asset Bundles (DABs) for consistent, version-controlled deployments across environments.

Platform, Infrastructure & DevOps:

- Own infrastructure provisioning and deployment using Terraform (Infrastructure as Code) for Databricks, ADF, and related Azure resources.
- Build and maintain Azure DevOps CI/CD pipelines for automated testing, deployment, and release management across dev/QA/prod environments.
- Deploy and manage applications on Azure Container Apps, using Docker for containerization of services and AI/data applications.
- Develop and support Databricks Apps for internal data tooling, dashboards, or self-service platforms.
- Ensure zero-downtime deployments, environment parity, and repeatable infrastructure across environments.

Governance, Security & Reliability:

- Implement and maintain data governance using Databricks Unity Catalog (access control, column masking, row-level filtering, lineage).
- Design and enforce security policies (ABAC-style) across managed tables in collaboration with data governance stakeholders.
- Build monitoring, alerting, and SLA dashboards to ensure pipeline reliability and data quality.




- Perform Spark/Databricks performance tuning cluster sizing, partitioning, caching, and cost optimization.

Collaboration:

- Partner with data architects, analysts, and business stakeholders to translate requirements into scalable data and AI solutions.
- Document architecture, pipelines, and operational runbooks; present designs to technical and non-technical stakeholders.
- Mentor junior engineers on Databricks, Python development, and platform engineering best practices.

Preferred candidate profile:

- Robust hands-on experience with:
- Azure Data Factory (ADF) pipeline design, linked services, generic/config-driven ingestion frameworks
- Databricks PySpark, Spark SQL, Delta Lake, Autoloader, Lakeflow/Delta Live Tables, Asset Bundles (DABs)
- Databricks Unity Catalog governance, column masking, row-level security, lineage
- Databricks Apps building/deploying internal data applications
- Azure Container Apps and Docker containerized deployment of services and pipelines
- Terraform infrastructure as code for cloud and platform resources
- CI/CD with Azure DevOps — pipeline design, automated deployments, release management
- Strong Python development skills, including API development with FastAPI and Flask.
- Hands-on experience with LLM application engineering — RAG pipelines, LangGraph, LangChain, and MCP (Model Context Protocol) for tool/agent orchestration.
- Experience with Azure AI Foundry for building/managing LLM and agent workflows, and Azure AI Search for semantic and vector indexing.
- Experience with API integration — designing, consuming, and securing REST APIs across internal and third-party systems.
- Experience with vector databases, hybrid search, and re-ranking for RAG systems.

📌 Senior Data & AI Engineer (Pune)
🏢 Asrah Technologies
📍 Pune

Reply to this offer

Impress this employer describing Your skills and abilities, fill out the form below and leave Your personal touch in the presentation letter.

Subscribe to this job alert:

Get the latest job offers by email for: senior data & ai engineer (pune) / pune

Subscribe to this job alert:

Get the latest job offers by email for: senior data & ai engineer (pune) / pune