18 Nov
|
Asrah Technologies India
|
Pune
18 Nov
Asrah Technologies India
Pune
Job Summary:We are looking for a Senior Data Engineer to design, build, and operate our enterprise data platform on Azure and Databricks, with strong hands-on skills in GenAI/LLM application engineering. This role owns the full lifecycle of data pipelines and AI-powered applications from ingestion and orchestration to governance, API development, and deployment.Key Responsibilities:GenAI & LLM Application Engineering:- Design and build RAG pipelines, including embedding-based and graph/metadata-grounded retrieval approaches.- Orchestrate multi-step and multi-agent LLM workflows using LangGraph and LangChain.- Build and integrate tools/agents using MCP for structured tool-calling and context management.- Develop backend APIs for AI applications using FastAPI and Python Flask.- Build and manage LLM/agent workflows using Azure AI Foundry, and implement semantic + vector search using Azure AI Search indexes.- Implement prompt engineering, structured outputs, guardrails, and LLM evaluation/observability practices.Data Engineering & Pipeline Development:- Design and build scalable ETL/ELT pipelines using Azure Data Factory (ADF) and Databricks (PySpark, Spark SQL).- Develop and maintain Databricks Lakeflow (Delta Live Tables) pipelines across bronze/silver/gold medallion layers.- Use Databricks Autoloader for incremental, schema-evolving file ingestion into Delta Lake.- Build reusable, config-driven ingestion frameworks supporting multiple heterogeneous source systems.- Implement CDC, SCD Type 1/2, and incremental loading strategies.- Package and deploy Databricks workflows using Databricks Asset Bundles (DABs).Platform,
Infrastructure & DevOps:- Own infrastructure provisioning and deployment using Terraform for Databricks, ADF, and related Azure resources.- Build and maintain Azure DevOps CI/CD pipelines for automated testing, deployment, and release management.- Deploy and manage applications on Azure Container Apps, using Docker for containerization.- Develop and support Databricks Apps for internal data tooling.Governance, Security & Reliability:- Implement and maintain data governance using Databricks Unity Catalog.- Design and enforce security policies across managed tables.- Build monitoring, alerting, and SLA dashboards to ensure pipeline reliability and data quality.- Perform Spark/Databricks performance tuning, cluster sizing, partitioning, and cost optimization.Collaboration:- Partner with data architects, analysts, and business stakeholders to translate requirements into scalable data and AI solutions.- Document architecture, pipelines, and operational runbooks.- Mentor junior engineers on Databricks, Python development, and platform engineering best practices.Preferred Candidate Profile:- Solid hands-on experience with Azure Data Factory, Databricks (PySpark, Spark SQL, Delta Lake, Autoloader, Lakeflow, DABs), and Unity Catalog.- Proficiency in Databricks Apps, Azure Container Apps, Docker, and Terraform.- Strong Python development skills, including API development with FastAPI and Flask.- Hands-on experience with LLM application engineering (RAG, LangGraph, LangChain, MCP).- Experience with Azure AI Foundry, Azure AI Search, and vector databases.- Experience with CI/CD using Azure DevOps and API integration. (ref:hirist.tech)
📌 Asrah - Senior Data & AI Engineer (Pune)
🏢 Asrah Technologies India
📍 Pune