17 Nov
|
Asrah Technologies
|
Pune
17 Nov
Asrah Technologies
Pune
Role & responsibilities:
JOB SUMMARY:We are looking for a Senior Data Engineer to design, build, and operate our enterprise data platform on Azure and Databricks, with strong hands-on skills in GenAI/LLM application engineering. This role owns the full lifecycle of data pipelines and AI-powered applications from ingestion and orchestration to governance, API development, and deployment. The ideal candidate is equally comfortable writing production-grade PySpark pipelines, building LLM-powered agents and RAG systems, and managing the DevOps/IaC backbone that deploys them all.Key Responsibilities: GenAI & LLM Application Engineering:
- Design and build RAG (Retrieval-Augmented Generation) pipelines, including embedding-based and graph/metadata-grounded retrieval approaches.
- Orchestrate multi-step and multi-agent LLM workflows using LangGraph and LangChain.
- Build and integrate tools/agents using MCP (Model Context Protocol) for structured tool-calling and context management.
- Develop backend APIs for AI applications using FastAPI and Python Flask.
- Build and manage LLM/agent workflows using Azure AI Foundry, and implement semantic + vector search using Azure AI Search indexes.
- Implement prompt engineering, structured outputs, guardrails, and LLM evaluation/observability practices.
Data Engineering & Pipeline Development:
- Design and build scalable ETL/ELT pipelines using Azure Data Factory (ADF) and Databricks (PySpark, Spark SQL)
- Develop and maintain Databricks Lakeflow (Delta Live Tables) pipelines across bronze/silver/gold medallion layers, including data quality expectations and materialized views.
- Use Databricks Autoloader for incremental, schema-evolving file ingestion into Delta Lake.
- Build reusable,
config-driven ingestion frameworks supporting multiple heterogeneous source systems (databases, APIs, SFTP, cloud storage, SaaS applications).
- Implement CDC, SCD Type 1/2, and incremental loading strategies.
- Package and deploy Databricks workflows using Databricks Asset Bundles (DABs) for consistent, version-controlled deployments across environments.
Platform, Infrastructure & DevOps:
- Own infrastructure provisioning and deployment using Terraform (Infrastructure as Code) for Databricks, ADF, and related Azure resources.
- Build and maintain Azure DevOps CI/CD pipelines for automated testing, deployment, and release management across dev/QA/prod environments.
- Deploy and manage applications on Azure Container Apps, using Docker for containerization of services and AI/data applications.
- Develop and support Databricks Apps for internal data tooling, dashboards, or self-service platforms.
- Ensure zero-downtime deployments, environment parity, and repeatable infrastructure across environments.
Governance, Security & Reliability:
- Implement and maintain data governance using Databricks Unity Catalog (access control, column masking, row-level filtering, lineage).
- Design and enforce security policies (ABAC-style) across managed tables in collaboration with data governance stakeholders.
- Build monitoring, alerting, and SLA dashboards to ensure pipeline reliability and data quality.
- Perform Spark/Databricks performance tuning cluster sizing, partitioning, caching, and cost optimization.
Collaboration:
- Partner with data architects, analysts, and business stakeholders to translate requirements into scalable data and AI solutions.
- Document architecture, pipelines, and operational runbooks; present designs to technical and non-technical stakeholders.
- Mentor junior engineers on Databricks, Python development, and platform engineering best practices.
Preferred candidate profile:
- Robust hands-on experience with:
- Azure Data Factory (ADF) pipeline design, linked services, generic/config-driven ingestion frameworks
- Databricks PySpark, Spark SQL, Delta Lake, Autoloader, Lakeflow/Delta Live Tables, Asset Bundles (DABs)
- Databricks Unity Catalog governance, column masking, row-level security, lineage
- Databricks Apps building/deploying internal data applications
- Azure Container Apps and Docker containerized deployment of services and pipelines
- Terraform infrastructure as code for cloud and platform resources
- CI/CD with Azure DevOps — pipeline design, automated deployments, release management
- Strong Python development skills, including API development with FastAPI and Flask.
- Hands-on experience with LLM application engineering — RAG pipelines, LangGraph, LangChain, and MCP (Model Context Protocol) for tool/agent orchestration.
- Experience with Azure AI Foundry for building/managing LLM and agent workflows, and Azure AI Search for semantic and vector indexing.
- Experience with API integration — designing, consuming, and securing REST APIs across internal and third-party systems.
- Experience with vector databases, hybrid search, and re-ranking for RAG systems.
📌 Senior Data & AI Engineer (Pune)
🏢 Asrah Technologies
📍 Pune