Job highlights

- Graduate degree with hands-on experience in Databricks (Spark/PySpark), Python API development, Azure AI Services, LLMs, Generative AI, RAG, vector databases, and AI governance

- Design, develop, and maintain scalable data pipelines, build and deploy AI/ML solutions and APIs, manage workflows with Apache Airflow, and collaborate with cross-functional teams

Responsibilities

- Design, develop, and maintain scalable data pipelines using Databricks (Spark / PySpark)

- Orchestrate and monitor workflows using Apache Airflow for batch, streaming, and incremental processing

- Build and optimize ELT pipelines in Snowflake, ensuring performance, reliability, and cost efficiency

- Develop production-quality Python code for data processing, transformations, integrations, and AI/ML workloads

- Design, build, and host APIs, AI services, and data products using up-to-date Python frameworks

- Deploy and manage APIs, ML models, and AI services using Azure-native services (Azure Functions, App Services, API Management, Azure AI Services)

- Design and implement AI/ML and Generative AI solutions, including model integration, prompt engineering, RAG (Retrieval-Augmented Generation), and intelligent workflow automation

- Build and manage vector databases, embeddings pipelines, semantic search capabilities, and knowledge retrieval frameworks

- Develop and operationalize ML pipelines, feature engineering workflows, model monitoring, and MLOps practices

- Leverage LLMs and Agentic AI frameworks to automate data onboarding, data quality validation, metadata management, and business processes





- Apply Azure cloud best practices for security, scalability, monitoring, resiliency, governance, and cost optimization

- Implement data quality checks, lineage tracking, observability, model evaluation, and AI governance controls

- Collaborate with Data Science, Analytics, Product, and Platform teams to build AI-powered products and intelligent data solutions

- Comply with all applicable Company policies, procedures, and business directives, changes including those relating to work location, team assignments, work schedules, and flexible work arrangements

Required Qualifications

- Graduate degree or equivalent experience

- Solid hands-on experience with Databricks (Spark / PySpark) in production environments

- Experience building and hosting REST APIs, AI services, and microservices using Python frameworks

- Experience with Azure AI Services, Azure OpenAI, Azure Machine Learning, or equivalent AI platforms

- Experience working with Large Language Models (LLMs), Generative AI, RAG architectures, vector databases, embeddings, and prompt engineering

- Experience implementing AI governance, model evaluation, privacy, security, and responsible AI practices

- Experience with CI/CD pipelines, Git-based version control, Infrastructure as Code, and deployment automation

- Solid working experience with Snowflake, including data modeling, optimization, and performance tuning

- Solid experience with Microsoft Azure, including cloud-native architecture and best practices

- Knowledge of Agentic AI frameworks (e.g., Semantic Kernel, LangChain, AutoGen, CrewAI, or similar) and intelligent workflow orchestration

📌 Associate Data Engineering - Databricks (Bengaluru)
🏢 Optum
📍 Bengaluru