Key Responsibilities

1. MLOps/LLMOps Pipeline Design

- Design and implement end-to-end CI/CD/CT pipelines for traditional ML models and LLM-based applications.
- Build reproducible workflows for data preparation, training, fine-tuning, validation, packaging, and deployment.
- Use MLflow for experiment tracking, model versioning, and lifecycle management.
- Implement Infrastructure-as-Code (Terraform, Databricks Asset Bundles) and Git-based deployment practices.
- Define promotion strategies (dev staging prod) with automated quality gates.

2. LLM Serving & Inference Optimization

- Deploy and manage LLMs (open-source and proprietary) using Databricks Model Serving / Mosaic AI, vLLM, TGI, or similar frameworks.
- Optimize inference latency, throughput, and cost through quantization, batching, KV-cache optimization, model distillation, and GPU utilization tuning.
- Configure autoscaling, provisioned throughput, rate limiting, and failover strategies.
- Manage AI Gateway configurations, API endpoints, and multi-model routing.

3. Automated LLM Evaluation Pipelines

- Build automated evaluation frameworks for LLM and RAG applications using MLflow LLM Evaluate, Mosaic AI Agent Evaluation, RAGAS, DeepEval, or similar tools.
- Implement LLM-as-a-judge, golden datasets, regression testing, and human-in-the-loop feedback loops.
- Track metrics such as groundedness, faithfulness, relevance, toxicity, hallucination rate, latency, and cost per query.
- Integrate evaluation as a mandatory gate in CI/CD before production releases.

4. Observability & Drift Detection

- Implement monitoring for model performance, data drift, concept drift,



and prediction quality using Databricks Lakehouse Monitoring and inference tables.
- Build LLM observability: tracing, token usage, latency, error rates, and prompt/response logging (MLflow Tracing, LangSmith, Langfuse, Arize, or OpenTelemetry).
- Set up dashboards and alerts (Grafana, Prometheus, Databricks SQL dashboards) with defined SLAs/SLOs.
- Drive root-cause analysis and automated retraining or rollback triggers.

5. Model & Prompt Governance (Unity Catalog)

- Govern models, prompts, features, vector indexes, and datasets using Unity Catalog.
- Implement prompt versioning, prompt registries, and model lineage tracking.
- Enforce access control, audit logging, PII handling, and responsible AI guardrails.
- Ensure compliance with internal security policies and regulatory requirements (e.g., GDPR, SOC 2, EU AI Act where applicable).

6. Databricks Workflows & Airflow

- Orchestrate ML and LLM pipelines using Databricks Workflows (Jobs) and Apache Airflow.
- Design robust DAGs with dependency management, retries, alerting, and SLA monitoring.
- Integrate orchestration with CI/CD tooling (GitHub Actions, Azure DevOps, GitLab CI, Jenkins).

7. RAG Pipeline Operations

- Operationalize Retrieval-Augmented Generation pipelines end to end: ingestion, chunking, embedding generation, indexing, retrieval, and generation.
- Manage vector databases (Databricks Vector Search, Pinecone, FAISS, Weaviate, pgvector, etc.) including index sync, refresh, and scaling.
- Monitor and improve retrieval quality (recall@k, MRR, context precision) and embedding freshness.
- Optimize chunking strategies, hybrid search, re-ranking, and caching for performance and cos

📌 LLMOps + MLOps Engineer (Hyderabad)
🏢 PwC
📍 Hyderabad