Location: India only | Remote

Type: Full time | Permanent

Role overview

Own the complete ML lifecycle for an enterprise LLM system that selects and executes browser-based tools. You will turn raw execution logs into high-quality training data, run fine-tuning experiments, build rigorous evaluations, and deploy and optimise models on AWS. This is an applied ML engineering role - not prompt engineering, analytics or model-ops-only work.

Core responsibilities

ML data and evaluation

- Build pipelines to ingest, parse, clean, normalise, enrich and deduplicate execution logs and documents.
- Convert raw logs into structured supervised examples with context, candidate tools, expected actions and reference labels.
- Create versioned training, validation and holdout datasets with privacy, schema, label and quality checks.
- Design leakage-aware splits using task- or session-level boundaries.
- Build reproducible evaluations for tool-selection accuracy, reference accuracy, hallucinations, disagreements and failure slices.
- Trace misleading metrics back to inputs, labels, dataset versions and model versions.

Model development

- Benchmark open-weight models such as Qwen, Llama and Mistral on quality, training effort, long-context behaviour and inference cost.
- Run supervised fine-tuning, knowledge distillation and parameter-efficient fine-tuning experiments.
- Apply sound experiment design: clear baselines, controlled comparisons, reproducible configurations and error analysis.
- Identify overfitting, leakage, sampling bias, label noise and uneven task coverage.
- Maintain versioned datasets, prompts, training configurations, checkpoints and evaluation code.
- Recommend which experiments justify additional training or inference cost.





ML platform and MLOps

- Take models through deployment, testing, monitoring and controlled release.
- Benchmark serving configurations for throughput, latency, GPU memory, concurrency and cost at an agreed quality level.
- Evaluate vLLM or comparable runtimes, INT8 quantisation, checkpoint compatibility and KV-cache requirements.
- Measure cost per successful task or action, not just cost per token.
- Implement model versioning, CI/CD, shadow inference, disagreement logging, feature-flag releases and rollback.
- Monitor data drift, quality regressions, serving errors, latency and cost.

LLM workflow engineering

- Build pipelines involving document parsing, chunking, embeddings, vector indexing and retrieval.
- Implement function or tool calling, multi-step workflows, retries, fallbacks and state handling.
- Design observability across prompts, retrieved context, model outputs, tool calls and downstream failures.
- Write production-grade, tested Python for large-scale document and execution-data processing.

Essential qualifications

- 7+ years of Python experience.
- 5+ years of AWS experience.
- 7+ years across ML, data, backend or applied AI engineering, with production delivery experience.
- Strong supervised ML fundamentals, including validation design, overfitting, leakage, sampling bias and error analysis.




- Hands-on transformer fine-tuning using PyTorch, Hugging Face or equivalent tooling.
- Direct ownership of a completed model training, fine-tuning, distillation or benchmarking project.
- Experience preparing ML/LLM datasets and building model evaluation code.
- Experience with experiment tracking, model versioning, CI/CD, deployment, monitoring and rollback.
- Working knowledge of GPU inference, quantisation, batching, concurrency, tail latency and cost optimisation.
- Experience with large-scale data processing using Spark, Kafka, Flink or comparable technologies.
- Strong SQL and practical experience with relational and NoSQL systems.
- Production experience with RAG, embeddings, vector databases, function calling or agentic workflows.
- Ability to work autonomously and own outcomes without detailed specifications.

Preferred background

- Startup or similarly high-ownership environment.
- AWS S3, Glue, IAM, EC2 GPU instances, CloudWatch and model registries.
- vLLM, NVIDIA GPU optimisation or AWS G5-family instances.
- Qwen, Llama or Mistral model families.
- LoRA, QLoRA, knowledge distillation or long-context evaluation.
- Browser automation, test execution logs or tool-selection workflows.
- Enterprise or regulated environments where privacy, auditability and governance matter.

What to include with your application

- A model you trained, fine-tuned, distilled or benchmarked, including the dataset, validation approach, metrics and your contribution.
- An ML data, evaluation, deployment or inference system you owned, including scale, failure handling and measurable results.
- Your current Indian city, earliest start date and interview availability.

📌 Senior AI/ML Data Scientist or Engineer - LLM Fine-Tuning, Evaluation & Inference (India)
🏢 Vamstar
📍 India