Role Summary

The Python Developer (AI/ML) turns trained models into reliable production services. You will build the asynchronous inference pipelines and GPU workers that process tenant media at scale, serve models with predictable latency, and provide the MLOps tooling that makes releases repeatable.

Key Responsibilities

Inference Pipelines

- Pipeline stages: Build pipelines for decode, frame sampling, pre-processing, GPU inference, post-processing (NMS, tracking, geo-projection) and result persistence, with versioned, testable stages.

- Video decoding: Use hardware-accelerated decode (NVDEC via FFmpeg/DeepStream) and zero-copy GPU memory paths where possible to avoid CPU bottlenecks.

- Live and batch: Support live-stream inference with bounded latency and frame dropping policy, and batch reprocessing of historical media.

GPU Workers and Orchestration

- Job workers: Implement async, queue-based workers (Celery, Ray, Kafka consumers or SQS) with idempotency, retries with backoff, dead-letter queues, priority classes and per-tenant fairness.

- GPU efficiency: Manage GPU memory, batching and concurrency; implement graceful preemption and checkpointing for long jobs; expose queue-depth metrics that drive autoscaling.

- Resilience: Handle corrupt media, OOM and driver errors, and partial failures without losing or duplicating results.

Model Serving

- Serving: Deploy models behind NVIDIA Triton (dynamic batching, model ensembles, concurrent model instances) or ONNX Runtime/TensorRT services; implement health, warm-up and canary rollout of model versions.

- Interfaces: Expose job APIs with FastAPI (submit, status, cancel, results) with authentication,



tenant scoping and usage reporting for billing.

MLOps

- Registry and promotion: Operate MLflow (or equivalent) model registry with staging/production stages, reproducible builds and rollback.

- CI for models: Automate model packaging, container builds, regression tests against golden sets and performance benchmarks before promotion.

- Observability: Instrument latency (p50/p95/p99), throughput, GPU utilisation (DCGM), error rate and cost per job; share dashboards with SRE.

Required Skills and Experience

- 4–6 years of Python development with solid asyncio and concurrency understanding.

- Queue-based distributed workers (Celery, Ray, Kafka, RabbitMQ or SQS).

- Docker; production deployment experience.

- GPU inference: CUDA basics, TensorRT or ONNX Runtime; understanding of batching and memory trade-offs.

- MLflow or similar for tracking and registry.

- Testing discipline: pytest, integration tests with real media samples.

- Working knowledge of FFmpeg and OpenCV.

Preferred / Added Advantage

- Triton Inference Server, KServe, Ray Serve.

- DeepStream or GStreamer pipelines.

- Kubernetes GPU scheduling (device plugin, MIG, time-slicing).

- Performance profiling (Nsight, py-spy).

- FastAPI and Pydantic; gRPC.

Key Deliverables

- Inference service and worker framework with documentation.

- Model serving configuration and rollout procedure.

- MLOps pipeline for packaging, testing and promoting models.

- Throughput, latency and cost benchmarks.

Education

Bachelor’s degree in Computer Science, Engineering or equivalent experience.

Pay: ₹1,000,000.00 - ₹1,200,000.00 per year

Work Location: In person

📌 Sr. Python Developer (Hyderabad)
🏢 Spatial Alphabet
📍 Hyderabad