Job Title: Senior Observability Engineer (Grafana | Prometheus | Loki | Terraform | Open Telemetry)

Experience: 7-12 Years
Employment Type: Full-Time Contract
Work Mode: 100% Remote (Work from Anywhere)

Job Description:

We are seeking an experienced Senior Observability Engineer to lead and scale our observability platform for a globally distributed engineering organization. The ideal candidate should have strong hands-on expertise in Grafana, Prometheus, Loki, Terraform, and OpenTelemetry, along with 2-3 years of programming experience in Java or Python.

You will be responsible for designing and managing enterprise-grade monitoring, logging, and distributed tracing solutions, enabling engineering teams to build highly reliable and observable systems.

Key Responsibilities

Observability Platform

- Design, implement, and manage enterprise observability solutions using Grafana, Prometheus, Loki, Tempo/Jaeger, and OpenTelemetry.
- Build and maintain monitoring dashboards, alerts, and visualization platforms.
- Define and manage SLIs, SLOs, and error budgets.
- Improve alerting strategies and reduce operational noise.
- Partner with engineering teams to drive observability adoption and best practices.

Infrastructure as Code (Terraform)

- Develop and maintain Terraform modules for observability infrastructure.
- Automate provisioning of monitoring, logging, and tracing resources.
- Manage Terraform state, workspace strategies, and CI/CD integrations.
- Collaborate with Platform and DevOps teams on IaC standards.

Instrumentation & Distributed Tracing





- Implement OpenTelemetry instrumentation across applications and services.
- Enable distributed tracing for microservices architectures.
- Standardize logging, metrics, and tracing practices.
- Build workflows correlating metrics, logs, and traces for faster troubleshooting.

Programming & Automation

- Develop automation tools, scripts, exporters, and integrations using Java or Python.
- Create custom monitoring solutions and Prometheus exporters.
- Automate dashboard provisioning, alert management, and reporting.

SRE & Incident Management

- Support incident response, RCA, and post-mortem activities.
- Develop operational runbooks and monitoring playbooks.
- Drive reliability improvements through observability insights.

Mandatory Skills

- Grafana
- Prometheus
- Loki
- Terraform
- OpenTelemetry
- Distributed Tracing
- SLI / SLO / Error Budgets
- Java or Python Programming
- Monitoring & Alerting
- CI/CD Integration

Desired Skills

- VictoriaMetrics / Thanos / Cortex
- Grafana Tempo / Jaeger / Zipkin
- Kubernetes
- AWS, Azure, or GCP Monitoring Services
- PagerDuty / OpsGenie
- eBPF Observability Tools (Pixie, Cilium, Parca)

Candidate Profile

- 7+ years in Platform Engineering, SRE, DevOps, or Observability Engineering.
- Expert-level experience with Grafana, Prometheus, and Loki in production environments.
- Solid Terraform and Infrastructure-as-Code expertise.
- 2-3+ years of Java or Python development experience.
- Experience implementing OpenTelemetry-based observability.
- Strong communication and stakeholder management skills.
- Comfortable working in a fully remote environment.

📌 Senior Observability Engineer (Bengaluru)
🏢 Allegis Global Solutions (AGS)
📍 Bengaluru