Observability Engineer (Grafana / Prometheus / VictoriaMetrics)

Job Overview

We are looking for an Observability Engineer to join our team and lead the development and maintenance of scalable monitoring systems using Grafana, Prometheus, and VictoriaMetrics. This role plays a crucial part in enhancing visibility, performance, and reliability across distributed infrastructure and applications.

Section 1: Grafana Responsibilities & Skills

- Develop and maintain intuitive dashboards, visualizations, and panels for real-time monitoring.
- Create and manage alert rules, notification policies, and contact points for effective alerting.
- Build dashboards using stat, table, pie chart, bar chart, time series, and heatmap visualizations.
- Perform advanced panel transformations including filtering, joining, merging, grouping, and calculations.
- Implement dashboard variables, agile filtering, and dashboard linking.
- Use JSON API and Infinity plugin (UQL) for nested API responses.
- Integrate data sources including Elasticsearch, CloudWatch, Druid, and custom JSON APIs.
- Manage dashboard and alert configurations as code using Bitbucket/GitHub repositories.
- Deploy observability changes using GoCD pipelines following GitOps practices.
- Validate dashboard and alert behavior post deployment.
- Manage Grafana role-based access controls and security group integrations.
- Configure alert silencing for maintenance windows and known downtimes.
- Integrate alerts with Email, Teams, Slack, PagerDuty, ServiceNow, and Webhooks.
- Perform troubleshooting and root cause analysis using dashboards, metrics, and Kibana logs.
- Work with PromQL and Lucene queries for monitoring and log analysis.

Section 2: Prometheus & VictoriaMetrics Responsibilities & Skills

- Design and manage observability solutions using Prometheus and VictoriaMetrics.
- Configure and maintain VictoriaMetrics single-node and clustered deployments.
- Automate onboarding of services and nodes using service discovery,



Ansible, and Terraform.
- Create and manage Prometheus scrape jobs, relabeling rules, recording rules, and federation.
- Configure Prometheus remote_write and VictoriaMetrics ingestion pipelines.
- Manage metric retention policies, scrape intervals, and query optimizations.
- Optimize storage and performance by handling high-cardinality metrics.
- Configure vmagent, vmalert, vmselect, and vmstorage components.
- Onboard and manage exporters including Node Exporter, Windows Exporter, SNMP Exporter, Blackbox Exporter, cAdvisor, kube-state-metrics, and JMX/WebLogic Exporters.
- Monitor Linux infrastructure using Node Exporter.
- Monitor Windows infrastructure using Windows Exporter.
- Configure SNMP Exporter for monitoring routers, switches, firewalls, SD-WAN devices, storage systems, and enterprise hardware.
- Configure JMX/WebLogic Exporters for JVM-based applications including heap, garbage collection, threads, datasource, and application metrics.
- Deploy and manage Kubernetes monitoring using kube-state-metrics, cAdvisor, and Prometheus Operator.
- Monitor virtualization and storage environments including VMware, NAS, SAN, and NetApp.
- Troubleshoot scrape failures, exporter issues, missing metrics, and ingestion bottlenecks.
- Implement secure access and authentication mechanisms for Prometheus and VictoriaMetrics environments.

Preferred Qualifications

- 2+ years of experience in observability, DevOps, or SRE roles.
- Strong hands-on experience with Grafana, Prometheus, and VictoriaMetrics.
- Experience configuring exporters such as Node Exporter, Windows Exporter, SNMP Exporter, and JMX/WebLogic Exporters.
- Strong understanding of PromQL and monitoring best practices.
- Experience with Git-based configuration management.
- Exposure to GoCD or similar CI/CD tools.
- Familiarity with Docker and Kubernetes environments.
- Experience with infrastructure automation tools such as Ansible or Terraform.
- Ability to work collaboratively with DevOps, development, and support teams.

📌 Observability Engineer - Grafana Prometheus & VictoriaMetrics (Navi Mumbai)
🏢 Apptad
📍 Navi Mumbai