Site Reliability Engineer (Hyderabad)

Site Reliability Engineer (Hyderabad)

17 Nov
|
MarbleX
|
Hyderabad

17 Nov

MarbleX

Hyderabad

Role Overview

We are looking for an experienced Site Reliability Engineer (SRE) to manage, operate, and maintain the ingestion layer of a large-scale observability platform. The role involves ensuring continuous collection and delivery of metrics, logs, and traces from monitored application and infrastructure systems into centralized data stores.

The ideal candidate should have robust hands-on experience in Linux administration, OpenTelemetry, Vector, Prometheus exporters, Kafka, Grafana, and scripting. The candidate will be responsible for day-to-day operations, application onboarding, telemetry validation, incident management, troubleshooting, and automation.

Since the platform supports critical business operations, ensuring uninterrupted telemetry collection and maintaining service continuity will be the primary responsibility.

Key Responsibilities

1. Ingestion Monitoring & Operations

- Monitor and maintain the end-to-end observability ingestion layer, including Node Exporter, JMX Exporter, OpenTelemetry agents and collectors, Vector agents, SNMP, Blackbox, and syslog collection.
- Monitor ingestion health, scrape success, ingestion rates, consumer-group lag, buffer utilization, queue depth, and data freshness.
- Identify and resolve silent source loss, pipeline failures, and telemetry collection gaps.
- Perform routine health checks, capacity checks, certificate renewals, patching, and component upgrades.

2. Agent & Collector Deployment

- Deploy, configure, upgrade, and maintain telemetry agents across application and infrastructure hosts.
- Onboard applications and infrastructure components, including Kafka, Cassandra, KeyDB, PostgreSQL, network, and security systems.
- Maintain Vector agent and aggregator pipelines, including file sources, multiline parsing, VRL transformations, disk buffering, and sink configurations.
- Ensure proper telemetry labels and resource attributes across metrics, logs, and traces.

3. Incident Management & Troubleshooting

- Act as the first responder for ingestion-related alerts and incidents.
- Troubleshoot exporter failures, pipeline stalls, connectivity issues, parsing errors,



and consumer lag.
- Acknowledge P1 alerts within 5 minutes and P2 alerts within 15 minutes.
- Perform root cause analysis, implement preventive actions, and follow defined escalation procedures.
- Ensure proper incident documentation and timely shift handovers.

4. Application Onboarding & Telemetry Validation

- Validate network connectivity, routing, firewall rules, port reachability, DNS resolution, and certificate validity.
- Ensure telemetry completeness and accuracy after every onboarding or configuration change.
- Investigate data gaps, count mismatches, cardinality issues, and coverage discrepancies.
- Follow approved onboarding runbooks and verification procedures.

5. Automation & Scripting

- Automate repetitive operational tasks using Bash/Shell scripting and Python.
- Develop scripts for agent deployment, configuration templating, verification checks, log analysis, and shift reporting.
- Follow Git-based version control and configuration-as-code practices.

6. Dashboards, Alerting & Documentation

- Build and maintain Grafana dashboards and alert rules for monitoring ingestion health.
- Validate alert rules and ensure referenced metrics are available.
- Maintain operational documentation, including runbooks, SOPs, onboarding guides, shift handover notes, and incident reports.

7. Security & Compliance

- Ensure secure configuration of collection endpoints, OTLP receivers, and log mounts.
- Prevent exposure of unauthenticated ingestion endpoints on routable interfaces.
- Follow approved security controls, patching procedures, and operational compliance standards.

Required Technical Skills
- Linux Administration: Strong experience with RHEL/Ubuntu, command-line operations, systemd, networking,



and performance troubleshooting.
- OpenTelemetry: Java agent instrumentation, Collector pipelines, OTLP, receivers, processors, and exporters.
- Vector.dev: File sources, multiline parsing, VRL transformations, disk buffering, and sinks.
- Prometheus Exporters: Node Exporter, JMX Exporter, SNMP Exporter, Blackbox Exporter, and PromQL.
- Prometheus / VictoriaMetrics: Scrape configuration, relabelling, and remote write.
- Log & Trace Management: VictoriaLogs (LogsQL) or OpenSearch; exposure to trace stores.
- Apache Kafka: Topics, partitions, consumer groups, offsets, lag, and retention.
- Containerization: Docker and container operations; working knowledge of Kubernetes.
- Grafana: Dashboard creation, panel queries, and alert rule configuration.
- Scripting: Bash/Shell scripting and Python for operational automation.
- Networking: TCP/IP, routing, DNS, TLS, firewall rules, and port analysis.
- Version Control: Git and configuration-as-code practices.
- Incident Management: Incident triage, escalation, root cause analysis, and documentation.

24x7 Shift & Support Requirements
- Must be willing to work in a 24x7 rotational shift environment.
- Shifts will rotate across morning, evening, and night schedules, including weekends and public holidays.
- Take primary on-call responsibility for ingestion alerts during assigned shifts.
- Complete documented shift handovers covering open incidents, ingestion health, suppressed alerts, and ongoing activities.
- Follow defined incident acknowledgement and escalation timelines.
- Ensure operational continuity and timely incident resolution.

Key Expectations
- Ensure uninterrupted telemetry collection and ingestion across monitored systems.
- Maintain high availability and reliability of the observability ingestion layer.
- Proactively identify and resolve ingestion failures and data collection gaps.
- Ensure accurate, complete, and consistent telemetry data.
- Reduce manual effort through automation and operational improvements.
- Maintain proper documentation and follow established operational procedures.

📌 Site Reliability Engineer (Hyderabad)
🏢 MarbleX
📍 Hyderabad

Reply to this offer

Impress this employer describing Your skills and abilities, fill out the form below and leave Your personal touch in the presentation letter.

Subscribe to this job alert:

Get the latest job offers by email for: site reliability engineer (hyderabad) / hyderabad

Subscribe to this job alert:

Get the latest job offers by email for: site reliability engineer (hyderabad) / hyderabad