18 Oct
|
Tenarai Technologies
|
Bengaluru
18 Oct
Tenarai Technologies
Bengaluru
Title:Observability Engineer (New Relic & OpenTelemetry) Primary swim lane Configure (lead) • supports Enhance and Scope recommendations Experience 710 years total; 4+ years hands-on observability; 2+ years New Relic
:1. Key Responsibilities
Agent deployment & migration
- Datadog cut-over for Meridian — Replace the Datadog agent with the New Relic infrastructure/APM agent on the ~54 Meridian nodes (Docker on EC2), updating onboarding scripts so new client instances come up instrumented.
- Meridian MCP from zero — Instrument the MCP pipeline end to end (Airflow, Step Functions, Lambda, Neptune, AgentCore Gateway) — currently no observability exists.
- Pattern selection — Apply the agent-first pattern; where an agent cannot be installed (e.g., SaaS such as HashiCorp Vault, Cloudflare, WorkOS, MongoDB Atlas) fall back to OpenTelemetry, cloud integrations, API polling, SNMP or synthetic checks, and document when to use each.
OpenTelemetry & cloud integrations
- OTel collectors — Add New Relic exporters/receivers alongside existing Prometheus and OTel flows so services dual-ship during migration.
- Cloud telemetry — Configure AWS integrations (CloudWatch metric streams, CloudTrail, EC2, ECS/Fargate, S3, Neptune, Lambda, Step Functions) across the correct AWS accounts; handle Azure Monitor and GCP sources for cross-cloud workflows.
- Database coverage — Stand up MySQL, PostgreSQL, MongoDB Atlas and Neptune monitoring for the four Meridian/MCP data stores.
Operational logs
- Log forwarding — Add New Relic as a destination next to Loki and Azure Monitor for operational (not security-pipeline) logs; keep MDR security telemetry clearly out of scope.
- Parsing & patterns — Build log parsing rules and pattern extraction so logs are queryable in NRQL, reusing prior art from Datadog, Loki and Azure Monitor.
Alerting, incident routing & validation
- Alert conditions — Draft alert policies and conditions per technology (e.g., MySQL, Airflow, EC2) with owning engineers, tuning for noise versus coverage.
- Jira-first routing — Route alerts into Jira work items of the right severity so existing on-call schedules, paging and Teams notifications apply; do not duplicate on-call in New Relic.
- Service health — Implement synthetic transactions and health-endpoint/webhook patterns that can drive the status page and customer portal service status.
- End-to-end proof — For each component, prove signal ? New Relic ? dashboard ? alert ? Teams + Jira works, then capture it as a runbook/playbook.
Tracking & enablement
- Maintain the Produce/Consume inventory status for each component (metrics, logs, events, traces, agent; dashboard, alert, Teams, Jira, portal, status, MCP).
- Write the playbooks Cyderes engineers will follow so observability becomes part of the "definition of done.
"Service group Technologies
Meridian — Provision Docker, AWS ECR, AWS SSM, Jenkins CI/CD pipelines; SaaS Customer Manager, Update Manager (per-client provisioning)
Meridian — Ingress Tunnel-free and tunnel-dependent connector VMs (~800 connectors), tunnel proxy server, MySQL data cache on AWS EC2 (per-client containers)
Meridian — Process / Analyze Apache Airflow (scheduler/workers) with PostgreSQL backend, Action Manager; Python ML Merger; LLM VM (Ollama, IBM Granite)
Meridian — Visualize / Store Multi-tenant Web UI and API; Cloudflare (WAF/CDN/DNS/DDoS, R2); WorkOS (SSO/SAML); MongoDB Atlas; AWS S3
Meridian MCP Airflow + Merger (produce) ? S3 Uploader (export) ? AWS Step Functions / Lambda staging (transform) ? Amazon Neptune snapshot, bulk load, validation (load) ? Bedrock AgentCore Gateway, Lambda, Claude, MCP clients (consume)
Vault HashiCorp Vault (secrets storage and rotation; SaaS dependency for every service — agentless monitoring)
COAL (Deploy) AWS ECS, AWS Fargate; frontend, backend and worker services
Platform AWS Control Tower, AFT (Account Factory for Terraform) pipeline, Terraform, GitHub Actions, Argo CD, Backstage
Integrations Jira (incidents, on-call schedules, paging), Microsoft Teams, public status page, customer portal, Cyderes MCP
Clouds / legacy AWS (primary, growing under private pricing), Azure and GCP; legacy Prometheus, Thanos, Grafana, Loki, Datadog,
Azure Monitor
- Required Technical Skills
Area Expectation
Current Relic (must) Infrastructure and APM agents, NRQL, log management and parsing, alerts and applied intelligence, synthetics, workflows/destinations, cloud integrations
OpenTelemetry (must) Collector configuration, receivers/exporters, semantic conventions, SDK/auto-instrumentation basics, dual-shipping
Linux & containers (must) Linux administration, Docker and Docker Compose, systemd, syslog, agent packaging and rollout
AWS (must) EC2, ECS/Fargate, Lambda, Step Functions, S3, Neptune, CloudWatch, CloudTrail, IAM for monitoring roles, multi-account setup
Legacy tooling (must) Working knowledge of Prometheus, Thanos, Grafana, Loki and Datadog to migrate dashboards/alerts and reuse prior art
Data stores Monitoring MySQL, PostgreSQL, MongoDB Atlas and graph DBs (Neptune); Airflow metrics
Integrations Jira alert integration and severity mapping; Microsoft Teams; webhooks
Scripting Bash and Python for automation; comfortable reading Terraform to hand off configs
- Domain & Functional Knowledge
- SRE and operations practice — SLIs/SLOs, golden signals (rate, errors, duration, saturation), MTTR, alert fatigue management.
- Multi-tenant SaaS — per-client ingress containers versus shared services; tenant-safe dashboards (one customer never sees another's data).
- Security/MDR context — understanding that at Cyderes "logs" often means customer security telemetry; separate operational logs from the product pipeline.
- Observability migrations — running legacy and new platforms in parallel, validating parity, then decommissioning.
- Stakeholder working style — gathering requirements from app developers and getting sign-off on dashboards and alerts.
- Experience & Qualifications
- 7–10 years in infrastructure, SRE or DevOps roles, including 4+ years dedicated to observability/monitoring.
- 2+ years hands-on New Relic delivery; at least one migration from Datadog, Prometheus/Grafana or similar to New Relic.
- Production OpenTelemetry collector deployments at scale.
- Experience instrumenting AWS-heavy, containerized workloads; exposure to Azure and GCP.
- Bachelor's degree in Computer Science/Engineering or equivalent experience.
📌 Observability Engineer/lead/Architect (Bengaluru)
🏢 Tenarai Technologies
📍 Bengaluru