To design, implement, and optimize a full-stack observability strategy using the Splunk Observability Cloud (formerly SignalFx) and Splunk Enterprise/Cloud. You will ensure that engineering teams have 360-degree visibility into system health, moving the organization from reactive "firefighting" to proactive "pattern-based" incident prevention.
Responsibilities
- Data Orchestration: Architect the ingestion of the "Three Pillars" (Metrics, Logs, Traces) using OpenTelemetry (OTel) collectors.
- Aggregation Strategy: Develop logic to aggregate high-cardinality data to reduce "noise" while maintaining "signal" for troubleshooting.
- Analytical Modeling: Use SPL (Search Processing Language) and SignalFlow to perform pattern analysis, detecting anomalies before they trigger traditional threshold alerts.
- Visual Storytelling: Build executive and technical dashboards that correlate disparate data points (e.g., showing how a spike in 500-errors in Logs relates to a specific span in a Trace).
1. Telemetry & Data Specialization
- Logs: Proficiency in "Logging-in-Context." You must be able to link logs directly to trace IDs so developers can jump from a failing trace to the specific line of code in the logs.
- Metrics: Expertise in SignalFlow (Splunk s background streaming analytics language). You should know how to calculate percentiles ($P95, $P99$), rates of change, and historical averages.
- Traces: Deep understanding of Distributed Tracing. You must know how to instrument applications (Java, Python, Go) to capture spans and identify bottlenecks in microservices.
2.
Pattern Analysis & Aggregation
- Anomaly Detection: Ability to configure Metric Finder and MDetector using standard deviations or "Mean Absolute Deviation" to find outliers.
- Data Scrubbing: Skills in using Splunk Ingest Actions or Edge Processors to filter, mask, or aggregate data at the edge to save on license costs and improve search speed.
- Pattern Discovery: Using Splunk s machine learning commands (e. g. , findkeywords , cluster ) to group millions of log events into a few dozen "patterns" for faster root cause analysis.
3. Hands on - Dashboards & Visualization
- High-Cardinality Handling: Designing dashboards that don t "break" when viewing thousands of containers.
- Contextual Drill-downs: Building "Glass Tables" (in ITSI) or Unified Dashboards that allow a user to click a metric and immediately see the associated logs.
- Frameworks: Familiarity with the Dashboard Studio and JSON-based dashboard definitions for version control (GitOps).
Preferred Qualifications & Certifications
- Splunk Cloud Certified Metrics User: Focuses on the metrics and alerting side.
- Splunk Core Certified Power User: Essential for mastering complex SPL for log analysis.
- OpenTelemetry Expert: Knowledge of the OTel Collector configuration ( receivers , processors , exporters ) is currently the most "in-demand" skill for this role.
Disclaimer: This job posting has been aggregated from external source. Role details, content, and availability are subject to change. Applicants are advised to confirm the latest information directly on the company website before applying.