Specific Job Responsibilities

Pipeline Management: Maintain high-throughput streaming pipelines to ingest logs from various sources (Firewalls, Cloud, Endpoints) to a central destination.
Log Normalization: Write parsers to convert raw, messy logs into standard schemas (e.g., OCSF or ECS) for consistent querying.
Cost Optimization: Implement routing logic to send "high-value" data to the SIEM and "bulk" data to low-cost Object Storage (Data Lake).
Data Preparation: Clean and structure data to enable AI/ML detection models and advanced analytics.

Experience Required

Data Engineering: Proficiency in Python (for ETL) and SQL (for complex querying).
Streaming Tech: Experience with Message Queues (e.g., Kafka, Pub/Sub) and stream processing concepts.
Log Handling: Mastery of Regex and log parsing strategies for standard formats (Syslog, CEF, JSON).
Storage Architecture: Understanding of Data Lake principles (Parquet/Avro formats) vs. Data Warehouses.

Qualifications, Skills, & Knowledge

Experience with Vector Databases for storing embeddings.




Knowledge of Log Observability/Routing tools (middleware that routes logs).
Familiarity with Big Data frameworks (e.g., Spark, Flink).
PROFESSIONAL DEVELOPMENT EXPECTATIONS

Ability to embrace Clearwater's CLEAR core values (Commitment to Client Success, Lead with Accountability, Integrity & Collaboration, Excellence in All That We Do, Advance Colleague Success, Respect & Transparency) and culture.

The base salary range for this role is 35,000- 45,000]. Base salary is part of our total rewards package which also includes the prospect for merit-based salary increases, eligibility for our 401(k) plan, medical, dental, vision, life and disability insurances and leaves provided in line with your work state. Our robust time-off policy includes flexible paid time off, 11 paid holidays, and paid sick time. Total compensation, including base salary to be offered, will depend on elements unique to each candidate, including applicable

📌 Senior Security Data Engineer (India)
🏢 Clearwater
📍 India