Specific Job Responsibilities
- Pipeline Management: Maintain high-throughput streaming pipelines to ingest logs from various sources (Firewalls, Cloud, Endpoints) to a central destination.
- Log Normalization: Write parsers to convert raw, messy logs into standard schemas (e.g., OCSF or ECS) for consistent querying.
- Cost Optimization: Implement routing logic to send "high-value" data to the SIEM and "bulk" data to low-cost Object Storage (Data Lake).
- Data Preparation: Clean and structure data to enable AI/ML detection models and advanced analytics.

Experience Required
- Data Engineering: Proficiency in Python (for ETL) and SQL (for complex querying).
- Streaming Tech: Experience with Message Queues (e.g., Kafka, Pub/Sub) and stream processing concepts.
- Log Handling: Mastery of Regex and log parsing strategies for standard formats (Syslog, CEF, JSON).
- Storage Architecture: Understanding of Data Lake principles (Parquet/Avro formats) vs. Data Warehouses.

Qualifications, Skills, & Knowledge
- Experience with Vector Databases for storing embeddings.




- Knowledge of Log Observability/Routing tools (middleware that routes logs).
- Familiarity with Big Data frameworks (e.g., Spark, Flink).
- PROFESSIONAL DEVELOPMENT EXPECTATIONS

Ability to embrace Clearwater's CLEAR core values (Commitment to Client Success, Lead with Accountability, Integrity & Collaboration, Excellence in All That We Do, Advance Colleague Success, Respect & Transparency) and culture.

The base salary range for this role is 35,000- 45,000]. Base salary is part of our total rewards package which also includes the opportunity for merit-based salary increases, eligibility for our 401(k) plan, medical, dental, vision, life and disability insurances and leaves provided in line with your work state. Our robust time-off policy includes versatile paid time off, 11 paid holidays, and paid sick time. Total compensation, including base salary to be offered, will depend on elements unique to each candidate,

📌 Senior Security Data Engineer (India)
🏢 Clearwater
📍 India