Experience Level: Mid (3–5 years)We’re seeking a highly skilled Data Engineer to design, build, and maintain production-grade data pipelines that process and transform terabytes of data. In this role, you’ll collaborate closely with data scientists and other SWEs to ensure that our data infrastructure is scalable, reliable, and cost-effective. Key Responsibilities:

- Pipeline Development & Deployment:
- Architect, develop, and deploy batch and streaming pipelines using Airflow and containerized workflows for cyber-security use-cases.
- Containerize data-processing jobs with Docker, orchestrate with Kubernetes, and manage releases with Helm charts.

- Architect, develop, and deploy batch and streaming pipelines using Airflow and containerized workflows for cyber-security use-cases.
- Containerize data-processing jobs with Docker, orchestrate with Kubernetes, and manage releases with Helm charts.
- Distributed Computing:

- Build high-throughput data transformations using Dask or Apache Spark.
- Maintain training data clusters across hybrid (on-prem and cloud environments).
- Optimize training jobs for performance, resiliency, and cost.

- Build high-throughput data transformations using Dask or Apache Spark.
- Maintain training data clusters across hybrid (on-prem and cloud environments).
- Optimize training jobs for performance, resiliency, and cost.
- Monitoring & Reliability:

- Implement observability (logging, metrics, alerting) to maintain pipeline health and SLA adherence.
- Troubleshoot, debug, and resolve data-processing failures in production.

- Implement observability (logging, metrics, alerting) to maintain pipeline health and SLA adherence.
- Troubleshoot, debug, and resolve data-processing failures in production.
- Collaboration & Best Practices:

- Work with cross-functional teams to define data contracts, schemas, and quality checks.
- Enforce software engineering best practices: CI/CD, code reviews, automated testing, and documentation.





- Work with cross-functional teams to define data contracts, schemas, and quality checks.
- Enforce software engineering best practices: CI/CD, code reviews, automated testing, and documentation.
- Data Modeling & Storage:

- Design and maintain data models and schemas for AI/ML continuous training use cases.
- Load data into cloud storage and lakes, ensuring performance and accessibility.

- Design and maintain data models and schemas for AI/ML continuous training use cases.
- Load data into cloud storage and lakes, ensuring performance and accessibility.

Required Qualifications

- 3–5 years of professional experience designing and operating production data pipelines at scale.
- Containerization & Orchestration: Expertise with Docker, Kubernetes, and Helm.
- Workflow Management: Hands-on experience building DAG-based pipelines in Apache Airflow.
- Programming: Strong proficiency in Python for data engineering tasks.
- Distributed Frameworks: Practical experience with Dask or Apache Spark for large-scale data processing.
- Cloud Fundamentals: Familiarity with deploying and managing services in a cloud environment.

Preferred Skills & Nice-to-Haves

- Compiled Languages: Experience writing data services in Go or Rust.
- GCP Proficiency: Hands-on with Google Cloud services (e.g., Pub/Sub, Big Query, Cloud Storage, GKE). Equivalent experience in other public cloud providers is fine.
- ML Pipelines: Exposure to deploying cross-cluster model-training workflows using Ray or similar frameworks.
- Infrastructure as Code: Familiarity with Terraform for deployment.
- Security & Compliance: Knowledge of data governance, encryption, and role-based access control.

What We Offer

- Competitive salary and equity package.
- Flexible work hours.
- Health, dental, and vision perks
- Collaborative, mission-driven environment

If you’re passionate about building robust data platforms and thrive in a fast-paced, growth-oriented environment, we’d love to hear from you! Please send your resume and a brief cover letter highlighting relevant projects to [email protected]

📌 Senior Data Engineer (Production Training Pipelines) (Varthur)
🏢 Versa Networks
📍 Varthur