Job Description : - Core experience : Minimum 5 years of hands-on experience in data processing pipeline and API development, with robust proficiency in Python (PySpark) and Scala a must- Big data & distributed systems : Apache Spark, Databricks, Blob Storage, BigQuery and managed Spark (Dataproc) on GCP; processing structured, semi-structured and multi-modal data (text, image, audio, video)- Lakehouse & table formats : Delta Lake, Apache Iceberg and Apache Hudi- Data modelling & stores : Designing data schemas, tables and partitioning strategies across SQL and NoSQL databases such as Cosmos DB and Redis- Pipelines & orchestration : Building batch and streaming ETL/ELT processes over pub/sub platforms (Kafka, Azure Event Hubs, Google Cloud Pub/Sub), orchestrated with Airflow on Kubernetes, in containerized, scalable and serverless architectures- Infrastructure as Code : Terraform or equivalent- Performance & cost optimization : Spark tuning, shuffle optimization, caching, cluster sizing,



query optimization, debugging long-running jobs and cloud cost control- Data quality & lineage : Defining data quality rules and data lineage definitions, and creating monitoring alerts out of them- Testing for data pipelines : Data validation, regression tests, reconciliation and end-to-end pipeline testing- Observability & reliability : Centralized logging, metrics, tracing, job monitoring, SLAs/SLOs and incident troubleshooting- ML engineering : Feature store, feature and training pipelines, model deployment and serving, model monitoring with drift detection and retraining (MLflow / MLOps)- AI-assisted development : Hands-on experience using tools such as GitHub Copilot, Claude Code or equivalent for code generation, debugging, testing and developer productivity (ref:hirist.tech)

📌 ValueLabs - Data Solution Engineer (Hyderabad)
🏢 VALUELABS SOLUTIONS
📍 Hyderabad