Data Engineer (Spark/Scala)

About The Role
We are seeking an experienced Data Engineer to design, build, and optimize complex data workflows across on-premises and cloud environments. This role requires deep hands-on expertise in Apache Spark, Databricks, and Scala/PySpark, along with strong SQL and Python skills, to build robust, high-performance data pipelines. You will work extensively on complex on-prem workflows, integrating data across multiple file systems and formats, migrating and modernizing legacy processes, and ensuring efficient, reliable data movement across heterogeneous environments.

Key Responsibilities
- Design, develop, and maintain large-scale data pipelines using Apache Spark, Databricks, Scala Spark, and PySpark
- Build and support complex on-premises data workflows, including migration/hybrid on-prem-to-cloud integration patterns
- Integrate data across diverse file systems (on-prem file shares, NAS, HDFS, S3) and formats JSON, Parquet, Fixed-Length, CSV, Excel, Avro
- Write efficient, optimized SQL for data extraction, transformation,



and loading across relational databases
- Connect to and extract data efficiently from various source databases, tuning queries and pipelines for performance at scale
- Develop and maintain workflow orchestration using Airflow (or similar schedulers) for reliable, monitored pipeline execution
- Write clean, production-grade Python code for data processing, automation, and tooling
- Build and maintain unit/integration tests for data pipelines to ensure data quality and reliability
- Create and maintain explicit technical documentation for pipelines, data flows, and system architecture
- Troubleshoot and resolve data pipeline failures, performance bottlenecks, and data quality issues in complex, multi-system workflows
- Collaborate with cross-functional teams (data science, analytics, application engineering) to support downstream data consumption
- Support cloud integration efforts, particularly with Azure, as wor

📌 Data Engineer-Spark,Scala (Hyderabad)
🏢 Zorba AI
📍 Hyderabad