We are looking for an experienced
Data Engineer (Spark/Scala)
with robust hands-on expertise in
Apache Spark, Databricks, Scala, PySpark, Python, and SQL
.

The role involves designing and developing large-scale data pipelines across
on-premises and cloud environments
, working with multiple file systems and data formats, modernizing legacy workflows, and supporting hybrid data architectures.

Key Responsibilities
- Design, develop, and maintain scalable data pipelines using Apache Spark, Databricks, Scala Spark, and PySpark.
- Build and support complex on-premises data workflows and hybrid on-prem-to-cloud data integration solutions.
- Integrate data across HDFS, NAS, on-prem file shares, Amazon S3, and other storage platforms.
- Work with multiple data formats including JSON, Parquet, CSV, Avro, Fixed-Length, and Excel.
- Develop optimized SQL queries for data extraction, transformation, and loading.
- Connect to multiple relational and non-relational databases and implement performance-efficient data extraction strategies.




- Develop and maintain workflow orchestration using Apache Airflow or similar scheduling tools.
- Write clean, production-grade Python code for data processing, automation, and engineering utilities.
- Develop unit, integration, and data-quality tests for data pipelines.
- Troubleshoot pipeline failures, performance bottlenecks, data quality issues, and complex multi-system integration problems.
- Support migration and modernization of legacy on-premises data processes to hybrid/cloud environments.
- Collaborate with Data Scientists, Analysts, Application Engineers, and other stakeholders.
- Create technical documentation covering pipelines, data flows, architecture, and data lineage.
- Support cloud integration initiatives, particularly across Azure environments.
- Leverage coding assistants and AI agents to improve development productivity and automate engineering tasks.

Required Skills
- Strong hands-on experience with Apache Spark and Data

📌 Data Engineer (Chennai)
🏢 Zorba AI
📍 Chennai