Job Title:
Senior Data Engineer / Data Engineering Developer (Databricks & Scala)

Role Summary

We are seeking a highly skilled Data Engineer with strong expertise in
Databricks, Apache Spark, and Scala
to design, develop, and optimize large-scale data processing solutions. The ideal candidate will have hands-on experience building distributed data pipelines, implementing scalable ETL frameworks, and delivering cloud-native data solutions on Azure.

Key Responsibilities
- Design, develop, and maintain scalable data pipelines using Databricks, Apache Spark, and Scala.
- Build ingestion, transformation, and validation frameworks for high-volume data processing.
- Develop and optimize Spark jobs for performance, scalability, and reliability.
- Implement and manage Databricks workflows, job orchestration, and scheduling.
- Troubleshoot production issues, performance bottlenecks, and pipeline failures.
- Optimize cluster utilization, Spark execution plans, and data processing efficiency.
- Work with cross-functional teams to design data architectures and integration solutions.




- Implement best practices for coding, testing, CI/CD, and deployment.
- Support production environments and participate in incident resolution and root cause analysis.
- Collaborate in Agile teams to deliver high-quality data products.

Required Skills
- Strong hands-on experience in:
- Databricks
- Apache Spark
- Scala
- Spark SQL
- PySpark
- Experience in:
- Distributed data processing frameworks
- ETL/ELT pipeline development
- Data modeling and data warehousing concepts
- Performance tuning and optimization of Spark applications
- Cloud experience:
- Microsoft Azure
- Azure Data Lake Storage (ADLS)
- Azure Data Factory (ADF)
- Strong SQL development and query optimization skills.
- Experience with source control systems such as Git/GitHub.
- Solid analytical and debugging skills.

Preferred Skills
- Delta Lake
- Unity Catalog
- CI/CD pipelines (Azure DevOps, Harness)
- Snowflake
- Data Quali

📌 Data Engineering Developer (Hyderabad)
🏢 State Street
📍 Hyderabad