Job Title:
Senior Data Engineer / Data Engineering Developer (Databricks & Scala)

Role Summary

We are seeking a highly skilled Data Engineer with solid expertise in
Databricks, Apache Spark, and Scala
to design, develop, and optimize large-scale data processing solutions. The ideal candidate will have hands-on experience building distributed data pipelines, implementing scalable ETL frameworks, and delivering cloud-native data solutions on Azure.

Key Responsibilities
Design, develop, and maintain scalable data pipelines using Databricks, Apache Spark, and Scala.
Build ingestion, transformation, and validation frameworks for high-volume data processing.
Develop and optimize Spark jobs for performance, scalability, and reliability.
Implement and manage Databricks workflows, job orchestration, and scheduling.
Troubleshoot production issues, performance bottlenecks, and pipeline failures.
Optimize cluster utilization, Spark execution plans, and data processing efficiency.
Work with cross-functional teams to design data architectures and integration solutions.




Implement best practices for coding, testing, CI/CD, and deployment.
Support production environments and participate in incident resolution and root cause analysis.
Collaborate in Agile teams to deliver high-quality data products.

Required Skills
Strong hands-on experience in:
Databricks
Apache Spark
Scala
Spark SQL
PySpark
Experience in:
Distributed data processing frameworks
ETL/ELT pipeline development
Data modeling and data warehousing concepts
Performance tuning and optimization of Spark applications
Cloud experience:
Microsoft Azure
Azure Data Lake Storage (ADLS)
Azure Data Factory (ADF)
Solid SQL development and query optimization skills.
Experience with source control systems such as Git/GitHub.
Solid analytical and debugging skills.

Preferred Skills
Delta Lake
Unity Catalog
CI/CD pipelines (Azure DevOps, Harness)
Snowflake
Data Quali

📌 Data Engineering Developer Hyderabad
🏢 State Street
📍 Hyderabad