Location: Hyderabd

Experience Range: 5 to 10

We are looking for an experienced AWS Databricks Developer with solid hands-on expertise in Apache Spark, Scala, JAR development, and AWS cloud services. The candidate will be responsible for designing, developing, packaging, and deploying scalable data-processing applications on Databricks.

The role requires strong experience in developing reusable Spark frameworks and libraries using Scala, building JAR files, optimizing distributed workloads, and implementing enterprise-grade data pipelines on AWS.

Key Responsibilities

- Design and develop scalable data-processing applications using Apache Spark and Scala.
- Develop reusable frameworks, utilities, and libraries and package them as JAR files.
- Deploy and execute Scala-based JAR applications on AWS Databricks clusters.
- Build and maintain high-volume batch and streaming data pipelines.
- Develop ETL and ELT workflows using Databricks, Spark, Scala, and SQL.
- Integrate Databricks with AWS services such as S3, Glue, IAM, Secrets Manager, CloudWatch, and Redshift.
- Implement data ingestion and transformation solutions for structured, semi-structured, and unstructured data.
- Perform Spark performance tuning, including partitioning, caching, join optimization, and memory management.
- Develop and maintain Delta Lake tables and data-processing workflows.
- Configure Databricks jobs, workflows, clusters, libraries, and access controls.
- Implement error handling, logging, auditing, monitoring, and restart mechanisms.
- Write unit and integration tests for Scala and Spark applications.




- Build CI/CD pipelines to automate JAR creation, testing, and deployment.
- Troubleshoot data pipeline failures, cluster issues, and production defects.
- Participate in solution design, code reviews, release planning, and production support.
- Collaborate with data architects, analysts, DevOps teams, and business stakeholders.

Mandatory Technical Skills

Programming and Big Data

- Scala
- Apache Spark
- Spark SQL
- SQL
- Core Java concepts
- Object-oriented and functional programming
- Distributed data-processing concepts

Databricks

- AWS Databricks
- Databricks notebooks and jobs
- Databricks Workflows
- Cluster configuration and management
- Databricks Asset Bundles or equivalent deployment practices
- Delta Lake
- Unity Catalog
- Databricks libraries
- Spark performance optimization
- Job monitoring and troubleshooting

JAR Development

- Scala-based JAR development
- Building executable and reusable Spark libraries
- JAR packaging and dependency management
- Maven or SBT
- Creating assembly or fat JAR files
- Managing third-party dependencies
- Deploying JARs to Databricks clusters and jobs
- Versioning and release management

AWS Services

- Amazon S3
- AWS Glue
- AWS IAM
- Amazon CloudWatch
- AWS Secrets Manager
- Amazon Redshift
- Amazon EMR
- AWS Lambda
- Amazon SNS and SQS
- AWS Step Functions

DevOps and Development Tools

- Git
- Maven or SBT
- Jenkins, GitHub Actions, or AWS CodePipeline
- CI/CD pipeline development
- SonarQube
- JUnit, ScalaTest, or equivalent testing framework
- Jira and Agile development tools

📌 Aws Data Engineer (Hyderabad)
🏢 Tata Consultancy Services
📍 Hyderabad