Key Responsibilities:
Design and develop scalable data-processing applications using Apache Spark and Scala.
Develop reusable frameworks, utilities, and libraries and package them as JAR files.
Deploy and execute Scala-based JAR applications on AWS Databricks clusters.
Build and maintain high-volume batch and streaming data pipelines.
Develop ETL and ELT workflows using Databricks, Spark, Scala, and SQL.
Integrate Databricks with AWS services such as S3, Glue, IAM, Secrets Manager, CloudWatch, and Redshift.
Implement data ingestion and transformation solutions for structured, semi-structured, and unstructured data.
Perform Spark performance tuning, including partitioning, caching, join optimization, and memory management.
Develop and maintain Delta Lake tables and data-processing workflows.
Configure Databricks jobs, workflows, clusters, libraries, and access controls.
Implement error handling, logging, auditing, monitoring, and restart mechanisms.
Write unit and integration tests for Scala and Spark applications.
Build CI/CD pipelines to automate JAR creation, testing, and deployment.
Troubleshoot data pipeline failures, cluster issues, and production defects.
Participate in solution design, code reviews, release planning,



and production support.
Collaborate with data architects, analysts, DevOps teams, and business stakeholders.

Desired Candidate Profile:
Robust hands-on coding experience in Scala and Apache Spark.
Proven experience in developing and deploying Scala-based JAR applications.
Solid understanding of Spark architecture, distributed computing, and performance tuning.
Experience in building enterprise-scale data pipelines using AWS Databricks.
Good understanding of data lake, data warehouse, and Delta Lake concepts.
Ability to troubleshoot complex production issues independently.
Solid analytical, debugging, and problem-solving skills.
Good communication and stakeholder-management capabilities.
Experience working in Agile or Scrum delivery environments

Preferred Skills:
Spark Structured Streaming
Apache Kafka
Python or PySpark
AWS Lake Formation
Terraform or CloudFormation
Apache Airflow
Snowflake
Data warehouse and data lake architecture
Databricks Asset Bundles
Docker and Kubernetes
Experience with BFSI or other regulated domains
AWS or Databricks certification

📌 Aws Databricks Spark Scala Jar Developer Hyderabad (India)
🏢 Tata Consultancy Services
📍 India