Key Responsibilities:

- Design and develop scalable data-processing applications using Apache Spark and Scala.
- Develop reusable frameworks, utilities, and libraries and package them as JAR files.
- Deploy and execute Scala-based JAR applications on AWS Databricks clusters.
- Build and maintain high-volume batch and streaming data pipelines.
- Develop ETL and ELT workflows using Databricks, Spark, Scala, and SQL.
- Integrate Databricks with AWS services such as S3, Glue, IAM, Secrets Manager, CloudWatch, and Redshift.
- Implement data ingestion and transformation solutions for structured, semi-structured, and unstructured data.
- Perform Spark performance tuning, including partitioning, caching, join optimization, and memory management.
- Develop and maintain Delta Lake tables and data-processing workflows.
- Configure Databricks jobs, workflows, clusters, libraries, and access controls.
- Implement error handling, logging, auditing, monitoring, and restart mechanisms.
- Write unit and integration tests for Scala and Spark applications.
- Build CI/CD pipelines to automate JAR creation, testing, and deployment.
- Troubleshoot data pipeline failures, cluster issues, and production defects.
- Participate in solution design, code reviews, release planning,



and production support.
- Collaborate with data architects, analysts, DevOps teams, and business stakeholders.

Desired Candidate Profile:

- Strong hands-on coding experience in Scala and Apache Spark.
- Proven experience in developing and deploying Scala-based JAR applications.
- Strong understanding of Spark architecture, distributed computing, and performance tuning.
- Experience in building enterprise-scale data pipelines using AWS Databricks.
- Good understanding of data lake, data warehouse, and Delta Lake concepts.
- Ability to troubleshoot complex production issues independently.
- Solid analytical, debugging, and problem-solving skills.
- Good communication and stakeholder-management capabilities.
- Experience working in Agile or Scrum delivery environments

Preferred Skills:

- Spark Structured Streaming
- Apache Kafka
- Python or PySpark
- AWS Lake Formation
- Terraform or CloudFormation
- Apache Airflow
- Snowflake
- Data warehouse and data lake architecture
- Databricks Asset Bundles
- Docker and Kubernetes
- Experience with BFSI or other regulated domains
- AWS or Databricks certification

📌 AWS Databricks Spark Scala Jar Developer (Hyderabad)
🏢 Tata Consultancy Services
📍 Hyderabad