Job Description

Job Requirements*

/n

We are seeking an experienced Scala / Pyspark associate to design, develop, and maintain scalable data solutions on the Amazon Cloud Platform (AWS). The ideal candidate will have strong expertise in building up-to-date data pipelines, data warehousing, big data processing, and cloud-native analytics solutions.

The role requires close collaboration with business stakeholders, data architects, data scientists, and application teams to deliver reliable and high-performance data platforms.

/n

Key Responsibilities*

/n

/n

- Design, develop, and optimize scalable data pipelines using scala and Pyspark in AWS environment.

/n

- Build and maintain batch and real-time data ingestion and processing frameworks.

/n

- Develop enterprise-grade data warehousing solutions using Scala and Pyspark.

/n

- Analyze existing Scala and Spark applications and identify migration requirements.

/n

- Convert Scala-based ETL, batch, and streaming pipelines into PySpark frameworks.

/n

- Optimize PySpark jobs for performance, scalability, and resource utilization.

/n

- Support cloud modernization initiatives on AWS/Databricks/Snowflake platforms.

/n

- Implement ETL/ELT processes for structured and unstructured data.

/n

- Integrate data from multiple sources including databases, APIs, files, and streaming platforms.

/n

- Ensure data quality, governance, security, and compliance across data platforms.

/n





- Automate deployment and operational processes using CI/CD and Infrastructure as Code (IaC).

/n

- Monitor data pipelines and troubleshoot production issues.

/n

- Collaborate with Data Architects, Business Analysts, and Data Scientists to translate business requirements into technical solutions.

/n

- Implement data models, metadata management, and data lineage best practices.

/n

- Support migration of on-premises or multi-cloud data platforms to AWS.

/n

- Lead the migration, modernization, and optimization of Scala and Apache Spark workloads on Cloud environment, including conversion of Scala-based Spark applications to PySpark, performance tuning, cluster optimization, dependency management, and ensuring scalable, cost-effective, and resilient data processing solutions.

/n

/n

Required Technical Skills

/n

/n

- Scala

/n

- PySpark

/n

- AWS Glue

/n

- AWS S3

/n

- Step Functions

/n

- Glue

/n

- Lambda

/n

- Pub/Sub

/n

- Python

/n

- SQL (Advanced)

/n

- Event Bridge

/n

- ECS

/n

- EKS

/n

- Snowflake

/n

/n

Section IV - Job Qualifications & Skills

/n

Section

/n

Details / Example Content

/n

Domain

/n

CMTS

/n

Soft Skills

/n

/n

- Excellent communication

/n

- Team collaboration

/n

- Documentation and knowledge sharing

/n

/n

Education Requirements

/n

Bachelor's/master's in computer science or equivalent (Preferred)

/n

Certifications

/n

AWS Cloud Data Engineer (Preferred)

/n

Snow pro certifications

📌 Software Associate (Bengaluru)
🏢 Tata Consultancy Services
📍 Bengaluru