Job Description

Job Requirements*

/n

We are seeking an experienced Scala / Pyspark associate to design, develop, and maintain scalable data solutions on the Amazon Cloud Platform (AWS). The ideal candidate will have robust expertise in building up-to-date data pipelines, data warehousing, big data processing, and cloud-native analytics solutions. The role requires close collaboration with business stakeholders, data architects, data scientists, and application teams to deliver reliable and high-performance data platforms.

/n

Key Responsibilities*

/n

/n
Design, develop, and optimize scalable data pipelines using scala and Pyspark in AWS environment.
/n
Build and maintain batch and real-time data ingestion and processing frameworks.
/n
Develop enterprise-grade data warehousing solutions using Scala and Pyspark.
/n
Analyze existing Scala and Spark applications and identify migration requirements.
/n
Convert Scala-based ETL, batch, and streaming pipelines into PySpark frameworks.
/n
Optimize PySpark jobs for performance, scalability, and resource utilization.




/n
Support cloud modernization initiatives on AWS/Databricks/Snowflake platforms.
/n
Implement ETL/ELT processes for structured and unstructured data.
/n
Integrate data from multiple sources including databases, APIs, files, and streaming platforms.
/n
Ensure data quality, governance, security, and compliance across data platforms.
/n
Automate deployment and operational processes using CI/CD and Infrastructure as Code (IaC).
/n
Monitor data pipelines and troubleshoot production issues.
/n
Collaborate with Data Architects, Business Analysts, and Data Scientists to translate business requirements into technical solutions.
/n
Implement data models, metadata management, and data lineage best practices.
/n
Support migration of on-premises or multi-cloud data platforms to AWS.
/n
Lead the migration, modernization, and optimization of Scala and Apache Spark workloads on Cloud workplace

📌 Software Associate Bengaluru (India)
🏢 Tata Consultancy Services
📍 India