Senior Big Data Engineer - 4916

IT · Development Team

Overview

Location: hybrid · Hyderabad

Employment Type: full time

Experience: 7+ years

Compensation: INR 1,666,080 - 2,082,600

Overview

Senior Big Data Engineer role focused on designing, developing, and supporting large-scale data processing applications on AWS Cloud. The position builds scalable data platforms, optimizes high-volume batch workloads, and develops API-driven data solutions. Emphasis includes distributed computing frameworks, cloud-native microservices, and CI/CD automation.

Key Responsibilities

• Design, develop, implement, and maintain scalable Big Data applications and data processing pipelines on AWS Cloud.

- Build and support API-based microservices and batch processing solutions using Java, Scala, Kotlin, and Python.
- Develop and optimize large-scale distributed data processing applications utilizing Apache Spark, Hadoop Ecosystem, HDFS, MapReduce, and Cassandra.
- Create and maintain RESTful APIs and SOAP Web Services for enterprise data integration.
- Develop cloud-native solutions leveraging AWS services including EMR, EC2, ECS, S3, Airflow, AWS Step Functions, and AWS API Gateway.
- Perform performance tuning and optimization of Spark, Hadoop, EMR, Java, and Python-based applications.
- Develop Linux-based automation scripts using Shell Scripting and Python.
- Collaborate with cross-functional teams including Data Engineering, Architecture, DevOps, and Product teams.
- Implement and maintain CI/CD pipelines to automate application deployment and infrastructure provisioning.
- Participate in code reviews, troubleshoot production issues, and ensure adherence to engineering best practices.

Required Qualifications

• 7+ years of experience in software development, data engineering, and Big Data application development.




- Proven experience building scalable data platforms and distributed processing systems on AWS Cloud.
- Solid hands-on experience with Scala, PySpark, Java, Kotlin, and Python.
- Experience with Apache Spark, Hadoop Ecosystem, HDFS, MapReduce, and Cassandra.
- REST API development and SOAP Web Services experience.
- Microservices Architecture experience.
- Experience with AWS services: Amazon EMR, EC2, ECS, S3, AWS API Gateway, AWS Step Functions, Apache Airflow.
- Linux Administration experience.
- Shell Scripting and Python Automation experience.
- CI/CD pipeline development and automation using Maven, Gradle, Jenkins, Artifactory, AWS CodeCommit, AWS CloudFormation.

Technical Skills

• Scala
- PySpark
- Java
- Kotlin
- Python
- Apache Spark
- Hadoop Ecosystem
- HDFS
- MapReduce
- Cassandra
- REST API development
- SOAP Web Services
- Microservices Architecture
- Amazon EMR
- EC2
- ECS
- S3
- AWS API Gateway
- AWS Step Functions
- Apache Airflow
- Linux Administration
- Shell Scripting
- Python Automation
- Maven
- Gradle
- Jenkins
- Artifactory
- AWS CodeCommit
- AWS CloudFormation
- CI/CD Pipeline Development and Automation

Nice-to-have

• Snowflake
- Git
- GitHub
- GitLab
- Bitbucket
- IntelliJ IDEA
- PyCharm
- Agile/Scrum

Tools/Platforms

• AWS Cloud
- Amazon EMR
- EC2
- ECS
- S3
- AWS API Gateway
- AWS Step Functions
- Apache Airflow
- Jenkins
- Artifactory
- AWS CodeCommit
- AWS CloudFormation
- Maven
- Gradle
- Linux
- Cassandra
- Hadoop
- Apache Spark
- Snowflake
- GitHub
- GitLab
- Bitbucket
- IntelliJ IDEA
- PyCharm

Preferred Qualifications

• Experience with Snowflake for data warehousing and analytics.
- Experience developing automated data ingestion and transformation pipelines using Spark and Snowflake.
- Familiarity with DevOps and source control tools including Git, GitHub, GitLab, Bitbucket, IntelliJ IDEA, and PyCharm.
- Experience working within Agile/Scrum development environments.

📌 Senior Big Data Engineer - 4916_7+ yrs_Hyderabad (Hyderabad)
🏢 Arminus
📍 Hyderabad