Role: Senior Big Data Engineer
Position Type: Full-Time Contract (40hrs/week)
Contract Duration: 6+ months
Work Schedule: 8 hours/day (Mon-Fri)
Location: Hybrid 2x days onsite (Wed/Thur) - Hyderabad, India Job Description

We are seeking an experienced Senior Big Data Engineer with 7+ years of expertise in designing, developing, and supporting large-scale data processing applications on AWS Cloud. The ideal candidate will have strong hands-on experience with Big Data technologies, distributed computing frameworks, cloud-native microservices, batch processing, and CI/CD automation. This role will be responsible for building scalable data platforms, optimizing high-volume batch workloads, and developing API-driven data solutions that support critical business initiatives.

Key Responsibilities

- Design, develop, implement, and maintain scalable Big Data applications and data processing pipelines on AWS Cloud.
- Build and support both API-based microservices and batch processing solutions using Java, Scala, Kotlin, and Python.
- Develop and optimize large-scale distributed data processing applications utilizing:
- Apache Spark
- Hadoop Ecosystem
- HDFS
- MapReduce
- Cassandra

- Create and maintain RESTful APIs and SOAP Web Services for enterprise data integration.
- Develop cloud-native solutions leveraging AWS services including:

- EMR
- EC2
- ECS
- S3
- Airflow
- AWS Step Functions
- AWS API Gateway

- Perform performance tuning and optimization of Spark, Hadoop, EMR, Java, and Python-based applications.
- Develop Linux-based automation scripts using Shell Scripting and Python.
- Collaborate with cross-functional teams including Data Engineering, Architecture, DevOps, and Product teams.
- Implement and maintain CI/CD pipelines to automate application deployment and infrastructure provisioning.
- Participate in code reviews, troubleshoot production issues, and ensure adherence to engineering best practices.

Required Qualifications Experience
- 7+ years of experience in software development, data engineering, and Big Data application development.
- Proven experience building scalable data platforms and distributed processing systems on AWS Cloud.

Technical Skills Programming Languages
- Strong hands-on experience with:
- Scala
- PySpark
- Java
- Kotlin
- Python





Big Data Technologies
- Apache Spark
- Hadoop Ecosystem
- HDFS
- MapReduce
- Cassandra

API Microservices
- REST API development
- SOAP Web Services
- Microservices Architecture

AWS Cloud Services
- Amazon EMR
- EC2
- ECS
- S3
- AWS API Gateway
- AWS Step Functions
- Apache Airflow

Operating Systems Scripting
- Linux Administration
- Shell Scripting
- Python Automation

CI/CD DevOps
- Maven
- Gradle
- Jenkins
- Artifactory
- AWS CodeCommit
- AWS CloudFormation
- CI/CD Pipeline Development and Automation

Preferred Qualifications
- Experience with Snowflake for data warehousing and analytics.
- Experience developing automated data ingestion and transformation pipelines using Spark and Snowflake.
- Familiarity with DevOps and source control tools including:
- Git
- GitHub
- GitLab
- Bitbucket
- IntelliJ IDEA
- PyCharm

- Experience working within Agile/Scrum development environments.

We are seeking an experienced Senior Big Data Engineer with 7+ years of expertise in designing, developing, and supporting large-scale data processing applications on AWS Cloud. The ideal candidate will have robust hands-on experience with Big Data technologies, distributed computing frameworks, cloud-native microservices, batch processing, and CI/CD automation. This role will be responsible for building scalable data platforms, optimizing high-volume batch workloads, and developing API-driven data solutions that support critical business initiatives. Key Responsibilities

- Design, develop, implement, and maintain scalable Big Data applications and data processing pipelines on AWS Cloud.
- Build and support both API-based microservices and batch processing solutions using Java, Scala, Kotlin, and Python.
- Develop and optimize large-scale distributed data processing applications utilizing:
- Apache Spark
- Hadoop Ecosystem
- HDFS
- MapReduce
- Cassandra





- Create and maintain RESTful APIs and SOAP Web Services for enterprise data integration.
- Develop cloud-native solutions leveraging AWS services including:

- EMR
- EC2
- ECS
- S3
- Airflow
- AWS Step Functions
- AWS API Gateway

- Perform performance tuning and optimization of Spark, Hadoop, EMR, Java, and Python-based applications.
- Develop Linux-based automation scripts using Shell Scripting and Python.
- Collaborate with cross-functional teams including Data Engineering, Architecture, DevOps, and Product teams.
- Implement and maintain CI/CD pipelines to automate application deployment and infrastructure provisioning.
- Participate in code reviews, troubleshoot production issues, and ensure adherence to engineering best practices.

Required Qualifications Experience
- 7+ years of experience in software development, data engineering, and Big Data application development.
- Proven experience building scalable data platforms and distributed processing systems on AWS Cloud.

Technical Skills
- Programming Languages

Strong hands-on experience with:
- Scala
- PySpark
- Java
- Kotlin
- Python

Big Data Technologies
- Apache Spark
- Hadoop Ecosystem
- HDFS
- MapReduce
- Cassandra

API Microservices
- REST API development
- SOAP Web Services
- Microservices Architecture

AWS Cloud Services
- Amazon EMR
- EC2
- ECS
- S3
- AWS API Gateway
- AWS Step Functions
- Apache Airflow

Operating Systems Scripting
- Linux Administration
- Shell Scripting
- Python Automation

CI/CD DevOps
- Maven
- Gradle
- Jenkins
- Artifactory
- AWS CodeCommit
- AWS CloudFormation
- CI/CD Pipeline Development and Automation

Preferred Qualifications
- Experience with Snowflake for data warehousing and analytics.
- Experience developing automated data ingestion and transformation pipelines using Spark and Snowflake.
- Familiarity with DevOps and source control tools including:
- Git
- GitHub
- GitLab
- Bitbucket
- IntelliJ IDEA
- PyCharm

- Experience working within Agile/Scrum development environments.

",

Disclaimer: This job posting has been aggregated from external source. Role details, content, and availability are subject to change. Applicants are advised to confirm the latest information directly on the company website before applying.

📌 Senior Big Data Engineer (Hyderabad)
🏢 Sky Systems
📍 Hyderabad