Role: Mid-level Big Data Engineer

Position Type: Full-Time Contract (40hrs/week)

Contract Duration: 6 months + extendable

Work Schedule: 8 hours/day (Mon-Fri)

Location: Hybrid - Wed/Thursday to be in onsite in Hyderabad, India

About the role:

- Design, develop, test, and deploy scalable Big Data solutions on AWS Cloud.

- Build and maintain batch and streaming data pipelines using Scala, Python, Spark, and PySpark.

- Analyze, process, and transform large volumes of structured and unstructured data.

- Develop and support distributed data processing frameworks and data ingestion platforms.

- Integrate new data sources and tools into existing big data ecosystems.

- Optimize performance of Spark, Hadoop, EMR, and distributed computing workloads.

- Design and implement cloud-native APIs and microservices supporting data products and operational platforms.

- Collaborate with product owners, architects, QA teams, and business stakeholders throughout the software development lifecycle.

- Automate data workflows, deployments, monitoring, and operational processes.

- Evaluate emerging Big Data, AI, and cloud technologies and prototype innovative solutions.

- Ensure adherence to security, governance, reliability, and software engineering best practices.

Required Skills:

- Bachelor's degree in Computer Science, Computer Engineering, Information Technology, or equivalent experience.

- 5-8 years of experience in software engineering and Big Data application development.

- Strong hands-on experience with: Scala, Python, Apache Spark / PySpark, Hadoop ecosystem (HDFS, MapReduce, Hive),



Kafka or similar event-streaming technologies

- Strong AWS experience with services such as: EMR, S3, EC2, ECS/EKS, Airflow (MWAA preferred), Step Functions, API Gateway, Lambda, DynamoDB and/or Aurora/RDS

- Experience building and supporting large-scale batch and API-based data processing solutions in AWS.

- Strong Linux and shell scripting skills.

- Experience with data ingestion, transformation, data modeling, schema design, and data lifecycle management.

- Strong understanding of performance tuning for Spark, Hadoop, EMR, and distributed systems.

- Experience with Agile/Scrum development methodologies.

- Understanding of secure software development practices and cloud security fundamentals.

- Robust analytical, troubleshooting, and problem-solving skills.

- Excellent written and verbal communication skills.

Preferred Skills

- Experience with Generative AI, AI-assisted development tools (GitHub Copilot, Amazon Q, Cursor, etc.), and AI-driven automation.

- Understanding of Large Language Models (LLMs), Retrieval-Augmented Generation (RAG), Vector Databases, and AI/ML solution integration.

- Experience building AI-enabled data platforms and data pipelines supporting ML workloads.

- Experience with DevOps and CI/CD tools such as Git, GitHub, GitLab, Bitbucket, Jenkins, Maven, Gradle, and Artifactory.

- Experience with containerization technologies such as Docker and Kubernetes.

- Knowledge of automated testing frameworks and data quality validation tools.

- Experience working in enterprise-scale cloud environments.

- AWS certification(s) and/or Databricks certification(s) are a plus.

📌 Mid-level Big Data Engineer (Hyderabad)
🏢 SkySys
📍 Hyderabad