Big Data Engineer (Bengaluru)

Big Data Engineer (Bengaluru)

19 Nov
|
SP Software
|
Bengaluru

19 Nov

SP Software

Bengaluru

Mandatory Skills

PySpark, Scala, Apache Spark, AWS, Amazon EMR, AWS Glue, AWS Step Functions, AWS Lambda, ETL Development and Big Data Pipeline Development.

Role & responsibilities

We are looking for an experienced Senior Big Data Engineer with strong expertise in PySpark, Scala, Apache Spark and AWS cloud services. The candidate will be responsible for designing, developing, implementing and optimizing scalable big data pipelines and ETL solutions. The role requires hands-on experience with AWS services such as Amazon EMR, AWS Glue, AWS Step Functions and AWS Lambda, along with expertise in data processing, pipeline orchestration, monitoring and performance optimization.

Key Responsibilities

- Design, develop and maintain scalable big data pipelines using PySpark, Scala and Apache Spark.
- Build and manage ETL/ELT workflows for processing large volumes of structured and unstructured data.
- Develop and optimize data processing solutions using Amazon EMR and AWS Glue.
- Implement workflow orchestration and automation using AWS Step Functions and AWS Lambda.
- Monitor, troubleshoot and optimize data pipelines to ensure performance, reliability and data quality.
- Work with Databricks and Snowflake for data processing, integration and analytics.
- Leverage additional AWS services to build secure,



scalable and efficient data engineering solutions.
- Collaborate with cross-functional teams to understand data requirements and deliver reliable data solutions.
- Use Git for source code management, version control and collaborative development.
- Explore and leverage AI-related technologies and capabilities to improve data engineering workflows.

Required Technical Skills

- Strong hands-on experience in PySpark, Scala and Apache Spark.
- Extensive experience with AWS cloud services.
- Practical experience with Amazon EMR and AWS Glue.
- Experience in AWS Step Functions and AWS Lambda.
- Strong knowledge of ETL development and big data pipeline architecture.
- Experience in data pipeline monitoring, troubleshooting and performance optimization.
- Working knowledge of Git and version control.

Positive-to-Have Skills

- Experience with Databricks and Snowflake.
- Knowledge of AI / Generative AI concepts and applications.
- Exposure to additional AWS services such as S3, Redshift, Athena and CloudWatch.
- Experience with data quality, workflow automation and distributed data processing.

Preferred Skills Databricks, Snowflake, AI Knowledge, AWS Services, Data Pipeline Monitoring and Performance Optimization.

📌 Big Data Engineer (Bengaluru)
🏢 SP Software
📍 Bengaluru

Reply to this offer

Impress this employer describing Your skills and abilities, fill out the form below and leave Your personal touch in the presentation letter.

Subscribe to this job alert:

Get the latest job offers by email for: big data engineer (bengaluru) / bengaluru

Subscribe to this job alert:

Get the latest job offers by email for: big data engineer (bengaluru) / bengaluru