Role Summary

We are looking for an experienced PySpark Developer with robust hands-on expertise in big data processing, distributed computing, and data engineering. The ideal candidate will have deep experience building scalable data pipelines, transforming large datasets, and working with Spark-based ecosystems in production environments.

Key Responsibilities
Design, build, and maintain scalable data pipelines using PySpark and Apache Spark
Develop productive ETL/ELT workflows for batch and near-real-time processing
Optimize Spark jobs for performance, reliability, and cost efficiency
Work with large structured and unstructured datasets
Integrate data from multiple sources such as databases, APIs, files, and cloud storage
Write reusable, modular, and maintainable PySpark code
Troubleshoot job failures, data quality issues, and performance bottlenecks
Collaborate with data architects, analysts, platform teams, and business stakeholders
Implement data validation, monitoring, and logging frameworks
Support deployment, scheduling, and orchestration of data pipelines
Participate in design reviews, code reviews,



and technical discussions
Mentor junior engineers and contribute to team best practices

Required Skills
Strong hands-on experience with PySpark and Apache Spark
Deep understanding of Spark concepts such as RDDs, DataFrames, datasets, partitioning, caching, shuffling, joins, and window functions
Solid Python programming skills
Experience with SQL and relational databases
Knowledge of big data concepts and distributed data processing
Hands-on experience with ETL/ELT pipeline development
Good understanding of performance tuning and optimization techniques in Spark
Experience with version control tools like Git
Familiarity with Linux/Unix environments
Strong debugging and analytical skills

Disclaimer: This job posting has been aggregated from external source. Role details, content, and availability are subject to change. Applicants are advised to confirm the latest information directly on the company website before applying.

📌 Pyspark Developer / Senior Data Engineer Bengaluru
🏢 IBU Consulting
📍 Bengaluru