Key Responsibilities
- Design, build, and maintain scalable data pipelines using PySpark and Apache Spark
- Develop efficient ETL/ELT workflows for batch and near-real-time processing
- Optimize Spark jobs for performance, reliability, and cost efficiency
- Work with large structured and unstructured datasets
- Integrate data from multiple sources such as databases, APIs, files, and cloud storage
- Write reusable, modular, and maintainable PySpark code
- Troubleshoot job failures, data quality issues, and performance bottlenecks
- Collaborate with data architects, analysts, platform teams, and business stakeholders
- Implement data validation, monitoring, and logging frameworks
- Support deployment, scheduling, and orchestration of data pipelines
- Participate in design reviews, code reviews, and technical discussions




- Mentor junior engineers and contribute to team best practices

Required Skills
- Strong hands-on experience with PySpark and Apache Spark
- Deep understanding of Spark concepts such as RDDs, DataFrames, datasets, partitioning, caching, shuffling, joins, and window functions
- Strong Python programming skills
- Experience with SQL and relational databases
- Knowledge of big data concepts and distributed data processing
- Hands-on experience with ETL/ELT pipeline development
- Good understanding of performance tuning and optimization techniques in Spark
- Experience with version control tools like Git
- Familiarity with Linux/Unix environments
- Robust debugging and analytical skills

📌 Pyspark Developer (Telangana)
🏢 and let's build it together
📍 Telangana