Educational Requirements

Bachelor of Engineering,BTech,BSc,BCA,MSc,MCA,MTech

Service Line

Data Analytics Unit

Responsibilities

- Develop and maintain data pipelines using PySpark
- Process and analyze large-scale datasets in distributed environments
- Design and implement ETL/ELT workflows
- Optimize Spark jobs for performance and scalability
- Work with data stored in HDFS, Hive, or cloud storage (S3, ADLS)
- Collaborate with data engineers, analysts, and business teams
- Ensure data quality, integrity, and governance
- Debug and troubleshoot data processing issues
- Automate workflows using scheduling tools (Airflow, Oozie, etc.)
- Write clean, scalable, and efficient code

Required Skills Qualifications
- Strong proficiency in Python and PySpark
- Valuable experience with Apache Spark (RDDs, DataFrames, Spark SQL)
- Knowledge of Hadoop ecosystem (HDFS, Hive)
- Experience in ETL pipeline development
- Familiarity with SQL and database concepts




- Experience with data formats (Parquet, ORC, JSON, CSV)
- Basic understanding of distributed computing concepts
- Exposure to version control tools (Git)

Additional Responsibilities Preferred Skills (Nice-to-Have)
- Experience with cloud platforms (AWS, Azure, GCP)
- Knowledge of Databricks or EMR environments
- Familiarity with workflow orchestration tools (Airflow)
- Exposure to Kafka or real-time data streaming
- Understanding of Delta Lake / Lakehouse architecture
- Experience with NoSQL databases (MongoDB, Cassandra)
- Knowledge of CI/CD pipelines and DevOps practices
- Basic understanding of machine learning workflows

Technical and Professional Requirements

Primary skills: Pyspark

Preferred Skills

Technology->Big Data - Data Processing->PySpark

📌 PySpark Developer (Karnataka)
🏢 Infosys
📍 Karnataka