Role & responsibilities

- Design, develop, and maintain scalable data pipelines and ETL/ELT workflows using Python and PySpark.

- Develop efficient PySpark applications for processing large volumes of structured and semi-structured data.

- Write optimized Python scripts and modules for data processing, automation, and transformation.

- Develop complex SQL queries for data extraction, transformation, validation, and analysis.

- Perform data cleansing, transformation, aggregation, and validation across large datasets.

- Optimize PySpark jobs for performance, scalability, memory utilization, and execution time.

- Work with distributed computing concepts such as Spark architecture, DataFrames, RDDs, partitions, joins, caching, and transformations/actions.

- Build and maintain robust and reusable data engineering frameworks.

- Troubleshoot data pipeline failures and resolve performance and data-quality issues.

- Implement data validation, error handling, logging, and monitoring mechanisms.

- Collaborate with Data Scientists, Analytics teams, Software Engineers, and Business stakeholders.

- Participate in code reviews and follow coding, testing,



and documentation best practices.

- Contribute to the design and implementation of scalable cloud-based data solutions.

Preferred candidate profile

- 7+ years of overall experience in Data Engineering / Big Data / Python development.

- Strong hands-on experience with Python.

- Strong hands-on experience with PySpark / Apache Spark.

- Strong proficiency in SQL and relational databases.

- Good understanding of ETL/ELT concepts and data pipeline development.

- Experience working with large datasets and distributed data processing.

- Strong understanding of Spark DataFrames, transformations, actions, joins, partitioning, caching, and optimization techniques.

- Experience with data formats such as JSON, CSV, Parquet, and Avro.

- Positive understanding of data structures, algorithms, and object-oriented programming in Python.

- Experience with Git/version control and software development best practices.

- Strong debugging, analytical, and problem-solving skills.

📌 Senior Data Engineer (Bengaluru)
🏢 wissen technology
📍 Bengaluru