Role & responsibilities
Design, develop, and maintain scalable data pipelines and ETL/ELT workflows using Python and PySpark.
Develop efficient PySpark applications for processing large volumes of structured and semi-structured data.
Write optimized Python scripts and modules for data processing, automation, and transformation.
Develop complex SQL queries for data extraction, transformation, validation, and analysis.
Perform data cleansing, transformation, aggregation, and validation across large datasets.
Optimize PySpark jobs for performance, scalability, memory utilization, and execution time.
Work with distributed computing concepts such as Spark architecture, DataFrames, RDDs, partitions, joins, caching, and transformations/actions.
Build and maintain robust and reusable data engineering frameworks.
Troubleshoot data pipeline failures and resolve performance and data-quality issues.
Implement data validation, error handling, logging, and monitoring mechanisms.
Collaborate with Data Scientists, Analytics teams, Software Engineers, and Business stakeholders.
Participate in code reviews and follow coding, testing,



and documentation best practices.
Contribute to the design and implementation of scalable cloud-based data solutions.

Preferred candidate profile
7+ years of overall experience in Data Engineering / Big Data / Python development.
Strong hands-on experience with Python.
Robust hands-on experience with PySpark / Apache Spark.
Solid proficiency in SQL and relational databases.
Positive understanding of ETL/ELT concepts and data pipeline development.
Experience working with large datasets and distributed data processing.
Strong understanding of Spark DataFrames, transformations, actions, joins, partitioning, caching, and optimization techniques.
Experience with data formats such as JSON, CSV, Parquet, and Avro.
Good understanding of data structures, algorithms, and object-oriented programming in Python.
Experience with Git/version control and software development best practices.
Strong debugging, analytical, and problem-solving skills.

📌 Senior Data Engineer Bengaluru (India)
🏢 wissen technology
📍 India