Job Description

- Design, develop, and maintain scalable and high-performance data pipelines.

- Build distributed data processing solutions using Apache Spark / PySpark.

- Develop data ingestion, transformation, and integration processes from multiple data sources.

- Work with AWS data engineering and cloud processing services.

- Process structured, semi-structured, and unstructured data including JSON, logs, PDFs, and other file formats.

- Develop complex and optimized SQL queries for data processing and analysis.

- Work with both relational and non-relational databases.

- Implement data validation, quality checks, and monitoring processes.

- Troubleshoot complex data issues and perform root-cause analysis.

- Optimize data pipelines, queries, and distributed processing jobs for performance and scalability.

- Collaborate with stakeholders to understand business requirements and translate them into scalable data solutions.

- Take ownership of data engineering initiatives and deliver solutions independently.

- Stay updated with emerging AWS, Big Data, and Data Engineering technologies and best practices.

Requirements
- Hands-on Python + PySpark for large-scale data processing





- Solid/advanced SQL, including complex queries and optimization

- Strong AWS / cloud data engineering experience

- Apache Spark and distributed data processing

- ETL/ELT pipeline development

- Data ingestion, transformation and integration

- Structured, semi-structured and unstructured data — JSON, logs, PDFs, etc.

- Relational and non-relational databases

- Data quality, validation, performance tuning and optimization

- Deep-dive data analysis and troubleshooting

- Scalable data solution design

Requirements
Hands-on Python + PySpark for large-scale data processing Strong/advanced SQL, including complex queries and optimization Strong AWS / cloud data engineering experience Apache Spark and distributed data processing ETL/ELT pipeline development Data ingestion, transformation and integration Structured, semi-structured and unstructured data — JSON, logs, PDFs, etc. Relational and non-relational databases Data quality, validation, performance tuning and optimization Deep-dive data analysis and troubleshooting Scalable data solution design

📌 Senior Data Engineer (India)
🏢 Mags Consulting
📍 India