– Data Engineer

Required Skills & Technical Expertis

SQL – High

- Strong expertise in writing, debugging, and optimizing complex SQL queries.

- Ability to handle large datasets and perform efficient data analysis.

Python / PySpark – High

- Hands-on experience in Python and PySpark for data processing and transformation.

- Experience in developing and maintaining scalable data pipelines.

Spark on YARN – High

- Solid understanding of Spark job execution in YARN environments.

- Experience in job monitoring, performance optimization, and troubleshooting.

Hive / Data Modelling – High

- Experience working with Hive tables, data structures, and data warehouse modelling.

- Understanding of data organization and storage concepts.

Scheduling & Orchestration – Medium

- Familiarity with Piper or similar scheduling and orchestration tools.

- Understanding of automated job scheduling and workflow management.

Data Governance & Security – Medium

- Understanding of PII handling, data privacy, and data deletion workflows.

- Awareness of data governance and security practices.

Data Quality & Validation – Medium

- Experience implementing data validation and quality checks.

- Ability to identify and resolve data discrepancies.

File-Based Ingestion – Medium

- Experience with SFTP, batch CSV ingestion,



and file-processing workflows.

- Understanding of file-based data ingestion processes.

Data Infrastructure & Architecture – Medium

- Understanding of data pipeline architecture and infrastructure components.

- Knowledge of scalable data processing environments.

Key Responsibilities

- Design, develop, and maintain scalable data pipelines using Python and PySpark.

- Write and optimize SQL queries for efficient data processing.

- Develop and manage Spark jobs in YARN environments.

- Work with Hive tables and data warehouse models.

- Implement scheduling, orchestration, and monitoring of data workflows.

- Ensure data quality, validation, governance, and security.

- Manage file-based data ingestion through SFTP and batch processing.

- Troubleshoot pipeline issues and ensure reliable data delivery.

Indium is an AI services company that helps enterprises build, scale, and innovate with cutting-edge technology. Backed by the prominent private equity firm EQT, Indium has spent 26 years providing excellence in digital engineering for Fortune 500 firms. With an "AI-first by design" approach, Indium specializes in Agentic AI, Data & Analytics, Application Engineering, and Quality Engineering.

📌 Senior Data Engineer (Hyderabad)
🏢 Indium
📍 Hyderabad