20 Nov
|
Indium
|
Hyderabad
Job Description – Data Engineer
Required Skills & Technical Expertis
SQL – High
Strong expertise in writing, debugging, and optimizing complex SQL queries.
Ability to handle large datasets and perform productive data analysis.
Python / PySpark – High
Hands-on experience in Python and PySpark for data processing and transformation.
Experience in developing and maintaining scalable data pipelines.
Spark on YARN – High
Strong understanding of Spark job execution in YARN environments.
Experience in job monitoring, performance optimization, and troubleshooting.
Hive / Data Modelling – High
Experience working with Hive tables, data structures, and data warehouse modelling.
Understanding of data organization and storage concepts.
Scheduling & Orchestration – Medium
Familiarity with Piper or similar scheduling and orchestration tools.
Understanding of automated job scheduling and workflow management.
Data Governance & Security – Medium
Understanding of PII handling, data privacy, and data deletion workflows.
Awareness of data governance and security practices.
Data Quality & Validation – Medium
Experience implementing data validation and quality checks.
Ability to identify and resolve data discrepancies.
File-Based Ingestion – Medium
Experience with SFTP,
batch CSV ingestion, and file-processing workflows.
Understanding of file-based data ingestion processes.
Data Infrastructure & Architecture – Medium
Understanding of data pipeline architecture and infrastructure components.
Knowledge of scalable data processing settings.
Key Responsibilities
Design, develop, and maintain scalable data pipelines using Python and PySpark.
Write and optimize SQL queries for productive data processing.
Develop and manage Spark jobs in YARN environments.
Work with Hive tables and data warehouse models.
Implement scheduling, orchestration, and monitoring of data workflows.
Ensure data quality, validation, governance, and security.
Manage file-based data ingestion through SFTP and batch processing.
Troubleshoot pipeline issues and ensure reliable data delivery.
Indium is an AI services company that helps enterprises build, scale, and innovate with cutting-edge technology. Backed by the prominent private equity firm EQT, Indium has spent 26 years providing excellence in digital engineering for Fortune 500 firms. With an "AI-first by design" approach, Indium specializes in Agentic AI, Data & Analytics, Application Engineering, and Quality Engineering.
📌 Senior Data Engineer Hyderabad
🏢 Indium
📍 Hyderabad