19 Nov
|
Umanist NA
|
Bengaluru
19 Nov
Umanist NA
Bengaluru
# Data Platform Engineer
•
Experience:
*
* 4–10 Years (Relevant Experience)
•
Location:
*
* Bangalore
• Work Mode:*
* Hybrid
• Employment Type:*
* Full time ## Role Overview We are looking for a skilled Senior Data Engineer – Platform & Operations to manage and improve enterprise data platforms, with responsibilities spanning data engineering, pipeline development, production support and data operations. The role involves building and maintaining data pipelines, managing large-scale data processing workflows, monitoring platform performance and resolving data-related issues. You will also work closely with business stakeholders and technical teams to understand requirements and deliver reliable data solutions. The ideal candidate should have strong hands-on experience with Python, PySpark, SQL and Big Data technologies, along with an understanding of production environments, workflow orchestration, data quality and application operations. ##
Key Responsibilities
### 1. Data Engineering & Pipeline Development
• Build, maintain and enhance scalable data pipelines using Python, PySpark, SQL and Apache Spark.
• Develop data processing solutions for both batch and streaming workloads.
• Work with large datasets using Hadoop, Hive, HDFS and Spark-based technologies.
• Handle data extraction, transformation, integration and processing across multiple sources.
• Implement data models and transformation logic to support business and reporting requirements.
• Identify performance bottlenecks and optimize existing data pipelines.
• Investigate data inconsistencies, processing failures and technical defects.
• Contribute to platform enhancements, automation and data architecture improvements. ### 2. Data Platform Monitoring & Operations
• Monitor scheduled data workflows, applications, microservices and reporting activities.
• Manage data pipeline execution, scheduling and workflow dependencies using Apache Airflow.
• Investigate failed jobs, delayed data processing and application-related issues.
• Perform controlled data reloads,
job restarts and reprocessing activities whenever required.
• Monitor data availability, freshness and overall platform performance.
• Support production deployments, releases and ongoing platform maintenance.
• Ensure smooth functioning of data processing systems and minimize operational disruptions. ### 3. Production Support & Incident Management
• Manage production incidents, service requests and change activities following ITIL practices.
• Investigate recurring technical issues and perform root-cause analysis.
• Coordinate with application, infrastructure, development and database teams for issue resolution.
• Work with ITSM platforms such as ServiceNow to track and manage operational tickets.
• Maintain technical documents, troubleshooting guides and operational knowledge resources.
• Provide timely updates to stakeholders regarding production issues and their resolution. ### 4. Data Quality & Testing
• Perform data validation and quality checks across different processing stages.
• Identify missing records, incorrect values, inconsistencies and data processing errors.
• Conduct unit, integration, regression and functional testing for data solutions.
• Validate data transformations, reporting outputs and downstream data consumption.
• Develop mechanisms to improve data accuracy, reliability and monitoring.
• Ensure that data pipelines meet expected quality and performance standards. ### 5. Reporting & Business Intelligence
• Develop and maintain reports and dashboards using Power BI.
• Monitor scheduled report refreshes and troubleshoot data connectivity or refresh failures.
• Work with business users to understand reporting requirements and data needs.
• Ensure accurate and consistent data availability for business reporting. ### 6. Stakeholder & Requirement Management
• Collaborate with business users, data engineers, architects and application teams to understand technical requirements.
• Convert business expectations into technical specifications and data engineering solutions.
• Participate in requirement-gathering sessions and technical discussions.
• Evaluate the impact of new requirements on existing data pipelines and platform architecture.
• Identify opportunities to leverage available data for additional business use cases.
• Coordinate with cross-functional teams to ensure smooth implementation of requirements.
• Communicate technical challenges, dependencies and proposed solutions effectively. ## Required Technical Skills Required Technical Skills
• Strong programming skills in Python, PySpark and SQL.
• Hands-on experience with Hadoop, Hive, HDFS, Apache Spark and Spark SQL.
• Good understanding of ETL processes, data transformation, data modelling and pipeline optimization.
• Experience with batch processing and Spark Structured Streaming.
• Working knowledge of MongoDB and relational databases.
• Experience working with Parquet and Avro file formats.
• Hands-on exposure to Apache Kafka and Kafka-based data processing.
• Experience with Apache Airflow for workflow scheduling, monitoring and troubleshooting.
• Knowledge of Power BI for dashboard development and report refresh management.
• Understanding of CI/CD pipelines and deployment tools such as GitHub, Bitbucket or equivalent.
• Working knowledge of Linux / RHEL operating systems.
• Understanding of ITIL processes, incident management and ServiceNow/ITSM tools.
• Familiarity with development tools such as VS Code, Jupyter Notebook or equivalent.
Skills:
hive,apache spark,apache kafka,spark,pyspark,batch processing,mongodb,apache airflow,etl processes,python,ci/cd pipelines,hdfs,hadoop,sql
📌 Data Platform Engineer (Bengaluru)
🏢 Umanist NA
📍 Bengaluru