Data Platform Engineer (Bengaluru)

Data Platform Engineer (Bengaluru)

19 Nov
|
Umanist NA
|
Bengaluru

19 Nov

Umanist NA

Bengaluru

# Data Platform Engineer
•

Experience:
*
* 4–10 Years (Relevant Experience)
•

Location:
*
* Bangalore
• Work Mode:*
* Hybrid
• Employment Type:*
* Full time ## Role Overview We are looking for a skilled Senior Data Engineer – Platform & Operations to manage and improve enterprise data platforms, with responsibilities spanning data engineering, pipeline development, production support and data operations. The role involves building and maintaining data pipelines, managing large-scale data processing workflows, monitoring platform performance and resolving data-related issues. You will also work closely with business stakeholders and technical teams to understand requirements and deliver reliable data solutions. The ideal candidate should have strong hands-on experience with Python, PySpark, SQL and Big Data technologies, along with an understanding of production environments, workflow orchestration, data quality and application operations. ##

Key Responsibilities
### 1. Data Engineering & Pipeline Development
• Build, maintain and enhance scalable data pipelines using Python, PySpark, SQL and Apache Spark.
• Develop data processing solutions for both batch and streaming workloads.
• Work with large datasets using Hadoop, Hive, HDFS and Spark-based technologies.
• Handle data extraction, transformation, integration and processing across multiple sources.
• Implement data models and transformation logic to support business and reporting requirements.
• Identify performance bottlenecks and optimize existing data pipelines.
• Investigate data inconsistencies, processing failures and technical defects.
• Contribute to platform enhancements, automation and data architecture improvements. ### 2. Data Platform Monitoring & Operations
• Monitor scheduled data workflows, applications, microservices and reporting activities.
• Manage data pipeline execution, scheduling and workflow dependencies using Apache Airflow.
• Investigate failed jobs, delayed data processing and application-related issues.
• Perform controlled data reloads,



job restarts and reprocessing activities whenever required.
• Monitor data availability, freshness and overall platform performance.
• Support production deployments, releases and ongoing platform maintenance.
• Ensure smooth functioning of data processing systems and minimize operational disruptions. ### 3. Production Support & Incident Management
• Manage production incidents, service requests and change activities following ITIL practices.
• Investigate recurring technical issues and perform root-cause analysis.
• Coordinate with application, infrastructure, development and database teams for issue resolution.
• Work with ITSM platforms such as ServiceNow to track and manage operational tickets.
• Maintain technical documents, troubleshooting guides and operational knowledge resources.
• Provide timely updates to stakeholders regarding production issues and their resolution. ### 4. Data Quality & Testing
• Perform data validation and quality checks across different processing stages.
• Identify missing records, incorrect values, inconsistencies and data processing errors.
• Conduct unit, integration, regression and functional testing for data solutions.
• Validate data transformations, reporting outputs and downstream data consumption.
• Develop mechanisms to improve data accuracy, reliability and monitoring.
• Ensure that data pipelines meet expected quality and performance standards. ### 5. Reporting & Business Intelligence
• Develop and maintain reports and dashboards using Power BI.
• Monitor scheduled report refreshes and troubleshoot data connectivity or refresh failures.




• Work with business users to understand reporting requirements and data needs.
• Ensure accurate and consistent data availability for business reporting. ### 6. Stakeholder & Requirement Management
• Collaborate with business users, data engineers, architects and application teams to understand technical requirements.
• Convert business expectations into technical specifications and data engineering solutions.
• Participate in requirement-gathering sessions and technical discussions.
• Evaluate the impact of new requirements on existing data pipelines and platform architecture.
• Identify opportunities to leverage available data for additional business use cases.
• Coordinate with cross-functional teams to ensure smooth implementation of requirements.
• Communicate technical challenges, dependencies and proposed solutions effectively. ## Required Technical Skills Required Technical Skills
• Strong programming skills in Python, PySpark and SQL.
• Hands-on experience with Hadoop, Hive, HDFS, Apache Spark and Spark SQL.
• Good understanding of ETL processes, data transformation, data modelling and pipeline optimization.
• Experience with batch processing and Spark Structured Streaming.
• Working knowledge of MongoDB and relational databases.
• Experience working with Parquet and Avro file formats.
• Hands-on exposure to Apache Kafka and Kafka-based data processing.
• Experience with Apache Airflow for workflow scheduling, monitoring and troubleshooting.
• Knowledge of Power BI for dashboard development and report refresh management.
• Understanding of CI/CD pipelines and deployment tools such as GitHub, Bitbucket or equivalent.
• Working knowledge of Linux / RHEL operating systems.
• Understanding of ITIL processes, incident management and ServiceNow/ITSM tools.
• Familiarity with development tools such as VS Code, Jupyter Notebook or equivalent.

Skills:
hive,apache spark,apache kafka,spark,pyspark,batch processing,mongodb,apache airflow,etl processes,python,ci/cd pipelines,hdfs,hadoop,sql

📌 Data Platform Engineer (Bengaluru)
🏢 Umanist NA
📍 Bengaluru

Reply to this offer

Impress this employer describing Your skills and abilities, fill out the form below and leave Your personal touch in the presentation letter.

Subscribe to this job alert:

Get the latest job offers by email for: data platform engineer (bengaluru) / bengaluru

Subscribe to this job alert:

Get the latest job offers by email for: data platform engineer (bengaluru) / bengaluru