Role Overview

We are looking for an experienced Data Engineer to design, develop, and optimize scalable data ingestion and processing pipelines. The candidate will work across batch and real-time data processing, object storage, analytical data layers, and reusable data transformation frameworks. The role requires strong hands-on engineering skills and the ability to build reliable, configurable, and maintainable data solutions.

Key Responsibilities

· Design, develop, and maintain batch and real-time data ingestion pipelines.

· Build data processing workflows using Python, SQL, Apache Spark, and PySpark.

· Implement streaming pipelines using Apache Kafka, including buffering, replay, and failure recovery.

· Develop layered data architectures covering raw, cleansed, transformed, and curated datasets.

· Build reusable components for data cleansing, joins, aggregations, validation, and transformations.

· Integrate object storage with analytical table formats and processing engines.

· Implement schema discovery, schema evolution, data quality checks, and metadata management.

· Work with structured and semi-structured formats, including Parquet, CSV, and JSON.

· Develop data decoding and transformation workflows based on provided data dictionaries, format specifications, scaling rules, and engineering units.

· Optimize Spark jobs, data partitioning, incremental processing, and analytical workloads.

· Implement logging, monitoring, error handling, and recovery mechanisms for data pipelines.





· Collaborate with backend and frontend engineers to integrate data services with application APIs and user interfaces.

· Containerize services using Docker and support development and deployment in Linux-based, on-premises environments.

· Write test cases, troubleshoot pipeline failures, and maintain technical documentation.

Required Skills

· 5–6 years of hands-on experience in data engineering and pipeline development.

· Solid proficiency in Python, SQL, Apache Spark, and PySpark.

· Practical experience with Apache Kafka and real-time or near-real-time data processing.

· Experience designing batch and streaming pipelines, including retries, checkpointing, replay, and fault handling.

· Hands-on experience with object storage such as SeaweedFS, MinIO, or S3-compatible storage.

· Experience with Apache Iceberg or comparable lakehouse table formats, including schema evolution, partitioning, and metadata management.

· Strong understanding of Parquet, CSV, JSON, schema management, and data validation.

· Experience building layered data architectures and reusable transformation frameworks.

· Experience implementing joins, aggregations, incremental processing, and data quality rules.

· Experience optimizing Spark jobs and handling large datasets efficiently.

· Familiarity with PostgreSQL or equivalent relational databases.

· Experience with REST APIs, Docker, Linux, Git, and software debugging.

· Ability to develop configurable pipelines that can be extended to new datasets and processing requirements.

Pay: ₹1,010,000.00 - ₹1,880,000.00 per year

Work Location: Hybrid remote in Bangalore City, Bengaluru, Karnataka

📌 Senior Data Engineer (Bengaluru)
🏢 Software Mechanics
📍 Bengaluru