Role Overview

Looking for a hands-on Data Engineer to design, develop and maintain scalable data pipelines, data-processing services and integration solutions supporting enterprise applications, analytics and AI-enabled solutions.

The candidate should possess strong Python and SQL programming skills, along with practical experience in data ingestion, ETL/ELT, API integration, relational and NoSQL databases, data modeling and data-quality engineering.

The role requires someone who can independently develop production-quality data components while collaborating with application developers, architects, AI/ML engineers, business analysts and other engineering teams.

Key Responsibilities

· Design, develop and maintain scalable data ingestion, transformation and processing pipelines.

· Develop production-quality data engineering components using Python.

· Write complex and optimized SQL queries for data extraction, transformation, validation and analysis.

· Develop and maintain ETL/ELT pipelines for structured and semi-structured data.

· Integrate data from databases, enterprise applications, REST APIs, files and other data sources.

· Develop reusable Python modules for ingestion, transformation, validation, logging and exception handling.

· Design and implement relational database schemas and data models.

· Work with SQL and NoSQL databases based on application requirements.

· Implement data-quality controls including validation, reconciliation, deduplication and completeness checks.

· Develop and consume REST APIs for data integration.

· Work with enterprise search technologies such as Elasticsearch/OpenSearch or Apache Solr where required.





· Optimize database queries and pipelines for performance and scalability.

· Implement appropriate error handling, retry, logging and monitoring mechanisms.

· Develop automated unit, integration and data-quality tests.

· Troubleshoot data, database, API and pipeline-related issues across development and production environments.

· Follow Git-based development, code-review and CI/CD practices.

· Work collaboratively with architects and engineering teams to translate requirements into technical implementations.

· Create and maintain appropriate technical documentation.

Required Technical Skills

Skill Area Expected Proficiency

Python Strong – Mandatory

SQL Robust/Advanced – Mandatory

Data Engineering ETL/ELT, ingestion, transformation, validation

Skill Area Expected Proficiency

Python Libraries Pandas, NumPy, PyArrow or equivalent

Relational Databases SQL Server / PostgreSQL / MySQL

NoSQL MongoDB / document databases or equivalent

API Integration REST APIs, JSON, authentication

Data Modeling Relational modeling, schema design, dimensional concepts

Data Formats JSON, CSV, XML, Parquet

Search Technologies Elasticsearch/OpenSearch, Apache Solr

Version Control Git / GitHub / GitLab

Testing PyTest, unit testing, integration and data-quality testing

Linux Shell commands, scripting and troubleshooting

Containerization Docker





CI/CD Basic understanding of automated build/test/deployment pipelines

SQL Skills The Candidate Should Have Practical Experience With

Complex Joins

- CTEs

- Subqueries

- Window Functions

- Aggregations

- Views

- Stored Procedures

- Transactions

- Indexes

- Query Execution Plans

- Query Optimization

The engineer should understand how database design and indexing decisions affect application and pipeline performance.

Data Engineering Skills The candidate should understand an end-to-end data flow such as:

Source Systems → Ingestion → Validation → Transformation → Data Quality → Storage → Search/Analytics/Application Consumption

Experience Should Include

· Batch data processing

· Incremental data loading

· ETL/ELT

· Schema validation

· Data cleansing

· Deduplication

· Reconciliation

· Error handling

· Retry/recovery

· Data lineage fundamentals

· Metadata handling

AI/GenAI exposure – desirable

Educational Qualification

Bachelor's or Master's degree in Computer Science, Information Technology, Engineering, Data Science or a related technical discipline, or equivalent practical experience.

Professional Competencies The Candidate Should Demonstrate

· Strong analytical and problem-solving skills

· Good programming discipline

· Ability to independently troubleshoot technical issues

· Understanding of software engineering best practices

· Ability to work within Agile development teams

· Effective communication and collaboration

· Ability to understand technical requirements and translate them into working solutions

· Willingness to learn emerging data and AI technologies

📌 Data Engineer (Bengaluru)
🏢 algoleap
📍 Bengaluru