Role Overview

Looking for a hands-on Data Engineer to design, develop and maintain scalable data pipelines, data-processing services and integration solutions supporting enterprise applications, analytics and AI-enabled solutions.

The candidate should possess strong Python and SQL programming skills, along with practical experience in data ingestion, ETL/ELT, API integration, relational and NoSQL databases, data modeling and data-quality engineering.

The role requires someone who can independently develop production-quality data components while collaborating with application developers, architects, AI/ML engineers, business analysts and other engineering teams.

Key Responsibilities

- Design, develop and maintain scalable data ingestion, transformation and processing pipelines.
- Develop production-quality data engineering components using Python.
- Write complex and optimized SQL queries for data extraction, transformation, validation and analysis.
- Develop and maintain ETL/ELT pipelines for structured and semi-structured data.
- Integrate data from databases, enterprise applications, REST APIs, files and other data sources.
- Develop reusable Python modules for ingestion, transformation, validation, logging and exception handling.
- Design and implement relational database schemas and data models.
- Work with SQL and NoSQL databases based on application requirements.
- Implement data-quality controls including validation, reconciliation, deduplication and completeness checks.
- Develop and consume REST APIs for data integration.
- Work with enterprise search technologies such as Elasticsearch/OpenSearch or Apache Solr where required.




- Optimize database queries and pipelines for performance and scalability.
- Implement appropriate error handling, retry, logging and monitoring mechanisms.
- Develop automated unit, integration and data-quality tests.
- Troubleshoot data, database, API and pipeline-related issues across development and production environments.
- Follow Git-based development, code-review and CI/CD practices.
- Work collaboratively with architects and engineering teams to translate requirements into technical implementations.
- Create and maintain appropriate technical documentation.

Required Technical Skills

Skill Area Expected Proficiency

Python Strong Mandatory

SQL Strong/Advanced – Mandatory

Data Engineering ETL/ELT, ingestion, transformation, validation

Skill Area Expected Proficiency

Python Libraries Pandas, NumPy, PyArrow or equivalent

Relational Databases SQL Server / PostgreSQL / MySQL

NoSQL MongoDB / document databases or equivalent

API Integration REST APIs, JSON, authentication

Data Modeling Relational modeling, schema design, dimensional concepts

Data Formats JSON, CSV, XML, Parquet

Search Technologies Elasticsearch/OpenSearch, Apache Solr

Version Control Git / GitHub / GitLab

Testing PyTest, unit testing, integration and data-quality testing

Linux Shell commands, scripting and troubleshooting

Containerization Docker





CI/CD Basic understanding of automated build/test/deployment pipelines

SQL Skills

The candidate should have practical experience with:

Complex Joins • CTEs • Subqueries • Window Functions • Aggregations • Views • Stored Procedures • Transactions • Indexes • Query Execution Plans • Query Optimization

The engineer should understand how database design and indexing decisions affect application and pipeline performance.

Data Engineering Skills

The candidate should understand an end-to-end data flow such as:

Source Systems Ingestion Validation Transformation Data Quality Storage Search/Analytics/Application Consumption

Experience should include:

- Batch data processing
- Incremental data loading
- ETL/ELT
- Schema validation
- Data cleansing
- Deduplication
- Reconciliation
- Error handling
- Retry/recovery
- Data lineage fundamentals
- Metadata handling

AI/GenAI exposure – desirable

Educational Qualification

Bachelor's or Master's degree in Computer Science, Information Technology, Engineering, Data Science or a related technical discipline, or equivalent practical experience.

Qualified Competencies

The candidate should demonstrate:

- Strong analytical and problem-solving skills
- Good programming discipline
- Ability to independently troubleshoot technical issues
- Understanding of software engineering best practices
- Ability to work within Agile development teams
- Effective communication and collaboration
- Ability to understand technical requirements and translate them into working solutions
- Willingness to learn emerging data and AI technologies

📌 Data Engineer (Bengaluru)
🏢 Net Connect
📍 Bengaluru