: Senior Python ETL Developer – Data Engineering & Database Migration

Experience: 10+ Years

Role: Senior Python ETL Developer

Technical Focus: Python, Pandas, DataFrames, ETL/ELT, SQL, Database Migration, DuckDB, and Google Cloud Platform (GCP)

1. Job Summary

We are seeking an experienced Senior Python ETL Developer with 10+ years of overall IT experience and robust hands-on expertise in Python, Pandas DataFrames, ETL pipeline development, and relational databases. The candidate will be responsible for designing, developing, and implementing reliable data extraction, transformation, and loading (ETL/ELT) pipelines to migrate data from legacy Microsoft SQL Server databases to PostgreSQL on Google Cloud Platform (GCP).

The ideal candidate will have extensive experience in data processing, data quality validation, database migration, and performance optimization. Familiarity with DuckDB for local analytical processing, SQL-based transformations, and working with CSV or Parquet files is an added advantage but is not mandatory.

1. Key Responsibilities

- Design, develop, and maintain end-to-end ETL/ELT pipelines using Python and Pandas.
- Build reusable Python components for data extraction, transformation, validation, and loading.
- Use Pandas DataFrames and Series for data cleansing, filtering, aggregation, joins, merging, reshaping, and complex business transformations.
- Develop efficient data-processing workflows for large datasets using vectorized operations, chunking, and memory optimization.
- Extract data from Microsoft SQL Server and migrate transformed data to PostgreSQL.
- Implement bulk-loading strategies using PostgreSQL COPY and appropriate database drivers.
- Develop file-based processing workflows using CSV, JSON, and other structured data formats.
- Implement source-to-target mappings, schema transformations, data type conversions, and data quality checks.
- Build reconciliation mechanisms covering row counts, duplicate records, null values, aggregate comparisons, and business-rule validation.
- Develop restartable and idempotent ETL pipelines with batch processing, logging, error handling, and recovery mechanisms.
- Optimize Python code, Pandas operations, SQL queries, and database interactions for performance and scalability.
- Integrate migration workflows with GCP services, including Cloud SQL for PostgreSQL,



Google Cloud Storage, Secret Manager, and Cloud Logging.
- Write automated tests, maintain technical documentation, and participate in code reviews.
- Provide technical guidance and mentor developers on Python, ETL architecture, data processing, and database migration.

1. Required Skills and Qualifications

A. Python and Pandas
- 10+ years of overall IT experience, with substantial hands-on experience in Python development and data processing.
- Strong proficiency in Python 3.x, object-oriented programming, modular design, and exception handling.
- Extensive practical experience with Pandas DataFrames and Series.
- Hands-on experience with DataFrame operations, including merge, join, concat, groupby, pivot, melt, filtering, and aggregation.
- Strong understanding of vectorized operations, data type conversions, missing-value handling, and date/time transformations.
- Experience optimizing DataFrame operations for large datasets and managing memory efficiently.
- Familiarity with NumPy and other Python data-processing libraries.

B. ETL and Data Engineering
- Strong understanding of ETL/ELT architecture and data pipeline development.
- Experience building reusable, configurable, and maintainable ETL frameworks.
- Proficiency in data extraction, cleansing, transformation, normalization, and loading.
- Experience with full loads, incremental loads, batch processing, and data synchronization.
- Knowledge of data validation, source-to-target reconciliation, audit trails, and data quality management.
- Understanding of idempotency, transaction management, restartability, and failure recovery.

C. SQL and Databases
- Strong SQL skills, including joins, common table expressions (CTEs), aggregations, subqueries, and query optimization.
- Hands-on experience with Microsoft SQL Server and PostgreSQL.
- Experience with Python database connectivity libraries such as pyodbc, psycopg, or SQLAlchemy.
- Knowledge of database schemas, indexes, constraints,



transactions, and data types.
- Practical experience with database migration and bulk data loading.

D. Cloud and Engineering Practices
- Experience with Google Cloud SQL for PostgreSQL and Google Cloud Storage.
- Familiarity with GCP IAM, Secret Manager, and Cloud Logging.
- Experience with Docker, Git, automated testing using pytest, and CI/CD practices.
- Ability to troubleshoot data discrepancies, pipeline failures, and performance bottlenecks.

1. Good-to-Have Skills (Not Mandatory)

- Basic or working knowledge of DuckDB for analytical data processing and SQL-based transformations.
- Experience querying CSV and Parquet files directly using DuckDB.
- Familiarity with using DuckDB alongside Python and Pandas for data transformation and validation.
- Exposure to PyArrow, Apache Spark, or other data-processing technologies.
- Familiarity with Apache Airflow, Prefect, or similar workflow orchestration tools.
- Experience with Cloud Run Jobs or other GCP-based execution environments.

Note: DuckDB is a desirable skill and must not be treated as a mandatory screening requirement.

1. Expected Candidate Profile

- 10+ years of overall IT experience, with strong Python development and data engineering capabilities.
- Demonstrated hands-on expertise in Pandas and DataFrame-based data transformations.
- Proven experience designing, implementing, and troubleshooting production ETL pipelines.
- Experience migrating data from Microsoft SQL Server to PostgreSQL.
- Ability to independently drive technical design, development, testing, and deployment.
- Strong analytical, problem-solving, and communication skills.
- Ability to mentor team members and contribute to engineering best practices.

1. Core Technology Stack

Programming and Data Processing: Python 3.x, Pandas, DataFrames, NumPy, DuckDB (desirable) Data Engineering: ETL/ELT, data transformation, batch processing, data validation, reconciliation

Databases: Microsoft SQL Server, PostgreSQL, SQL, pyodbc, psycopg, SQLAlchemy

File Formats: CSV, JSON, Parquet

Cloud Platform: GCP Cloud SQL for PostgreSQL, Google Cloud Storage, Secret Manager, Cloud Logging, IAM

Engineering Tools: Docker, pytest, Git, CI/CD

Optional Technologies: PyArrow, Apache Spark, Apache Airflow, Prefect, Cloud Run Jobs

📌 Senior Python ETL Developer – Data Engineering & Database Migration (Bangalore Urban)
🏢 JRD Systems
📍 Bangalore Urban