Design and optimize high-throughput Python ETL/ELT pipelines for large enterprise datasets.
- Implement memory-efficient processing using generators, streaming, chunk-based processing, multiprocessing, threading, and parallel processing.
- Develop production-grade REST APIs and microservices using FastAPI / Flask.
- Automate analytical workflows, report generation, data reconciliation, and scheduled processes.
- Develop comprehensive unit and integration tests using pytest/unittest.
- Build and maintain CI/CD pipelines using GitHub Actions, Azure DevOps, or Jenkins.
- Develop cloud-native data solutions using AWS, Azure, or GCP.
- Participate in architecture discussions, code reviews, and technical mentoring.
- Maintain OpenAPI/Swagger documentation, technical documentation, and operational runbooks.
- Identify and resolve Python performance bottlenecks using cProfile, memory_profiler, asyncio, multiprocessing, and other optimization techniques.
MUST-HAVE SKILLS
Python 7+ Years
Data Engineering / Analytics Engineering 3+ Years pandas | NumPy | SQLAlchemy | requests
PySpark / Dask
FastAPI / Flask
REST API / Microservices
SQL & Data Processing pytest / unittest / tox
CI/CD – GitHub Actions / Azure DevOps / Jenkins
Cloud Data Services
CLOUD EXPERIENCE
Experience with one or more:
AWS: Lambda, Glue
Azure: Functions, Data Factory (ADF)
GCP: Cloud Functions, Dataflow
PERFORMANCE & SCALABILITY
Solid experience in
- Large-scale data processing
- Memory optimization
- Streaming/chunk-based processing
- Async programming using asyncio
- Multiprocessing / multithreading
- Profiling using cProfile / memory_profiler
- API response optimization
• Distributed processing using PySpark / Dask
DOMAIN EXPERIENCE Banking / Financial Services / FinTech / Regulatory Data Engineering experience is strongly preferred.
Candidates should have experience working with large-scale enterprise or financial datasets and building reliable, production-grade data pipelines.
IDEAL PROFILE
We are looking for a hands-on Senior Python/Data Engineer who combines strong software engineering fundamentals with practical data pipeline, API, cloud, testing, and performance optimization experience.
Interested candidates can share their updated CV at:
[email protected]
📌 Senior Python Developer (Delhi)
🏢 NLB Services
📍 Delhi