- Robust Python programming skills, including experience developing production data pipelines and integrating with APIs.
- Proficiency in SQL for data loading, validation, and troubleshooting.
- Hands-on experience with Snowflake and relevant Azure data and storage services.
- Experience building and operating workflows with Prefect or a comparable orchestration tool, with the ability to work in Prefect.
- Experience ingesting structured and semi-structured data, including formats such as CSV, JSON, and Parquet.
- Understanding of batch and incremental ingestion patterns, pagination, API rate limits, and schema evolution.
- Experience using Git, code reviews, and automated testing.
- Strong troubleshooting, documentation, and communication skills.
- Ability to work independently while coordinating across internal teams and external organizations.

Expected outcomes

- Reliable, timely ingestion of source data into the raw data layer.
- Maintainable pipelines with clear monitoring, validation, and recovery procedures.
- Documented integrations that can be supported and extended by other engineers.
- Prompt identification and resolution of ingestion issues.

Roles & Responsibilities





- Develop and maintain data ingestion pipelines using Python, Prefect, Azure, and Snowflake.
- Integrate data from enterprise applications, SaaS tools, and external partners through APIs, database connections, and file transfers.
- Land raw data in designated storage and Snowflake environments, preserving source fidelity and traceability.
- Implement scheduled and incremental ingestion, including retries, error handling, duplicate prevention, and recovery from failed runs.
- Monitor pipeline execution, troubleshoot issues, and resolve data delivery failures.
- Implement ingestion checks for completeness, freshness, schema changes, and reconciliation with source systems.
- Collaborate with business teams, application owners, and external partners to define source interfaces, data formats, delivery schedules, and access requirements.
- Follow established standards for secure access, credential management, logging, and handling sensitive data.
- Maintain pipeline documentation, source mappings, operational runbooks, and support handover materials.

📌 Data Engineer (Bengaluru)
🏢 InfoVision
📍 Bengaluru