Robust Python programming skills, including experience developing production data pipelines and integrating with APIs.
Proficiency in SQL for data loading, validation, and troubleshooting.
Hands-on experience with Snowflake and relevant Azure data and storage services.
Experience building and operating workflows with Prefect or a comparable orchestration tool, with the ability to work in Prefect.
Experience ingesting structured and semi-structured data, including formats such as CSV, JSON, and Parquet.
Understanding of batch and incremental ingestion patterns, pagination, API rate limits, and schema evolution.
Experience using Git, code reviews, and automated testing.
Robust troubleshooting, documentation, and communication skills.
Ability to work independently while coordinating across internal teams and external organizations.

Expected outcomes
Reliable, timely ingestion of source data into the raw data layer.
Maintainable pipelines with transparent monitoring, validation, and recovery procedures.
Documented integrations that can be supported and extended by other engineers.
Prompt identification and resolution of ingestion issues.

Roles & Responsibilities




Develop and maintain data ingestion pipelines using Python, Prefect, Azure, and Snowflake.
Integrate data from enterprise applications, SaaS tools, and external partners through APIs, database connections, and file transfers.
Land raw data in designated storage and Snowflake environments, preserving source fidelity and traceability.
Implement scheduled and incremental ingestion, including retries, error handling, duplicate prevention, and recovery from failed runs.
Monitor pipeline execution, troubleshoot issues, and resolve data delivery failures.
Implement ingestion checks for completeness, freshness, schema changes, and reconciliation with source systems.
Collaborate with business teams, application owners, and external partners to define source interfaces, data formats, delivery schedules, and access requirements.
Follow established standards for secure access, credential management, logging, and handling sensitive data.
Maintain pipeline documentation, source mappings, operational runbooks, and support handover materials.

📌 Data Engineer Bengaluru
🏢 InfoVision
📍 Bengaluru