This is a Permanent role with one of the valued client of VidPro.
Key Skills
- Govern data acquisition and ingestion architecture for enterprise-wide data & analytics integration.
- Establish reusable, scalable, and governed interfaces to all source systems across business domains.
- Define standards and ensure engineering quality across ingestion pipelines and upstream data flows.
- Act as escalation and design authority for ingestion, unstructured data, and advanced analytics integration points.
- Oversees ingestion and integration design across 100+ data sources.
- Supports 25+ data engineering and data science professionals with ingestion architecture and troubleshooting.
- Interfaces with source system experts, data scientists, platform architects, and functional IT stakeholders.
Main Tasks
- Define ingestion frameworks for batch, streaming, and APIs.
- Establish standards for JSON, CSV, XML, and unstructured data.
- Govern manual uploads and exception workflows.
- Collaborate with system owners to define interface specifications.
- Manage schema evolution and API reliability.
- Align ingestion architecture with platform and business requirements.
- Collaborate on ingestion for data science and ML pipelines
- Ensure robust API and event-driven integration for AI/LLM use cases.
- Support complex acquisition scenarios with scalability requirements.
- Review high-risk ingestion pipelines for performance and security.
- Support data engineers on error handling, retries, and load orchestration.
- Ensure ingestion aligns with governance and architecture.
- Maintain ingestion libraries and templates.
- Promote reuse and automation across engineering teams.
- Lead best practices in TDD, code reviews, and pipeline testing.
- Maintain architectural blueprints, templates, and best practices.
- Publish design guidelines and coding standards.
- Create re-usable architecture patterns for lakehouse environments.
- Monitor usage and implement auto-scaling policies.
- Analyze and optimize cluster configurations for cost-efficiency.
- Provide cost transparency and usage reporting to stakeholders.
Required Skills
- Degree in Computer Science or related field; certifications in Databricks or Microsoft Azure preferred.
- 6–10 years in data engineering with robust focus on enterprise-scale data & analytics integration, including ML and LLM scenarios.
- Experience building robust ingestion frameworks and reusable components.
- Track record of architectural ownership and peer enablement with diverse teams.
- Experience working in international teams across multiple time zones and cultures, preferably with teams in India, Germany, and the Philippines.
📌 Data Engineer (Bengaluru)
🏢 Continental
📍 Bengaluru
Reply to this offer
Impress this employer describing Your skills and abilities, fill out the form below and leave Your personal touch in the presentation letter.