Job Summary

Mission: To design, build, and maintain scalable, reliable, and secure data pipelines that enable seamless integration of diverse global data sources for Generative AI solutions. The Data Engineer ensures high-quality, timely, and accessible data for AI models by managing RAG indexes, monitoring pipeline health, and continuously onboarding new data sources. This mission supports the organisation's goal of delivering intelligent, data-driven experiences across enterprise applications.

Key Responsibilities:

- Design, implement, and maintain robust data pipelines and ETL processes to support business and product requirements.
- Monitor the performance and reliability of data pipelines, proactively addressing issues and optimizing workflows.
- Connect and integrate diverse data sources (structured and unstructured) across global systems.
- Identify and integrate new potential data sources to enhance the breadth and quality of available data.
- Build and maintain Retrieval-Augmented Generation (RAG) indexes for efficient data retrieval and model performance.
- Optimise indexing strategies for large-scale, multi-domain datasets.
- Ensure data quality, consistency, and compliance with enterprise standards.
- Implement monitoring and alerting for data pipeline health and SLA adherence.
- Work closely with AI/ML engineers, data scientists, and product teams to deliver end-to-end solutions.




- Collaborate with cross-functional teams to align data engineering efforts with project roadmaps and timelines.

Skill RequirementsQualification and Experience

- Bachelor's or Master's degree in computer science engineering with 12+ years of experience.
- Proficiency in programming languages: Python, PySpark, and SQL.
- Experience with data engineering tools, including Azure DataBricks, Azure DevOps, and VS Code.
- Strong understanding of working with REST APIs for data integration and automation.
- Hands-on experience with RAG architecture and vector databases.
- Familiarity with Generative AI concepts and LLM integration.
- Familiarity with cloud-based data platforms and scalable data architecture.

Other RequirementsPersonality requirement:

Ability to adapt to new working environments and technologies is crucial, as are your outstanding written and verbal communication skills. As a team player, you'll maintain transparent communication, providing daily updates within the team and towards stakeholders. Your customer-focused approach, coupled with an adaptable and flexible mindset, will be key in handling the varied work environment. A collaborative, positive-natured, and helpful attitude, along with a transparent mindset in all situations, sets the tone for success. Your ability to handle critical and urgent situations effectively and calmly rounds out the qualities we value.

📌 Senior Technical Manager (India)
🏢 HCL Technologies
📍 India