Role & responsibilities

Responsibilities:

- Design, develop, and maintain data pipelines and ETL processes for GenAI projects.
- Collaborate with data scientists and software engineers to implement machine learning models and algorithms.
- Optimize data infrastructure and storage solutions to ensure efficient and scalable data processing.
- Implement event-driven architectures to enable real-time data processing and analysis.
- Utilize containerization technologies like Kubernetes and Docker for productive deployment and scalability.
- Develop and maintain data lakes for storing and managing large volumes of structured and unstructured data.
- Implement and integrate LLM frameworks (Langchain, Semantic Kernel) for advanced language processing and analysis.
- Collaborate with cross-functional teams to design and implement solution architectures for GenAI projects.
- Utilize cloud computing platforms such as Azure or AWS for data processing, storage, and deployment.
- Monitor and troubleshoot data pipelines and systems to ensure smooth and uninterrupted data flow.
- Stay up-to-date with the latest advancements in GenAI technologies and recommend innovative solutions to enhance data engineering processes.
- Collaborate with cross-functional teams to understand business requirements and translate them into technical solutions.
- Document data engineering processes, methodologies, and best practices.
- Maintain solution architecture certificates and stay current with industry best practices.

Requirements:

- Python Proficiency: Minimum 3 years of hands-on experience building applications with Python.




- Scalable System Design: Solid understanding of designing and architecting scalable Python applications, particularly for Gen AI use cases, with a strong understanding of various components and systems architecture patterns to make cohesive and decoupled, scalable applications.
- Web Frameworks: Familiarity with Python web frameworks (Flask, FastAPI) for building web applications around AI models.
- Modular Design & Security: Demonstrated ability to design applications with modularity, reusability, and security best practices in mind (session management, vulnerability prevention, etc.,).
- Cloud-Native Development: Familiarity with cloud-native development patterns and tools (e.g., REST APIs, microservices, serverless functions).
- Cloud Deployments: Experience deploying and managing containerized applications on Azure/AWS (Azure Kubernetes Service, Azure Container Instances, or similar).
- Version Control (Git): Strong proficiency in Git for effective code collaboration and management.
- CI/CD: Knowledge of continuous integration and deployment (CI/CD) practices on cloud platforms.
- 3-5 years of relevant technical/technology experience, with a focus on GenAI projects.
- Strong programming skills in Python.
- Experience with data processing frameworks like Apache Spark or similar.
- Proficiency in SQL and database management systems.

Preferred Skills:

- Gen AI Frameworks: Experience with LLM frameworks or tools for interacting with LLMs such as LangChain, Semantic Kernel, LlamaIndex
- Data Pipelines: Experience in setting up data pipelines for model training and real-time inference.

📌 GenAI Data Engineer with GenAI projects expertise, Python, Azure/AWS (Bengaluru)
🏢 PwC
📍 Bengaluru