# AI Solutions Engineer
-

Location:
*
* Pune, Maharashtra
- Work Mode:*
* Work From Office (WFO)(890)
-

Experience:
*
* 9–12 Years
- Notice Period:*
* Immediate to 45 Days
- Preferred Candidates:*
* Local candidates based in Pune ## Role Overview We are seeking an experienced AI Solutions Engineer to develop, evaluate, test, monitor, and support AI agents and LLM-powered applications across their development and production lifecycle. The ideal candidate must have strong hands-on expertise in Python, LangChain, LangGraph, OpenAI, and Azure Databricks. The role involves agent orchestration, LLM evaluation, experiment lifecycle management, automated testing, observability, production monitoring, troubleshooting, and operational reliability. This is a client-facing position involving regular interaction with clients and technical teams. Excellent verbal and written communication skills are mandatory, with no compromise on communication quality. ##

Key Responsibilities
### 1. Agent Development & Orchestration
- Design, develop, integrate, and maintain AI agents using LangChain, LangGraph, and OpenAI-based technologies.
- Build Python-based services, REST APIs, and agent integration components using FastAPI and Pydantic.
- Work with agent orchestration frameworks such as OpenAI Agents SDK, Google ADK, and Pydantic-AI, where applicable.
- Support agent workflow development, integration, testing, and production operations. ### 2. LLM Evaluation & Experiment Lifecycle
- Design and implement evaluation frameworks to assess agent quality, response accuracy, reliability, consistency, and performance.
- Develop automated evaluation suites using pytest and custom pytest-based evaluations.
- Work with MLflow Evaluation, DeepEval, Ragas, and promptfoo for LLM and agent evaluation.
- Manage experiment execution, evaluation results, test iterations, and quality improvement cycles.
- Identify performance gaps and support continuous improvement of AI agent outputs. ### 3. Agent Observability & Tracing
- Implement tracing, monitoring, and observability for agent workflows and LLM applications.
- Use OpenTelemetry and OpenInference for telemetry and instrumentation.
- Work with Langfuse and LangSmith to trace agent execution, investigate failures, and analyze application behavior.




- Monitor agent performance and troubleshoot issues across application and integration layers. ### 4. Azure AI Stack
- Work with Azure OpenAI and Microsoft Foundry for AI application and agent-related workloads.
- Use Azure Monitor and Application Insights for telemetry, application monitoring, diagnostics, and troubleshooting.
- Support operational monitoring and performance analysis of Azure-hosted AI solutions. ### 5. Azure Databricks & Data Operations
- Work with Azure Databricks for data processing, SQL-based analysis, and operational workflows.
- Use Unity Catalog for data governance and access management.
- Work with Databricks Workflows/Jobs and SQL Warehouses.
- Support data-related workflows required for AI agent evaluation and operations. ### 6. CI/CD, Testing & Version Control
- Maintain source code and version control using Git and GitLab.
- Integrate automated tests and evaluation suites into CI/CD pipelines.
- Support repeatable testing, deployment validation, and release quality checks.
- Troubleshoot build, test, and deployment issues. ### 7. Production Monitoring & Reliability
- Monitor applications and services using Grafana, Prometheus, Azure Monitor, Kibana, and alerting platforms.
- Investigate production incidents, analyze logs and telemetry, and perform root-cause analysis.
- Support incident response, issue resolution, performance monitoring, and operational improvements.
- Help improve system reliability, observability, and troubleshooting efficiency. ### 8. Cloud Infrastructure, Containers & Security
- Work with Docker, Kubernetes, and Azure Kubernetes Service (AKS) for containerized workloads.
- Understand infrastructure-as-code practices using Terraform and Bicep.
- Apply security and identity management practices using Microsoft Entra ID, managed identities, OAuth2, Azure Key Vault, and role-based access control (RBAC).
- Support secure application access, credential management, and appropriate authorization controls. ### 9. Client & Stakeholder Collaboration




- Collaborate regularly with clients, engineering teams, and other stakeholders.
- Communicate technical issues, evaluation outcomes, incident updates, and resolutions clearly.
- Participate in technical discussions and troubleshooting sessions.
- Maintain skilled communication and timely coordination across teams. ## Mandatory Skills / Must-Haves The Following Skills Are Explicitly Mandatory For This Position
- Python
- LangChain
- LangGraph
- OpenAI
- Azure Databricks Candidates must also demonstrate excellent communication, analytical thinking, problem-solving, and troubleshooting capabilities appropriate for a client-facing engineering role. ## Nice-to-Have Skills ### Programming & APIs
- FastAPI
- REST APIs
- Pydantic
- pytest ### Agent Orchestration
- OpenAI Agents SDK
- Google ADK
- Pydantic-AI ### Agent Observability
- OpenTelemetry
- OpenInference
- Langfuse
- LangSmith ### LLM Evaluation & Experiment Lifecycle
- MLflow Evaluation
- DeepEval
- Ragas
- promptfoo
- Custom pytest evaluations ### Azure AI Stack
- Azure OpenAI
- Microsoft Foundry
- Azure Monitor
- Application Insights ### Databricks
- Unity Catalog
- Databricks Workflows/Jobs
- SQL Warehouses ### CI/CD & Version Control
- Git
- GitLab
- Automated testing pipelines ### Production Monitoring
- Grafana
- Prometheus
- Azure Monitor
- Kibana
- Alerting platforms ### Cloud & Containers
- Docker
- Kubernetes
- Azure Kubernetes Service (AKS)
- Terraform
- Bicep ### Security & Identity
- Microsoft Entra ID
- Managed identities
- OAuth2
- Key Vault
- Role-Based Access Control (RBAC) ## Candidate Screening Criteria
- **

Experience:
*
* 9–12 years(relevnat should be 8+yrs).
- **Employment stability:*
* Minimum 2 years in an organization.
- **Notice period:*
* Immediate to 45 days.
- **Communication:*
* Excellent verbal and written communication is mandatory; no compromise.
- **

Location:
*
* Local Pune candidates are preferred.
- **Work arrangement:*
* Work From Office is mandatory.
- **Shift flexibility:*
* willingness to work from office and in rotational shifts

Skills:
rest apis,ci/cd,communication,git,openai,sql,python,langgraph,docker,langchain,azure databricks,azure,kubernetes

📌 AI Solutions Engineer (Pune City)
🏢 Umanist NA
📍 Pune City