Job Purpose

To design and deliver AI-powered Business Intelligence solutions by integrating Databricks Genie, Semantic Models, and Metric Views on top of a robust end-to-end data engineering stack. The role focuses on enabling self-service analytics using natural language (GenAI), ensuring business-friendly data consumption, and building governed, scalable, and high-performance data ecosystems across batch and real-time pipelines.

Duties and Responsibilities

KEY ROLES / PRINCIPAL ACCOUNTABILITIES

- AI for BI GenAI Enablement
- Design and implement GenAI-powered BI solutions using Databricks Genie or MS Fabric
- Create and manage Data Rooms for business users with curated datasets
- Define and maintain Instructions Layer (prompt engineering for business context)
- Enable natural language to SQL/insights workflows for self-service analytics
- Drive adoption of Databricks One other platforms as a unified analytics interface

- Semantic Layer Metrics Engineering

- Design and manage Semantic Models for business abstraction
- Develop reusable and governed Metric Views (KPIs, aggregations, business definitions)
- Ensure consistency across BI tools (Power BI / Genie)
- Align semantic layer with business glossary and data governance policies

- Data Engineering Platform Development

- Build scalable pipelines using Azure Databricks (PySpark, SQL)
- Develop and orchestrate ETL workflows using Azure Data Factory
- Work with Delta Lake architecture (BronzeSilverGold layers)
- Enable real-time and batch data processing pipelines
- Enable data exposure via APIs for BI and downstream systems

- CI/CD DevOps

- Implement CI/CD pipelines for data and AI workflows
- Automate deployments across environments (Dev, QA, Prod)
- Ensure version control and reproducibility

Key Responsibilities

- Translate business problems into AI-driven BI solutions
- Own end-to-end delivery: ingestion transformation semantic layer AI consumption
- Design scalable data architecture aligned with lakehouse principles
- Ensure data quality,



governance, and metric consistency
- Collaborate with business teams to onboard them onto Genie-based analytics
- Optimize performance of queries, pipelines, and AI responses
- Establish best practices for AI in BI (prompting, semantic tuning, governance)
- Drive adoption of self-service BI with minimal dependency on tech teams

Key Decisions / Dimensions

- Define semantic layer design and metric definitions
- Decide GenAI prompting strategies and instruction frameworks
- Prioritize data vs AI optimization trade-offs
- Handle production issues with RCA and long-term fixes
- Drive architectural decisions for lakehouse + AI integration

Major Challenges

- Ensuring accuracy and trust in AI-generated insights
- Driving adoption of GenAI-based BI over traditional dashboards
- Maintaining semantic consistency across multiple tools
- Balancing performance, cost, and scalability
- Managing dependencies across data engineering, AI, and business teams

Required Qualifications and Experience

Must Have

- Azure Databricks PySpark, SQL, Delta Lake
- Robust experience in Semantic Modeling Metrics Layer design
- Hands-on with Databricks Genie / GenAI-based BI workflows
- Databricks One (Unified BI Experience)
- Prompt Engineering / Instruction tuning for AI systems
- Python (Pandas, PySpark, FastAPI)
- Azure Data Factory (ADF) for ETL pipelines
- Strong SQL and data modeling skills

Good to Have

- Cosmos DB / MongoDB (NoSQL concepts)
- Azure Data Explorer (KQL)

DATA STACK (MANDATORY FOR SCREENING)

- Databricks Lakehouse -- PySpark, SQL, Delta Lake
- AI for BI -- Databricks Genie, Genie Rooms, Instructions, Agents
- Semantic Layer -- Semantic Models, Metric Views
- ETL Orchestration -- Azure Data Factory
- Programming -- Python, C#/.NET
- Cloud Platform -- Azure (Preferred)
- DevOps -- CI/CD Pipelines, Git

SOFT SKILLS

- Strong analytical and problem-solving mindset
- Ability to bridge business + data + AI
- Effective stakeholder communication
- Ownership-driven and proactive approach
- Ability to work in fast-paced, evolving AI landscape

📌 Senior Data Engineer (Pune)
🏢 Bajaj Finance
📍 Pune