Total Experience Required : 7+ years Mode of Work: Work from Office
We are looking for a Data Engineer to own and extend the Databricks Lakehouse that powers every Bay6 AI product – Scout, Rebound, Guide, Pathways and Verify. You will work at the center of our data architecture, ingesting client CRM/SIS data (Slate, Banner, EAB Navigate, Stellic, Salesforce, etc.) You will typically build out a medallion pipeline (bronze/silver/gold) and maintain a multi-tenant Unity Catalog isolation framework across a growing client portfolio. You will partner closely with the engineering team and the AI solutions team to ensure that every agent has clean, governed and well modeled data to work on.

Bay6 AI operates as a headless solution with no native UI. All agent output write into client CRMs. Data accuracy and traceability aren't trivial concerns as they show up directly in a client's production CRM. You will be one of the few engineers who would deeply understand the full data path from a client's SIS all the way to an agent decision, which means real ownership and visibility into product outcomes.

Databricks - Spark (PySpark or Spark SQL, Delta Lake (medallion architecture), Unity Catalog (Multi-tenant)
Python & SQL
Experience owning multi-tenant data architecture
Experience in Higher Education technology (CRM/SIS/Degree Audit platforms, etc.)
Data Security and Governance
Exposure to agentic AI systems or LLM-adjacent data pipelines.
Experience with independently standing up net-new Unity Catalog structures for new tenants from scratch.

Design,



build and maintain Databricks pipelines across bronze/silver/gold layers for multiple higher education clients unifying data across the institution, each within its own Unity Catalog.
Create and implement strategies for moving client data from their CRM/SIS platforms via API calls or into drop zones (SFTP) for ingestion into Bay6 Databricks. Examples include Slate SFTP feeds, Banner derived warehouses, EAB Navigate 360 and Stellic, all core higher education platforms, with attention to schema drift and data quality.
Implement Data governance controls including Unity Catalog lineage and audit logging. Partner with the AI Solutions team to translate agent data requirement into concrete schemas and feature tables.
Support the onboarding of new institutional clients, each with their distinct CRM/SIS stacks and data dictionaries.
Work with engineering as the primary engineering/infrastructure partner on platform level Databricks and AWS decisions.

Required Experience:
At least 3 years of building production data pipelines, with hands-on Databricks experience (Spark, Delta Lake, Unity Catalog, etc.).
Solid SQL and Python skills with familiarity working across batch ETL/ELT patterns.
Experience with multi-tenant data architecture and a deep understanding of catalog isolation reasoning.
Working knowledge of FERPA, GLBA, SOC 2 or equivalent regulated-data experience (higher education, healthcare, financial services).
Experience with AWS (our deployment model is single-tenant private AWS per client).
Bachelor's/Master's in Computer Science, AI, Engineering or related field.

📌 Senior Engineer, Data Engineering (Hyderabad)
🏢 BAY6.AI
📍 Hyderabad