Databricks Data Engineer (Azure + Databricks AI)
Location: Coimbatore, Tamil Nadu, India
Employment Type: Full Time
Experience: 5–8+ years in data engineering, with 3+ years hands-on in Databricks
Notice Period: Immediate joiners to 30 days preferred
Shift: Partial US overlap, e.g., 12:30 PM – 9:30 PM IST

About the Role
We are looking for a Databricks Data Engineer to design, build, and optimize scalable data and AI pipelines on Microsoft Azure. You will own lakehouse architecture using Delta Lake and Unity Catalog, and help bring GenAI and machine learning use cases into production using Databricks Mosaic AI capabilities. You will work closely with data scientists, analysts, and business stakeholders to turn raw data into trusted, AI-ready assets.

Key Responsibilities
• Design and build batch and streaming pipelines in Azure Databricks using PySpark, Spark SQL, and Delta Lake, following medallion (Bronze/Silver/Gold) architecture.
• Develop declarative pipelines and orchestrate workloads using Lakeflow (Delta Live Tables) and Databricks Workflows/Jobs.




• Implement data governance, lineage, and access controls using Unity Catalog.
• Ingest data from multiple sources using Azure Data Factory, Event Hubs, ADLS Gen2, Auto Loader, and REST APIs.
• Build and support AI/GenAI solutions on Databricks, including RAG pipelines with Vector Search, model deployment through Model Serving, and LLM integration via Databricks Foundation Model APIs or Azure OpenAI.
• Manage the ML lifecycle with MLflow, covering experiment tracking, model registry, evaluation, and monitoring.
• Prepare feature sets and curated datasets for ML models and AI agents, using Databricks Feature Store and Agent Framework where applicable.
• Tune Spark jobs and clusters for performance and cost, including partitioning, liquid clustering/Z-ordering, Photon, and serverless compute.
• Implement CI/CD for data and AI assets using Databricks Asset Bundles, Azure DevOps or GitHub Actions, and infrastructure as

📌 Databricks Data Engineer (Coimbatore)
🏢 OneData Software Solutions
📍 Coimbatore