Looking for: Databricks Data Engineer (Databricks AI/ML)

Job Type: Contract

Location: Remote (India)

Role Summary

We are looking for an experienced Databricks Data Engineer to support multiple data engineering, analytics, and AI/ML initiatives. The resource will work closely with the onshore Data Architect and business teams to design, develop, deploy, and support scalable data solutions in Databricks. This role requires strong technical expertise, proactive communication, ownership, and the ability to collaborate effectively across multiple concurrent projects.

Required Skills

- 5+ years of experience as a Data Engineer.
- 3+ years of hands-on experience with Databricks.
- Strong SQL skills.
- Strong Python and PySpark development experience.
- Experience building scalable ETL/ELT pipelines using Databricks.
- Robust understanding of Delta Lake, Unity Catalog, and Databricks Workflows.
- Experience deploying and managing Databricks Asset Bundles (DAB) and CI/CD pipelines.
- Experience with Git and DevOps best practices.
- Experience ingesting data from APIs, SFTP, databases, and flat files (CSV, JSON, Parquet).
- Experience implementing data quality, audit logging, reconciliation, and monitoring frameworks.
- Knowledge of dimensional modeling and data warehousing concepts.
- Experience with Azure Data Factory or similar orchestration tools.

AI/ML & GenAI Experience

- Hands-on experience integrating AI/ML solutions within the Databricks ecosystem.
- Experience working with Databricks AI/ML capabilities, including MLflow and Model Serving.
- Experience integrating and deploying Databricks Genie or similar GenAI conversational analytics capabilities.
- Experience consuming and integrating LLM APIs and AI services is preferred.
- Understanding of Retrieval-Augmented Generation (RAG), vector search, embeddings, and AI application architecture is a plus.

Responsibilities

- Design, develop, and maintain scalable Databricks data pipelines.




- Build reusable ingestion and transformation frameworks using PySpark.
- Develop and optimize Spark jobs for performance and cost efficiency.
- Deploy and support Databricks Asset Bundles (DAB) across environments.
- Design and implement Delta Lake-based data solutions.
- Support AI/ML and GenAI solution development, including Databricks Genie integration and deployment.
- Integrate Databricks solutions with external applications and APIs.
- Implement data quality validations, monitoring, reconciliation, and audit processes.
- Troubleshoot production issues and provide timely resolution.
- Participate in architecture discussions, code reviews, and technical design sessions.
- Work closely with architects, product owners, and business stakeholders to deliver high-quality solutions.

Nice to Have

- Experience with Databricks Mosaic AI.
- Experience with Databricks Vector Search.
- Experience with Unity Catalog governance.
- Experience building AI agents or conversational AI solutions.
- Experience with Retail, Supply Chain, or Financial Services data platforms.
- Experience with Azure cloud services.

Soft Skills

- Excellent verbal and written communication skills.
- Strong ownership and accountability for deliverables.
- Ability to work independently with minimal supervision.
- Flexible and responsive to changing priorities.
- Strong collaboration skills with distributed teams.
- Proactive in identifying risks, blockers, and proposing solutions.
- Ability to support multiple concurrent projects while maintaining delivery commitments.

Engagement Expectations

- Availability for overlap with the US team during core collaboration hours.
- Responsive communication through Teams during working hours.
- Active participation in sprint planning, design discussions, and architecture reviews.
- Ability to adapt quickly to evolving business priorities.
- Demonstrates a collaborative mindset and takes ownership of assigned work from design through deployment and production support.

📌 Databricks Data Engineer (Databricks AI/ML) (India)
🏢 Diligente Technologies
📍 India