Coherent Pixel Systems is Hiring!

Consultant – Databricks Architect

Location: Fully Remote
Work Mode: Remote
Experience: 10–15 Years
Shift: 3:00 PM – 11:00 PM IST

Job DescriptionWe are looking for an experienced Databricks Architect with strong expertise in designing up-to-date data platforms, data ingestion frameworks, and scalable lakehouse solutions. The ideal candidate should have deep experience with Databricks and enterprise ingestion tools, along with a good understanding of AI/ML and Generative AI capabilities within the Databricks ecosystem.

Responsibilities

- Design and implement scalable Databricks Lakehouse architectures.
- Define batch, streaming, CDC, and real-time data ingestion patterns.
- Architect data pipelines using Databricks and tools such as ADF, Kafka, Fivetran, Informatica, Airbyte, or similar platforms.
- Establish best practices for Delta Lake, Medallion Architecture, Unity Catalog, governance, security, and performance optimization.
- Provide technical leadership across data engineering and analytics teams.
- Design solutions supporting analytics, machine learning,



and Generative AI/RAG use cases.
- Evaluate Databricks AI capabilities such as Mosaic AI, MLflow, Model Serving, and Vector Search.
- Define reusable architecture standards, frameworks, and implementation guidelines.

Requirements

- 10–15 years of industry experience in Data Engineering/Data Architecture with strong hands-on Databricks expertise.
- Strong knowledge of Apache Spark, PySpark, SQL, Python, Delta Lake, and Unity Catalog.
- Extensive experience with data ingestion, ETL/ELT, streaming, APIs, CDC, and integration tools.
- Experience with Azure, AWS, or GCP data platforms.
- Strong understanding of Lakehouse, Data Lake, Data Warehouse, and Medallion Architecture concepts.
- Knowledge of AI/ML, Generative AI, RAG, Vector Search, MLflow, and Databricks Mosaic AI.
- Strong architecture, stakeholder management, and technical leadership skills.
- Databricks and cloud certifications are preferred.

Work Location: In person

📌 Senior Data Engineer (Chennai)
🏢 Coherent Pixel Systems
📍 Chennai