Role & responsibilities

- Design end-to-end Lakehouse architectures on Databricks, including ingestion, transformation, storage, and consumption layers.
- Define best practices for Delta Lake, Unity Catalog, and workspace governance across environments.
- Architect scalable data pipelines using Spark, Delta Live Tables, and Databricks Workflows.
- Establish security, access control, and data governance standards within Databricks and the underlying cloud platform (Azure/AWS/GCP). • Guide performance tuning and cost optimization of Spark jobs and cluster configurations.
- Collaborate with data engineering, data science, and ML teams to enable a unified data and AI platform.
- Define CI/CD and DevOps practices for Databricks notebooks, jobs, and infrastructure (Repos, Terraform).
- Provide architectural oversight and mentorship to data engineering teams.
- Partner with stakeholders to align the Databricks platform roadmap with business and analytics needs.

Required Skills & Qualifications
- 10-14 years of overall data engineering/architecture experience, with at least 45 years hands-on with Databricks.
- Deep expertise in Apache Spark, Delta Lake,



and the Databricks Lakehouse architecture.
- Strong experience with Unity Catalog, workspace administration, and cluster/job management.
- Hands-on experience with at least one hyperscaler (Azure, AWS, or GCP) underlying the Databricks deployment.
- Proficiency in Python and/or Scala, and strong SQL skills.
- Experience designing batch and streaming data pipelines at enterprise scale.
- Solid understanding of data governance, lineage, and security practices in a Lakehouse environment.
- Strong communication skills and experience leading technical teams. Preferred Qualifications (Valuable to Have)
- Databricks Certified Data Engineer Professional or Databricks Certified Solutions Architect.
- Experience with MLflow and enabling ML/AI workloads on Databricks. • Exposure to orchestration tools such as Airflow or Azure Data Factory. • Experience migrating legacy data warehouses to a Lakehouse architecture.

Education Bachelor's degree in Computer Science, Engineering, or a related field (Master's preferred).

📌 Senior Architect_Databricks (Chennai)
🏢 Tredence
📍 Chennai