Exp: 15-20 Years (Databricks certification preferred)

We are seeking a Senior Data Engineer with in-depth knowledge of Databricks and Unity Catalog to serve as the subject matter expert for all things Databricks within our organization. This role requires deep expertise in Databricks, including CI/CD setup for data bricks, data lineage through Unity Catalog, and strong proficiency in ETL, SQL, and modern data engineering practices. You will be the go-to person for designing, implementing, and optimizing data solutions with Databricks.

Key Responsibilities

- Serve as the point of contact and subject matter expert for all Databricks-related activities, including architecture, development, and operational best practices.
- Should work closely with Sales team, propose data roadmap to prospects intending to migrate to cloud, create proof of concepts to showcase our expertise.
- Design, develop, and manage ETL/ELT pipelines in Databricks using Python (PySpark), integrating various data sources to support business operations.
- Leverage Unity Catalog to ensure data lineage, security, and governance are properly managed across the Databricks environment.
- Implement and maintain CI/CD pipelines for Databricks, ensuring smooth deployments, version control, and automation using Git and other DevOps tools.
- Build scalable data architectures, including Data Lakes, Lakehouses, and Data Warehouses, ensuring efficient data management and accessibility.
- Configure and optimize Databricks clusters, jobs, and workflows for both batch and streaming data processing to handle large-scale datasets.
- Stay up-to-date with the latest Databricks features and advancements, continuously enhancing our data engineering practices.




- Collaborate with cross-functional teams to implement data governance and ensure compliance with security and industry regulations.
- Monitor and tune Databricks workloads to ensure high performance and scalability, adapting to business needs as required.
- Provide training, guidance and mentorship to fellow cloud engineers, ensuring adherence to best practices and fostering a cooperative environment.

Qualifications

- 5+ years of experience in data engineering with significant expertise in Databricks and Apache Spark.
- Proficient in Unity Catalog for managing data lineage, security, and governance within the Databricks ecosystem.
- Experience of estimating and migrating legacy data warehouse workloads to Azure/Hybrid Cloud.
- Proficient in Unity Catalog for managing data lineage, security, and governance within the Databricks ecosystem.
- Experience building and optimizing ETL pipelines using tools like Azure Data Factory, Informatica, or similar.
- Strong understanding of CI/CD practices with experience in Git for version control and integration with Databricks.
- Expertise in SQL development and performance tuning for large-scale datasets.
- Knowledge of the Azure ecosystem, including data services like Azure Data Factory, Azure Data Lake and Azure Storage.
- Ability to work with both batch and streaming data processing pipelines.
- Experience with data modeling and dimensional design (e.g., star schema).
- Good understanding of data governance, compliance, and security best practices.
- Excellent communication and problem-solving skills, with the ability to manage multiple priorities.
- Ability to stay current on Databricks innovations and proactively introduce new features and capabilities to the team.

📌 Tech Director – Data Engineering (Bengaluru)
🏢 Inoptra Digital
📍 Bengaluru