To lead the design, development, and implementation of scalable data solutions using Databricks on cloud platforms. The role involves architecting data pipelines, optimizing performance, and ensuring data governance and security across enterprise-level analytics initiatives.
Key Result Areas and Activities :
- Architecture Design : Develop and maintain scalable, secure, and high-performance data architecture using Databricks.
- Data Pipeline Development : Lead the creation and optimization of ETL/ELT pipelines for structured and unstructured data.
- Cloud Integration : Ensure seamless integration of Databricks with cloud platforms like Azure and AWS.
- Performance Optimization : Monitor and enhance the performance of data workflows and Spark jobs.
- Stakeholder Collaboration : Work closely with data engineers, analysts, and business stakeholders to translate requirements into technical solutions.
Must Have Skills :
- Strong expertise in Apache Spark and Databricks platform.
- Hands-on experience with Python or Scala for data engineering tasks.
- Positive expertise in Gen AI/LLM automation/modernization projects.
- Proficiency in SQL for data manipulation and querying.
- Hands-on experience and awareness about latest features in Databricks like AI/BI Genie, Agentbricks, and Databricks Apps.
- Experience with Delta Lake, Lakehouse architecture, and data versioning.
- Knowledge of CI/CD pipelines and DevOps practices in data engineering.
- Familiarity with cloud platforms (Azure/AWS) and their data services.
- Understanding of data governance, security, and compliance frameworks.
- Contribute to presales and RFPs.
Good To Have Skills :
- Experience with MLflow and machine learning model deployment.
- Knowledge of Apache Airflow or other orchestration tools.
- Exposure to real-time data processing using Kafka or similar tools.
- Familiarity with Power BI, Tableau, or other BI tools.
- Certification in Databricks or relevant cloud technologies.