- Architect and optimize big data pipelines using Apache Spark Delta Lake and Databricks native tools
- Design scalable data ingestion and transformation workflows including batch and streaming e
- g
- Kafka Spark Structured Streaming
- Create integration guidelines to configure and integrate Databricks with other existing security tools relevant to data access control
- Implement data security and governance using Unity Catalog access controls and data classification techniques
- Support migration of legacy systems to Databricks on cloud platforms like Azure AWS or GCP
- Manage cloud platform operations with a focus on FinOps support optimizing resource utilization cost visibility and governance across multi cloud environments
Key Responsibilities:
- Collaboration Advisory
- Act as a technical advisor to data engineering and analytics teams guiding best practices and performance tuning
- Partner with architects and business stakeholders to align Databricks solutions with enterprise goals
- Lead proof of concept PoC initiatives to demonstrate Databricks capabilities for specific use cases
- Strategic Leadership Contributions
- Mentor junior engineers and promote knowledge sharing across teams
- Contribute to platform adoption strategies including training documentation and internal evangelism
- Stay current with Databricks innovations and recommend enhancements to existing architectures