Architect and optimize big data pipelines using Apache Spark, Delta Lake, and Databricks-native tools. Design scalable data ingestion and transformation workflows, including batch and streaming (e.g., Kafka, Spark Structured Streaming). Create integration guidelines to configure and integrate Databricks with other existing security tools relevant to data access control. Implement data security and governance using Unity Catalog, access controls, and data classification techniques. Support migration of legacy systems to Databricks on cloud platforms like Azure, AWS, or GCP. Manage cloud platform operations with a focus on FinOps support, optimizing resource utilization, cost visibility, and governance across multi-cloud environments.
Responsibilities
- Collaboration Advisory: Act as a technical advisor to data engineering and analytics teams, guiding best practices and performance tuning.
- Collaboration Advisory:
Partner with architects and business stakeholders to align Databricks solutions with enterprise goals.
- Collaboration Advisory: Lead proof-of-concept (PoC) initiatives to demonstrate Databricks capabilities for specific use cases.
- Strategic Leadership Contributions: Mentor junior engineers and promote knowledge sharing across teams.
- Strategic Leadership Contributions: Contribute to platform adoption strategies, including training, documentation, and internal evangelism.
- Strategic Leadership Contributions: Stay current with Databricks innovations and recommend enhancements to existing architectures.
Educational Requirement
- MBA
- MCA
- ME
- MSc
- MTech
- Bachelor of Engineering
- BCA
- BSc
- BTech