- We are looking for an experienced Senior Databricks / Machine Learning Engineer with 8+ years of experience for a part time job support assignment.
- The ideal candidate must have strong hands-on experience in Azure, Azure Databricks, Apache Spark, PySpark and Machine Learning, with experience building data ingestion, processing, ML experimentation and deployment workflows.
- Candidates must be available for a 3-hour assignment within the 6:00 PM
- 10:00 AM support window.
- MUST-HAVE TECHNOLOGIES - Cloud: Microsoft Azure Data Ingestion: Azure Synapse Pipelines Storage: Azure Data Lake Storage (ADLS) Data Processing: Azure Databricks Apache Spark PySpark Data Formats: Delta Lake / Parquet Machine Learning: Python Keras Machine Learning model development Hyperparameter Tuning: Optuna Experiment Tracking: MLflow Model Registry: MLflow Model Registry Workflow Orchestration: Databricks Jobs YAML Configurations Source Control: GitHub Artifact Management: JFrog Artifactory Documentation: Confluence Operational / AI Analysis: Genie Agents Future custom-agent ecosystem Deployment: Plant Servers Supporting Systems: DTR BOPH / Equipment Metadata Plant Telemetry Systems
KEY RESPONSIBILITIES
- Develop and support Azure Databricks data and ML solutions.
- Build scalable data processing pipelines using Spark/PySpark.
- Work with Azure Synapse Pipelines for data ingestion.
- Process and manage data in ADLS using Delta/Parquet formats.
- Develop ML solutions using Python and Keras.
- Implement hyperparameter optimization using Optuna.
- Manage ML experiments and model lifecycle using MLflow.
- Maintain MLflow Model Registry.
- Configure and manage Databricks Jobs and YAML-based workflows.
- Use GitHub for source control and JFrog Artifactory for artifact management.
- Support deployment of models/solutions to plant servers.
- Work with plant telemetry, equipment metadata, DTR and BOPH systems.
- Support operational analysis involving Genie agents and future custom-agent ecosystems.
- Troubleshoot data, ML and deployment issues.
- Maintain technical documentation in Confluence.
CANDIDATE MUST HAVE
- 8+ years overall IT experience
- Strong Azure Databricks experience
- Strong Machine Learning experience
- Strong Python + PySpark + Apache Spark
- Azure Data Lake Storage experience
- Azure Synapse Pipelines experience
- Keras / ML model development experience
- Optuna / hyperparameter tuning experience
- MLflow + MLflow Model Registry experience
- Databricks Jobs + YAML experience
- GitHub experience
- Experience with Delta / Parquet
- Good understanding of ML lifecycle and deployment
- Experience working with production/operational systems
- Available for 3 hours daily
- Available within 6 PM-10 AM support window.