ROLE OVERVIEW

We are looking for an experienced Senior Databricks / Machine Learning Engineer with 8+ years of experience for a part-time job support assignment.The ideal candidate must have solid hands-on experience in Azure, Azure Databricks, Apache Spark, PySpark and Machine Learning, with experience building data ingestion, processing, ML experimentation and deployment workflows.Candidates must be available for a 3-hour assignment within the 6:00 PM10:00 AM support window.

MUST-HAVE TECHNOLOGIES

Cloud:Microsoft Azure

Data Ingestion:Azure Synapse Pipelines

Storage:Azure Data Lake Storage (ADLS)

Data Processing:Azure DatabricksApache SparkPySpark

Data Formats:Delta Lake / Parquet

Machine Learning:PythonKerasMachine Learning model development

Hyperparameter Tuning:Optuna

Experiment Tracking:MLflow

Model Registry:MLflow Model Registry

Workflow Orchestration:Databricks JobsYAML Configurations

Source Control:GitHub

Artifact Management:JFrog Artifactory

Documentation:Confluence

Operational / AI Analysis:Genie AgentsFuture custom-agent ecosystem

Deployment:Plant Servers

Supporting Systems:DTRBOPH / Equipment MetadataPlant Telemetry Systems

KEY RESPONSIBILITIES

- Develop and support Azure Databricks data and ML solutions
- Build scalable data processing pipelines using Spark/PySpark
- Work with Azure Synapse Pipelines for data ingestion




- Process and manage data in ADLS using Delta/Parquet formats
- Develop ML solutions using Python and Keras
- Implement hyperparameter optimization using Optuna
- Manage ML experiments and model lifecycle using MLflow
- Maintain MLflow Model Registry
- Configure and manage Databricks Jobs and YAML-based workflows
- Use GitHub for source control and JFrog Artifactory for artifact management
- Support deployment of models/solutions to plant servers
- Work with plant telemetry, equipment metadata, DTR and BOPH systems
- Support operational analysis involving Genie agents and future custom-agent ecosystems
- Troubleshoot data, ML and deployment issues
- Maintain technical documentation in Confluence

CANDIDATE MUST HAVE

- 8+ years overall IT experience
- Strong Azure Databricks experience
- Strong Machine Learning experience
- Strong Python + PySpark + Apache Spark
- Azure Data Lake Storage experience
- Azure Synapse Pipelines experienceKeras / ML model development experienceOptuna / hyperparameter tuning experience
- MLflow + MLflow Model Registry experience
- Databricks Jobs + YAML experience
- GitHub experience
- Experience with Delta / Parquet
- Good understanding of ML lifecycle and deployment
- Experience working with production/operational systems
- Available for 3 hours daily
- Available within 6 PM10 AM support window

📌 Senior Databricks / Machine Learning Engineer (Hyderabad)
🏢 Supreme Consulting
📍 Hyderabad