15 Nov
|
Optum
|
Hyderabad
Job highlights
- 8+ years in SQL/PL-SQL and 6+ years with Azure cloud services, Databricks, Python/PySpark, and Databricks Delta Live Tables
- Design and develop scalable data pipelines, build cloud-native solutions, operationalize ML models, ensure data quality and security, optimize performance, and provide production support
Primary Responsibilities
- Design and develop scalable data pipelines using Databricks, Apache Spark, and Python on Azure
- Build cloud-native solutions leveraging Azure Data Lake, Azure Data Factory, and Delta Lake
- Collaborate with Data Science and AI teams to operationalize ML models and embed them into production workflows
- Develop and maintain feature stores, model input pipelines, and real-time/streaming frameworks
- Ensure data quality, governance, and security across the full data lifecycle
- Build reusable frameworks, accelerators, and automation scripts to improve engineering efficiency
- Optimize performance, scalability, and reliability of data workflows and batch/streaming pipelines
- Participate in Agile development processes, including sprint planning, code reviews, and CI/CD pipelines
- Provide production support and on-call coverage, ensuring system stability and rapid issue resolution
- Comply with all applicable Company policies, procedures, and business directives, changes including those relating to work location, team assignments, work schedules, and versatile work arrangements
Required Qualifications
- Bachelor's or Master's degree
- 8+ years of solid experience in SQL / PL-SQL
- 6+ years of hands-on experience with Azure cloud services (ADF, ADLS, Databricks)
- 6+ years of experience in Data Engineering with Python/PySpark
- 6+ years of experience in building ETL/ELT pipelines using Databricks
- 6+ years of experience working in Agile environments
- 2+ years of experience with Databricks Delta Live Tables (DLT)
- Experience with cloud migration (on-prem Databricks/Azure)
- Experience with unit testing, validation, and pipeline testing frameworks
- Familiarity with medallion architecture and SCD2 implementations
- Solid understanding of secure coding practices and vulnerability remediation
- Exposure to GenAI tools (e.g., GitHub Copilot) for engineering productivity
- Proven ability to analyze logs, troubleshoot production issues, and optimize performance
- Demonstrated capability to design and deploy AI-powered solutions responsibly
📌 Data Engineering Consultant - Azure Databricks (Hyderabad)
🏢 Optum
📍 Hyderabad