: Sr Cloud Data Engineer

Position Overview

We are seeking a highly skilled Cloud Data Engineer to design, develop, support, and optimize contemporary cloud-based data platforms. The ideal candidate will have strong hands-on experience with Azure, Databricks, Snowflake, Python, SQL, Apache Airflow, GitHub, and CI/CD practices.

This role requires both development and operational support capabilities, including investigating data pipeline failures, troubleshooting Airflow workflows, analyzing logs, implementing monitoring and observability solutions, and improving data quality processes. The candidate should also be comfortable leveraging AI-assisted development tools and prompt engineering techniques to improve engineering productivity and accelerate solution delivery.

________________________________________

Key Responsibilities

- Contribute to the migration of ETL workloads from on-premises platforms to Azure Cloud, Databricks, and Snowflake.
- Design, develop, and maintain scalable data pipelines and cloud-native data platform solutions using Databricks, Azure Data Factory (ADF), Self-hosted Integration Runtime (SHIR), Logic Apps, Azure Data Lake Storage Gen2 (ADLS Gen2), Blob Storage, and Snowflake.
- Review, analyze, and modernize existing on-premises ETL processes and architectures.
- Investigate, troubleshoot, and resolve production data pipeline failures, performance bottlenecks, and data processing issues.
- Monitor and support Airflow workflows including DAG execution, task failures, dependency management, and log analysis.
- Develop and enhance data observability, monitoring, alerting, and operational support capabilities across the data platform.
- Implement and maintain data quality validation, reconciliation, exception handling, and automated testing frameworks.
- Implement DevOps and CI/CD practices utilizing Git, GitHub, GitHub Actions, and automated deployment pipelines.




- Participate in code reviews and promote engineering best practices for maintainability, reliability, and scalability.
- Collaborate with cross-functional teams to ensure seamless integration, data movement, and system interoperability.
- Tune and optimize data workloads for performance, scalability, and cost efficiency across Databricks and Snowflake environments.
- Ensure data security, governance, and compliance with organizational and regulatory standards.
- Leverage AI-assisted development tools and prompt engineering techniques to accelerate code development, testing, troubleshooting, and documentation activities.

________________________________________

Required Qualifications

- Minimum of 8 years of experience as a Cloud Data Engineer (10+ years is preferred).
- Strong hands-on experience with Azure cloud data services including ADF, SHIR, Logic Apps, ADLS Gen2, and Blob Storage.
- Experience working with Databricks, Apache Spark, PySpark, and Snowflake.
- Strong Python development experience for data engineering, automation, troubleshooting, and operational support.
- Strong SQL development skills including complex queries, stored procedures, performance tuning, views, and ETL transformations.
- Hands-on experience with Apache Airflow, including workflow troubleshooting, DAG management, log analysis, and production support.
- Experience investigating failed batch processing jobs, identifying root causes, and implementing corrective actions.
- Experience implementing monitoring, observability, alerting,



and data quality solutions for enterprise data platforms.
- Strong experience with Git, GitHub, Git branching strategies, pull requests, version control best practices, and GitHub Actions.
- Familiarity with CI/CD concepts and automated deployment pipelines.
- Experience supporting automated testing, data validation, and reconciliation processes.
- Experience developing or supporting cloud migration and modernization initiatives.
- Experience working within Agile delivery methodologies.
- Experience using AI-assisted development tools such as GitHub Copilot, Databricks Assistant, ChatGPT, Claude, or similar solutions.
- Understanding of prompt engineering techniques to improve software development, troubleshooting, and automation outcomes.
- Excellent analytical, troubleshooting, and problem-solving skills.
- Strong communication and collaboration skills.
- Demonstrated ability to work independently, manage ambiguity, and deliver high-quality solutions under tight deadlines.

________________________________________

Preferred Qualifications

- Experience implementing Bronze/Silver/Gold (Medallion) architecture patterns.
- Experience with Delta Lake and Lakehouse architectures.
- Experience optimizing Databricks and Snowflake performance and cost efficiency.
- Experience implementing enterprise monitoring and observability platforms.
- Experience with data governance, metadata management, and data quality best practices.
- Experience with Snowflake data engineering and analytics workloads.
- Experience leveraging AI/ML solutions to automate data engineering workflows.
- Experience supporting healthcare payer domains including Claims, Membership, Enrollment, Provider, Clinical, and Financial data.
- Experience with additional orchestration and workflow management technologies.
- Certifications in Azure, Databricks, Snowflake, or related cloud technologies.

📌 Sr Cloud Data Engineer (Shruti) (Hyderabad)
🏢 HCL INDIA
📍 Hyderabad