15 Nov
|
WebSenor®
|
India
## Job highlights
- The ideal candidate will have strong hands-on experience with Databricks,Azure Cloud data services,Apache Spark,PySpark,and Snowflake,with a proven background in migrating and modernizing on-premises ETL workloads to cloud-based data platforms. . Databricks is the primary data technology,with Snowflake as a secondary data platform
- Required Qualifications Job Summary We are seeking a highly skilled Cloud Platform Engineer / Cloud Data Engineer to join our team for a Cloud Data Modernization initiative. The ideal candidate will have strong hands-on experience with Databricks, Azure Cloud data services, Apache Spark, PySpark, and Snowflake, with a proven background in migrating and modernizing on-premises ETL workloads to cloud-based data platforms. Databricks is the primary data technology, with Snowflake as a secondary data platform. Candidates should also have experience with Azure and/or AWS cloud infrastructure and data services. The role will involve designing and implementing contemporary cloud data solutions, developing scalable data pipelines, supporting ETL migrations, implementing DevOps and CI/CD practices, and ensuring data quality, security, observability, and performance.
Key Responsibilities
- Contribute to the migration and modernization of on-premises ETL workloads to Azure Cloud, Databricks, and Snowflake.
- Design, develop, and maintain scalable data engineering solutions using Databricks, Apache Spark, PySpark, and Python.
- Develop data pipelines using Azure Data Factory (ADF), Self-hosted Integration Runtime (SHIR), Logic Apps, ADLS Gen2, Blob Storage, and Snowflake.
- Review, analyze, and document existing on-premises ETL processes and data workflows.
- Translate legacy ETL processes into scalable and maintainable cloud-native solutions.
- Develop and optimize data ingestion, transformation, integration, and processing pipelines.
- Implement DevOps practices and CI/CD pipelines using GitHub Actions.
- Follow Git branching strategies,
pull requests, code reviews, and software engineering best practices.
- Collaborate with architects, data engineers, application teams, QA, business stakeholders, and other cross-functional teams.
- Tune and optimize Databricks, Spark, PySpark, and Snowflake workloads for performance and scalability.
- Implement data quality validation, monitoring, reconciliation, and observability processes.
- Support automated data validation and reconciliation frameworks.
- Implement appropriate logging, monitoring, alerting, and operational processes for data pipelines.
- Ensure data solutions comply with organizational security, privacy, governance, and regulatory standards.
- Participate in Agile ceremonies, sprint planning, technical discussions, and delivery activities.
- Troubleshoot production issues and proactively identify performance, reliability, and data-quality problems.
- Leverage AI-assisted development tools to improve engineering productivity, code quality, testing, and documentation.
- Work independently, manage ambiguity, prioritize tasks, and deliver high-quality solutions within established timelines. Primary Skills
- Databricks
- Azure Data Engineering
- Azure Data Factory (ADF)
- Self-hosted Integration Runtime (SHIR)
- Azure Data Lake Storage Gen2 (ADLS Gen2)
- Azure Blob Storage
- Logic Apps
- Apache Spark
- PySpark
- Python
- Snowflake
- ETL / ELT Development
- Cloud Data Migration
- Data Quality Validation
- GitHub Actions
- CI/CD Secondary Skills
- Delta Lake
- Lakehouse Architecture
- Medallion Architecture
- Bronze / Silver / Gold
- Snowflake Data Engineering
- Data Modeling
- Database Design
- Airflow
- Data Governance
- Data Observability
- Performance Optimization
- AWS Cloud
- Azure Cloud
- AI/ML for Data Engineering
- AI-Assisted Development Tools
- Healthcare Payer Data Required Qualifications
- Bachelor s degree in Computer Science, Engineering, Information Technology, or a related field.
- Minimum 3+ years of experience as a Cloud Data Engineer, Cloud Platform Engineer, or similar role.
- Strong hands-on experience with Databricks in enterprise data engineering environments.
- Experience with Azure data services, particularly ADF, ADLS Gen2, Blob Storage, Logic Apps, and SHIR.
- Strong experience with Apache Spark and PySpark.
- Hands-on experience with Snowflake for data engineering or analytics workloads.
- Experience developing and supporting data migration and cloud modernization initiatives.
- Strong experience with ETL development using on-premises databases and ETL technologies.
- Strong SQL development skills, including:
- Complex queries
- Stored procedures
- Views
- Joins
- Data transformations
- Experience with Python or other scripting languages for data processing and automation.
- Hands-on experience with GitHub, including branching, pull requests, and code reviews.
- Experience implementing GitHub Actions and CI/CD pipelines.
- Experience supporting automated testing and data quality validation processes.
- Strong understanding of Agile/Scrum delivery methodologies.
- Ability to learn new technologies quickly and adapt to changing project requirements. Disclaimer: This job posting and Location has been aggregated from external source. Role details, content, and availability are subject to change. Applicants are advised to confirm the latest information directly on the company website before applying.
📌 Cloud Platform Engineer (India)
🏢 WebSenor®
📍 India