17 Nov
|
Anaptyss
|
Noida
Job Information
- Industry Banking Financial Services
- Work Experience 5+ Years
- Job Type Full Time
- City Noida
- State/Province Uttar Pradesh
- Country India
About the Role
- We are looking for an experienced Senior Azure Data Engineer with 8+ years of hands-on experience in designing, developing, and maintaining scalable cloud-based data engineering solutions. The ideal candidate will have strong expertise in Azure Synapse Analytics, Azure Data Factory, Azure Databricks, Azure Data Lake Storage Gen2, Python, PySpark, Spark SQL, and Power BI.
- The role involves building robust ETL/ELT pipelines, implementing incremental and CDC-based data processing, optimizing distributed data workloads, and developing enterprise data models across Development, QA, and Production environments.
- The candidate should be capable of independently owning technical solutions, collaborating with architects and business stakeholders, contributing to Agile delivery, and mentoring other engineers.
Job Description
- Design, develop, and maintain scalable ETL/ELT pipelines using Azure Data Factory, Azure Databricks, Azure Synapse Analytics, Azure Storage, Azure Key Vault, Azure Functions, Logic Apps, and ADLS Gen2.
- Build data ingestion and transformation solutions using Python, PySpark, Spark SQL, and SQL for large-scale batch and distributed data processing.
- Configure and manage Azure Data Factory components, including linked services, datasets, integration runtimes, pipelines, triggers, parameters, and activities.
- Develop data solutions using Synapse SQL pools, Synapse Spark pools, Databricks clusters, notebooks, Azure Functions, and Logic Apps.
- Implement incremental data-loading frameworks using Delta Lake, MERGE operations, upserts, deduplication, and Change Data Capture (CDC).
- Design reusable and metadata-driven pipeline frameworks to support multiple data sources and business domains.
- Develop Python-based database integrations with SQL Server, Oracle, and MySQL, including stored procedure execution and dataset processing.
- Design and implement enterprise data warehouse and lakehouse solutions using dimensional modeling, Star Schema, Snowflake Schema,
and Slowly Changing Dimensions (SCD).
- Implement data-quality checks, exception handling, logging, monitoring, alerting, and operational support mechanisms.
- Optimize Spark workloads, SQL queries, pipeline execution, cluster configurations, and cloud resources for performance and cost efficiency.
- Develop and maintain Power BI semantic models, dashboards, and reports using Power Query and DAX.
- Implement CI/CD processes for promoting data engineering code across Development, QA, and Production environments using GitHub or Azure DevOps.
- Ensure source control, branching, pull requests, code reviews, testing, and deployment best practices.
- Support data governance and security through Databricks Unity Catalog, role-based access control (RBAC), and data-access policies.
- Participate in Agile/Scrum ceremonies, conduct code reviews, mentor engineers, and contribute to engineering standards.
- Collaborate with data architects, business stakeholders, BI developers, and engineering teams to translate requirements into scalable technical solutions.
- Own technical deliverables independently from solution design and development through testing, deployment, and production support.
Qualifications
Educational Qualifications
- Relevant educational qualifications in Computer Science, Information Technology, Engineering, Data Science, or a related discipline.
Experience
- 8+ years of hands-on experience in Data Engineering, with robust expertise in Azure Data Factory, Azure Databricks, Azure Synapse Analytics, Azure Data Lake Storage Gen2, and Azure Storage.
- Strong experience designing, developing, and maintaining scalable ETL/ELT pipelines and enterprise data engineering solutions.
- Proficiency in Python, PySpark, Spark SQL, and SQL, with experience using Pandas,
NumPy, and Python database connectivity libraries.
- Hands-on experience integrating Python applications with SQL Server, Oracle, and MySQL, including stored procedure execution and returned dataset processing.
- Strong understanding of Azure Data Factory components, including linked services, integration runtimes, datasets, pipelines, activities, triggers, and parameters.
- Experience working with Azure Databricks clusters, notebooks, Spark workloads, and distributed data processing.
- Experience with Azure Synapse Spark pools and SQL pools.
- Hands-on experience implementing incremental loads, Delta Lake processing, MERGE/upsert operations, deduplication, and CDC.
- Experience designing enterprise data warehouse and lakehouse solutions, including dimensional modeling, Star Schema, Snowflake Schema, fact and dimension tables, and SCD.
- Strong experience with GitHub or Azure DevOps and CI/CD-based code promotion across Development, QA, and Production environments.
- Experience developing Power BI semantic models, dashboards, and reports, with solid knowledge of Power Query and DAX.
- Experience implementing data-quality checks, logging, monitoring, exception handling, and pipeline alerting.
- Experience managing credentials and secrets securely using Azure Key Vault.
- Exposure to Databricks Unity Catalog, data lineage, RBAC, and enterprise data governance practices.
- Experience optimizing Spark workloads, SQL queries, data pipelines, cluster configurations, and cloud resource utilization.
- Experience working in Agile/Scrum environments, conducting code reviews, establishing engineering standards, and mentoring junior and mid-level engineers.
Skills
- Cloud Azure Technologies: Azure Data Factory, Azure Databricks, Azure Synapse Analytics, Azure Data Lake Storage Gen2 (ADLS Gen2), Azure Storage, Azure Functions, Logic Apps
Disclaimer: This job posting has been aggregated from external source. Role details, content, and availability are subject to change. Applicants are advised to confirm the latest information directly on the company website before applying.
📌 Senior Azure Data Engineer (Noida)
🏢 Anaptyss
📍 Noida