ROLES RESPONSIBILITIES Position Overview We are seeking an experienced Azure Data Engineer with 6-8 years of hands-on experience in designing developing and optimizing large-scale data platforms on Microsoft Azure The ideal candidate will possess deep expertise in Azure Databricks Azure Data Factory Delta Lake and up-to-date data engineering practices This role involves building scalable data pipelines enabling advanced analytics and ensuring efficient data availability for business and AI-driven use cases Key Responsibilities Data Engineering Pipeline Development Design build and maintain end-to-end data pipelines using Azure Databricks PySpark Scala ADF and Delta Lake Implement ETL ELT workflows with robust logging monitoring orchestration and error-handling capabilities Develop high-performance data ingestion frameworks for structured semi-structured and unstructured data Data Modeling Architecture Design and maintain data models including Bronze-Silver-Gold architectures on the Lakehouse Contribute to data lake and data warehouse architecture leveraging Azure Synapse Delta Lake and Databricks Unity Catalog Optimize data storage and compute costs using best practices in partitioning caching and workload management Databricks Expertise Write scalable and optimized PySpark Scala code for transformations and batch streaming workloads Implement Delta Lake features ACID transactions time travel optimization Z-order VACUUM Work with MLflow Databricks job clusters workflows and CI CD integration Azure Platform Integration Integrate data pipelines with Azure Data Factory Azure Functions Event Hub Service Bus and other Azure services Manage environments through Infrastructure as Code IaC using Terraform ARM or Bicep Configure and maintain data security and governance using Azure AD Key Vault Purview and role-based access Performance Optimization Quality Ensure high-performance processing and troubleshoot issues related to cluster performance cost and scalability Implement data quality frameworks unit testing and observability through tools like DQ Frameworks Databricks expectations or Great Expectations Collaboration Stakeholder Interaction Work closely with data architects analysts and business stakeholders to understand requirements and deliver data solutions Participate in Agile ceremonies sprint planning and documentation Provide thought leadership by recommending modern tools best practices and optimization strategies Required Skills Qualifications Technical Skills 6-8 years of experience in data engineering with at least 3 years in Azure Databricks Proficiency in PySpark Scala SQL Delta Lake and Databricks workflows Strong experience with Azure Data Factory for orchestration and pipeline management Hands-on experience with Azure Data Lake Storage Gen2 Azure Synapse SQL Pools or Serverless Event Hub and Key Vault Solid understanding of data warehousing distributed computing and Lakehouse principles Experience building and maintaining CI CD pipelines using Azure DevOps or GitHub Actions Knowledge of cloud security encryption access management and compliance standards EXPERIENCE 6-8 Years SKILLS Primary Skill Data Engineering Sub Skill s Data Engineering Additional Skill s AI ML Architecture Data Engineering databricks Azure Data Factory GenAI Fundamentals

📌 Azure Data Engineer (senior) (Maharashtra)
🏢 Infogain
📍 Maharashtra