21 Nov
|
Hirezy.ai
|
Mumbai
Summary:We are seeking a seasoned Senior Data Engineer to drive the development and deployment of robust data solutions that foster organizational growth and unlock new market opportunities. This pivotal role involves leading a diverse team to architect and execute cutting-edge analytical strategies that deliver tangible business value.Responsibilities:- Design and deploy scalable data pipelines utilizing various formats such as JSON, Parquet, CSV, and ORC for both batch and streaming ingestion.- Spearhead cloud migration initiatives by developing high-performance Spark pipelines that ensure efficiency and reliability.- Architect and maintain Medallion Architecture frameworks, including the implementation of Bronze, Silver, and Gold tables for structured data systems.- Optimize Spark codebases to enhance performance during cloud transitions and overall system efficiency.- Establish and enforce robust data modeling practices that adhere to strict governance standards.- Implement comprehensive data cataloging strategies using Unity Catalog to guarantee data integrity and quality.- Lead the creation and management of Delta Live Tables (DLT)
pipelines to ensure secure and tamper-resistant data operations.- Partner with clients to refine cloud migration strategies and enforce industry best practices in design and governance.Requirements:- Possess a minimum of five years of hands-on experience in data engineering with a demonstrated history of managing complex pipeline development and cloud migration projects.- Hold a Bachelor's degree or higher in Computer Science, Data Engineering, or a closely related field.- Demonstrate advanced proficiency in Spark, SQL, and Python, along with other essential data processing technologies.- Show deep expertise in optimizing on-premises Spark code for cloud environments and implementing Medallion Architecture patterns.- Maintain strong working knowledge of Databricks ecosystems, including specific experience with Delta Live Table (DLT) implementations.- Possess solid familiarity with AWS services, with additional exposure to GCP or Azure platforms being highly advantageous.- Exhibit excellent communication and collaboration abilities to effectively engage with both clients and internal cross-functional teams.- Hold relevant professional certifications in Data Engineering for AWS, GCP, or Azure as a preferred qualification. (ref:hirist.tech)
📌 Senior Data Engineer - Spark/Python (Mumbai)
🏢 Hirezy.ai
📍 Mumbai