21 Nov
|
MASTER MIND CONSULTANCY
|
Bengaluru
21 Nov
MASTER MIND CONSULTANCY
Bengaluru
Job Title: Data Engineer
Experience: 5-8 Years
Location: Bengaluru, Bellandur
Roles & Responsibilities
- Design, develop, and maintain scalable batch and streaming data pipelines using Azure Data Factory, Azure Databricks, Apache Spark, and Event Hubs.
- Ingest data from databases, APIs, files, SaaS applications, IoT sources, and operational systems.
- Build reusable data ingestion, transformation, validation, and processing frameworks using Python, SQL, and PySpark.
- Develop and maintain curated data products for analytics, reporting, and data science applications.
- Implement incremental loading, Change Data Capture (CDC), schema evolution, partitioning, and error-handling mechanisms.
- Build cloud data solutions using Azure Data Lake Storage, Databricks, Synapse Analytics, Azure SQL, and Microsoft Fabric.
- Apply Medallion Architecture and optimize data pipelines for performance, scalability, and cost efficiency.
- Implement data-quality checks, metadata management, data lineage, and governance standards.
- Ensure data security through access controls, encryption, secrets management, and compliance practices.
- Develop CI/CD pipelines using Azure DevOps or GitHub Actions for automated deployments.
- Monitor pipeline performance, troubleshoot failures,
and perform root-cause analysis.
- Collaborate with Data Architects, Data Scientists, Analysts, and business teams to deliver reliable data solutions.
- Maintain technical documentation, operational runbooks, and support production releases.
Requisites
- Bachelor//'s degree in Computer Science, Data Engineering, IT, Engineering, or a related field.
- 5-8 years of technology experience, including at least 3 years in Data Engineering, ETL/ELT, or cloud data platforms.
- Solid proficiency in Python, SQL, Spark/PySpark, and distributed data processing.
- Hands-on experience with Azure Data Factory, Azure Databricks, and Azure Data Lake Storage.
- Knowledge of Azure SQL, Synapse Analytics, or Microsoft Fabric.
- Experience developing batch/streaming data pipelines and data integration frameworks.
- Strong understanding of data modelling, Lakehouse architecture, and Delta Lake.
- Knowledge of data quality, metadata, data lineage, and data governance.
- Experience with Git, CI/CD, automated testing, and Azure DevOps.
- Understanding of cloud security, access management, monitoring, and production support.
- Strong analytical, troubleshooting, documentation, and communication skills.
📌 Data Engineer (Bengaluru)
🏢 MASTER MIND CONSULTANCY
📍 Bengaluru