Role & responsibilities

Design, build, and maintain scalable and robust data pipelines for ingesting, processing, and transforming large volumes of structured and unstructured data.

- Develop and optimize high performance Spark jobs using PySpark and Spark SQL within Azure Databricks.
- Implement data storage solutions using Azure Data Lake Storage (Gen2) following medallion architecture (Bronze, Silver, Gold layers) and best practices for data organization.
- Collaborate with data architects, analysts, and business stakeholders to understand data requirements and translate them into technical solutions.
- Implement data security and compliance measures, including access controls, encryption, and data masking within the Azure ecosystem.
- Perform data modeling to create effective, scalable schemas for both batch and real time analytics.
- Monitor, troubleshoot, and tune data pipelines and Databricks jobs for performance and cost effectiveness.
- Automate deployment and management of data solutions using CI/CD pipelines (Azure DevOps/GitHub Actions) and Infrastructure as Code (IaC) tools like Terraform or Bicep.
- Establish and enforce data quality checks and data governance standards across the data platform.




- Mentor junior data engineers and promote best practices in software development and data engineering.Mandatory Skills & Qualifications:
- 5+ years of professional experience in data engineering, with a proven track record of building enterprise grade data solutions.
- 3+ years of hands on experience with Microsoft Azure data services, specifically:
- Azure Databricks: Expert level proficiency in developing, tuning, and debugging Spark clusters and notebooks.
- Azure Data Lake Storage (Gen2): Deep experience in managing data lakes, including directory structure, security (RBAC & ACLs), and performance optimization.
- Medallion Architecture
- Expert programming skills in Python for data engineering tasks (e.g., Pandas, API interactions, unit testing).
- Expert level proficiency in PySpark for large scale data processing, including DataFrame API, Spark SQL, and understanding of Catalyst Optimizer.

Preferred candidate profile

7+ Years IN Azure Data Engineer, Data Bricks, Data Lake

Perks and benefits

Hybrid mode of Work,

Good Compensation

Cab Facility

📌 Azure Data Engineer (Bengaluru)
🏢 CGI
📍 Bengaluru