19 Nov
|
Concentrix
|
Bengaluru
19 Nov
Concentrix
Bengaluru
Job Overview
We are seeking an experienced and results-driven Data Engineer with 4–6 years of relevant experience in designing, developing, and optimizing scalable data solutions using Azure Databricks and related technologies.
The ideal candidate should have strong hands-on expertise in Azure Databricks, PySpark, Apache Spark, SQL, Python, ETL/ELT development, Delta Lake, Azure cloud services, CI/CD, and performance optimization. The role requires practical experience in building reliable, secure, and production-grade data pipelines.
Key Responsibilities
- Design, develop, and maintain scalable ETL/ELT data pipelines using Azure Databricks, PySpark, Python, and SQL.
- Build and optimize batch and large-scale data-processing solutions using Apache Spark.
- Develop data ingestion and transformation workflows for structured and semi-structured data.
- Implement reliable data-processing frameworks using Delta Lake capabilities.
- Perform data cleansing, validation, reconciliation, and quality checks to ensure accuracy and consistency.
- Optimize Spark jobs, Databricks notebooks, SQL queries, clusters, and data pipelines for performance and cost efficiency.
- Troubleshoot pipeline failures, data-quality issues, production incidents, and performance bottlenecks.
- Implement incremental data processing, partitioning, caching, join optimization, and file-management techniques.
- Develop reusable and configurable data-engineering components.
- Implement version control and automated deployment processes using CI/CD and DevOps practices.
- Collaborate with data architects, analysts, application teams, and business stakeholders to translate requirements into technical solutions.
- Maintain technical documentation covering pipeline architecture, transformation logic, dependencies, and operational procedures.
- Ensure solutions comply with organizational standards for security, governance, monitoring, and data protection.
Required Technical Skills
- 4–6 years of relevant experience in data engineering and cloud-based data platforms.
- Strong hands-on experience with Azure Databricks, PySpark, Apache Spark, Python, and advanced SQL.
- Proven expertise in developing scalable ETL/ELT pipelines for batch and large-scale data processing.
- Solid understanding of Delta Lake, including Delta tables, ACID transactions, schema evolution, merge/upsert operations, and table optimization.
- Experience with Azure data services such as Azure Data Lake Storage and Azure Data Factory.
- Solid knowledge of Spark and Databricks performance optimization, including partitioning, caching, join strategies, shuffle optimization, file sizing, and cluster configuration.
- Experience with Git-based version control, CI/CD pipelines, and DevOps practices for automated deployment across environments.
- Good understanding of data quality, security, monitoring, troubleshooting, and production support.
📌 Data Engineer (Bengaluru)
🏢 Concentrix
📍 Bengaluru