Job Title: Data Engineer – Azure Data Bricks, Azure Data Factory, Python, PySpark, SQL

Location: Hyderabad, Bengaluru, Chennai

Job Type: Full-Time

Job Overview:

We are looking for a skilled and motivated Data Engineer with expertise in Azure Data Bricks (ADB), Azure Data Factory (ADF), Python, PySpark, and SQL to join our dynamic team. As a Data Engineer, you will be responsible for designing, building, and maintaining robust data pipelines and working with large-scale datasets to enable efficient data processing, storage, and analytics.

Key Responsibilities:

- Design, develop, and optimize end-to-end data pipelines using Azure Data Factory (ADF) and Azure Data Bricks (ADB) to extract, transform, and load (ETL) data from various sources into structured and unstructured formats.

- Collaborate with Data Scientists, Analysts, and other engineers to ensure smooth data integration and data flow across multiple systems.

- Utilize Python and PySpark to process large datasets, implement complex transformations, and perform data wrangling tasks.

- Build and maintain scalable, high-performance data processing solutions on Azure.

- Write and optimize SQL queries to interact with relational databases and data warehouses (e.g., Azure SQL, SQL Server, or other cloud-based databases).

- Work with cloud storage solutions like Azure Data Lake, Azure Blob Storage, and Azure SQL Data Warehouse.

- Automate data workflows, monitor jobs, and ensure data integrity and consistency throughout the data pipeline lifecycle.

- Participate in code reviews, collaborate on best practices,



and work to continually improve the quality of the data engineering solutions.

- Troubleshoot and debug data-related issues, identify root causes, and implement fixes as needed.

Required Skills and Qualifications:

- Proven experience as a Data Engineer or in a similar data engineering role, with a focus on cloud-based data solutions (Azure preferred).

- Strong experience working with Azure Data Bricks (ADB) and Azure Data Factory (ADF) for data integration, transformation, and orchestration.

- Proficiency in Python for data processing, automation, and scripting.

- Solid understanding and hands-on experience with PySpark for big data processing and distributed data systems.

- Strong knowledge of SQL, including advanced querying, optimization, and working with relational databases (e.g., Azure SQL, SQL Server).

- Experience working with cloud storage solutions (Azure Data Lake, Blob Storage, etc.).

- Familiarity with data warehousing concepts and tools, especially in an Azure setting.

- Ability to troubleshoot and optimize data pipelines for scalability, performance, and reliability.

- Strong problem-solving skills and attention to detail.

- Excellent communication skills and ability to work collaboratively in cross-functional teams.

Preferred Qualifications:

- Experience with Azure Synapse Analytics, Azure Machine Learning, or Power BI.

- Knowledge of CI/CD practices and version control (e.g., Git).

- Familiarity with data modeling, data governance, and data security best practices.

- Certification in Microsoft Azure (e.g., Azure Data Engineer Associate) is a plus.

📌 ADB+ADF_ShivaniGautam (Bengaluru)
🏢 TalentOla
📍 Bengaluru