- Primary Skills: Azure Data bricks
- Secondary Skills: Python, PySpark
- Location – Bangalore
- Experience – 6 to 12 years
- Interview Mode - Virtual
- Shift timing – General Shift
- Notice period – Immediate Joiner or October 2nd Week LWD candidates
Data Engineer (Azure Databricks & PySpark) Job Title: Data Engineer – Azure Databricks (ADB) & PySpark
:
We are looking for a Data Engineer with strong experience in Azure Databricks (ADB) and PySpark to design, develop, and optimize large-scale data processing solutions. The candidate will be responsible for building scalable ETL/ELT pipelines, transforming data, and supporting analytics and reporting requirements.
Key Responsibilities:
- Develop and maintain data pipelines using Azure Databricks and PySpark .
- Design and implement ETL/ELT processes for data ingestion and transformation.
- Work with structured and unstructured data from multiple sources.
- Optimize Spark jobs for performance and scalability.
- Develop and maintain data models for analytics and reporting.
- Collaborate with business stakeholders and technical teams to understand data requirements.
- Ensure data quality, integrity, and governance standards are maintained.
Required Skills:
- Hands-on experience with Azure Databricks (ADB) .
- Robust proficiency in PySpark and Spark frameworks.
- Good experience in SQL and data transformation.
- Knowledge of Azure Data Lake Storage (ADLS) .
- Understanding of Data Warehousing and ETL/ELT concepts.
- Experience in performance tuning and optimization of Spark applications.
- Strong problem-solving and analytical skills.
Preferred Skills:
- Experience with Azure Data Factory (ADF) .
- Knowledge of Delta Lake , Medallion Architecture, and Data Lake concepts.
- Experience working in Banking, Finance, Insurance, or Retail domains.
- Familiarity with Git and CI/CD practices.