Mandatory Skills: Azure Databricks, PySpark, Python, SQL, Azure Data Factory, Azure Data Lake Storage (ADLS), Spark

:
Design, develop, and maintain scalable data engineering solutions using Azure Databricks and PySpark.
Develop and optimize PySpark applications for large-scale data processing and transformation.
Build and maintain data pipelines using Azure Data Factory (ADF) and Databricks.
Work with Azure Data Lake Storage (ADLS Gen2) for storing and processing structured and unstructured data.
Write complex SQL queries for data extraction, transformation, validation, and analysis.
Develop ETL/ELT workflows using Databricks notebooks and Spark.
Perform data cleansing, transformation, aggregation, and validation using PySpark.
Optimize Spark jobs, notebooks, queries,



and data pipelines for performance and scalability.
Implement incremental data loads and handle large-volume datasets.
Work with different data formats such as Parquet, JSON, CSV, and Delta Lake.
Develop and maintain Delta Lake tables and implement concepts such as partitioning and schema evolution.
Troubleshoot production data pipelines and resolve performance and data-quality issues.
Collaborate with data architects, analysts, developers, and business teams to understand data requirements.
Follow coding standards, security practices, CI/CD processes, and Agile methodologies.

📌 Walk In Data Engineer Hyderabad (India)
🏢 Hectadata India Technology Consulting
📍 India