Mandatory Skills: Azure Databricks, PySpark, Python, SQL, Azure Data Factory, Azure Data Lake Storage (ADLS), Spark

:

- Design, develop, and maintain scalable data engineering solutions using Azure Databricks and PySpark.

- Develop and optimize PySpark applications for large-scale data processing and transformation.

- Build and maintain data pipelines using Azure Data Factory (ADF) and Databricks.

- Work with Azure Data Lake Storage (ADLS Gen2) for storing and processing structured and unstructured data.

- Write complex SQL queries for data extraction, transformation, validation, and analysis.

- Develop ETL/ELT workflows using Databricks notebooks and Spark.

- Perform data cleansing, transformation, aggregation, and validation using PySpark.

- Optimize Spark jobs, notebooks, queries,



and data pipelines for performance and scalability.

- Implement incremental data loads and handle large-volume datasets.

- Work with different data formats such as Parquet, JSON, CSV, and Delta Lake.

- Develop and maintain Delta Lake tables and implement concepts such as partitioning and schema evolution.

- Troubleshoot production data pipelines and resolve performance and data-quality issues.

- Collaborate with data architects, analysts, developers, and business teams to understand data requirements.

- Follow coding standards, security practices, CI/CD processes, and Agile methodologies.

📌 Walk-in || Data Engineer (Hyderabad)
🏢 Hectadata India Technology Consulting
📍 Hyderabad