ADB + Pyspark + Kafka

- Design, develop and optimise scalable ETL/ELT pipelines using Databricks.
- Develop production-grade data transformations using Python, PySpark and SQL.
- Build Bronze/Silver/Gold data layers using Delta Lake and medallion architecture.
- Implement batch and, where required, streaming ingestion patterns.
- Work with Databricks Workflows and Unity Catalog for orchestration and governance.
- Integrate Databricks with Azure data services including ADLS Gen2 and Synapse.
- Implement data-quality checks, error handling, monitoring and performance optimisation.
- Develop automated deployments using CI/CD and follow software engineering best practices.
- Troubleshoot production data pipelines and ensure reliability and scalability.

Mandatory skills:

- Strong hands-on Databricks, Python/PySpark and SQL.
- Delta Lake and Databricks Workflows.
- ETL/ELT development and performance tuning.
- ADLS Gen2 and Azure data ecosystem.
- Unity Catalog exposure.
- Git and CI/CD experience.

2) JD :

Skills Azure Databricks+ Pyspark





Exp 6+ years

- Experience and Qualification:6–8 years of experience in data engineering, ETL development, or big data processing.
- Solid hands-on experience with Azure Databricks. Proficiency in Apache Spark and PySpark.
- Strong programming skills in Python and SQL. Experience with Azure Data Lake Storage Gen2.
- Experience developing pipelines using Azure Data Factory.
- Practical knowledge of Delta Lake and lakehouse architecture.
- Experience working with relational and NoSQL databases.
- Strong understanding of data modeling, ETL/ELT concepts, and data warehousing.
- Experience with Git-based source control and CI/CD practices.
- Knowledge of Azure services such as: Azure Key Vault Azure Synapse Analytics Azure SQL Database Azure Event Hubs Azure Monitor Experience with data security, access control, and governance in Azure environments.

📌 Azure DataBricks,PySpark (Hyderabad)
🏢 Kaizen Sra Technologies
📍 Hyderabad