Role & responsibilities

We are seeking a skilled Databricks Engineer with strong expertise in PySpark, Databricks, and Azure/AWS Cloud platforms. The ideal candidate will be responsible for building scalable data pipelines, processing large datasets, optimizing ETL workflows, and supporting data engineering initiatives within a contemporary cloud-based data ecosystem.

Key Responsibilities
Design, develop, and maintain scalable data pipelines using PySpark and Databricks.
Build and optimize ETL/ELT processes for processing large-scale structured and unstructured data.
Develop data ingestion frameworks from multiple data sources including databases, APIs, and cloud storage.
Work with Delta Lake, Spark SQL, and Databricks Workflows for efficient data processing.
Implement data quality checks, monitoring, and troubleshooting of data pipelines.
Collaborate with data architects, analysts, and business stakeholders to understand data requirements.
Optimize Spark jobs for performance, scalability, and cost efficiency.
Integrate Databricks solutions with cloud platforms such as Azure, AWS, or GCP.




Support CI/CD implementation and deployment of data engineering solutions.
Ensure data governance, security, and compliance standards are followed.

Required Skills
6+ years of experience in Data Engineering.
Strong hands-on experience with Databricks and PySpark.
Expertise in Apache Spark, Spark SQL, and DataFrame APIs.
Robust knowledge of ETL/ELT development and data warehousing concepts.
Experience with Delta Lake, Databricks Notebooks, and Databricks Workflows.
Proficiency in Python programming.
Robust SQL skills and database concepts.
Experience with cloud platforms such as Azure, AWS, or GCP.
Knowledge of Git, CI/CD pipelines, and DevOps practices.
Experience working with large-scale distributed data processing systems.

Preferred Skills
Experience with Azure Data Factory (ADF), Azure Data Lake Storage (ADLS), or AWS S3.
Knowledge of Kafka, Event Hubs, or streaming technologies.
Exposure to Data Modeling and Data Governance concepts.
Experience with Airflow or other workflow orchestration tools.
Databricks Certification is a plus.

📌 Databricks Engineer Hyderabad
🏢 CGI
📍 Hyderabad