Role & responsibilities

We are seeking a skilled Databricks Engineer with strong expertise in PySpark, Databricks, and Azure/AWS Cloud platforms. The ideal candidate will be responsible for building scalable data pipelines, processing large datasets, optimizing ETL workflows, and supporting data engineering initiatives within a contemporary cloud-based data ecosystem.

Key Responsibilities

- Design, develop, and maintain scalable data pipelines using PySpark and Databricks.
- Build and optimize ETL/ELT processes for processing large-scale structured and unstructured data.
- Develop data ingestion frameworks from multiple data sources including databases, APIs, and cloud storage.
- Work with Delta Lake, Spark SQL, and Databricks Workflows for efficient data processing.
- Implement data quality checks, monitoring, and troubleshooting of data pipelines.
- Collaborate with data architects, analysts, and business stakeholders to understand data requirements.
- Optimize Spark jobs for performance, scalability, and cost efficiency.
- Integrate Databricks solutions with cloud platforms such as Azure, AWS, or GCP.




- Support CI/CD implementation and deployment of data engineering solutions.
- Ensure data governance, security, and compliance standards are followed.

Required Skills

- 6+ years of experience in Data Engineering.
- Strong hands-on experience with Databricks and PySpark.
- Expertise in Apache Spark, Spark SQL, and DataFrame APIs.
- Strong knowledge of ETL/ELT development and data warehousing concepts.
- Experience with Delta Lake, Databricks Notebooks, and Databricks Workflows.
- Proficiency in Python programming.
- Strong SQL skills and database concepts.
- Experience with cloud platforms such as Azure, AWS, or GCP.
- Knowledge of Git, CI/CD pipelines, and DevOps practices.
- Experience working with large-scale distributed data processing systems.

Preferred Skills

- Experience with Azure Data Factory (ADF), Azure Data Lake Storage (ADLS), or AWS S3.
- Knowledge of Kafka, Event Hubs, or streaming technologies.
- Exposure to Data Modeling and Data Governance concepts.
- Experience with Airflow or other workflow orchestration tools.
- Databricks Certification is a plus.

📌 Databricks Engineer (Hyderabad)
🏢 CGI
📍 Hyderabad