Key Responsibilities : - Design, develop, and maintain scalable data pipelines using Databricks and Apache Spark.- Build and optimize ETL/ELT workflows for processing structured and unstructured data.- Develop data transformation and processing logic using Python, PySpark, and SQL.- Work with Delta Lake to manage reliable, scalable, and high-performance data storage.- Implement data ingestion solutions from multiple sources, including databases, APIs, files, and cloud storage.- Optimize Spark jobs, queries, and data pipelines for performance, scalability, and cost efficiency.- Implement data quality checks, validation rules, and monitoring mechanisms.- Collaborate with Data Architects, Data Scientists, Business Analysts, and other engineering teams to understand data requirements.- Work with cloud-based data platforms and services to build integrated data solutions.- Follow best practices for data security, governance, version control, and deployment.- Troubleshoot data pipeline failures and ensure data availability, accuracy,



and reliability.Required Skills & Qualifications : - Bachelor's degree in Computer Science, Information Technology, Engineering, or a related field.- Hands-on experience with Azure Databricks or Databricks.- Strong programming skills in Python and PySpark.- Robust proficiency in SQL and experience working with relational databases.- Good understanding of Apache Spark architecture and distributed data processing.- Experience developing and managing ETL/ELT data pipelines.- Hands-on experience with Delta Lake and Delta tables.- Understanding of data warehousing concepts, data modelling, and data integration.- Experience working with cloud platforms such as Microsoft Azure, AWS, or GCP.- Familiarity with Git and CI/CD practices for data engineering workflows.- Strong analytical, problem-solving, and debugging skills. (ref:hirist.tech)

📌 Celebal Technologies - Data Engineer - Databricks (Bengaluru)
🏢 CELEBAL TECHNOLOGIES PRIVATE
📍 Bengaluru