We are seeking a skilled and experienced Cloud Azure Databricks Data Engineer to join our dynamic team. The ideal candidate will have extensive experience in building, managing, and optimizing data pipelines and solutions on Azure Databricks. They should also possess strong expertise in PySpark, Python, SQL, Unity Catalog, and excellent communication skills.
Key Responsibilities
- Design, develop, and maintain scalable data pipelines and workflows on Azure Databricks.
- Delivery lead from onsite for all client communication and escalations.
- Utilize PySpark and Python for data processing, transformation, and integration.
- Develop and optimize SQL queries to enable efficient data retrieval and analysis.
- Implement and manage Unity Catalog for data governance, access control, and metadata management.
- Collaborate with cross-functional teams to understand business requirements and translate them into technical solutions.
- Optimize data pipelines for performance, scalability, and cost-effectiveness.
- Ensure the security, compliance, and integrity of data.
- Troubleshoot and resolve issues related to data pipelines and processes.
- Create and maintain documentation for data workflows, processes, and solutions.
- Communicate effectively with stakeholders to provide project updates and insights.
Required Skills and Qualifications
- 5+ years of experience in data engineering or a related field.
- Strong hands-on experience with Azure Databricks.
- Proficiency in PySpark and Python for large-scale data processing.
- In-depth knowledge of SQL for data analysis and transformation.
- Experience with Unity Catalog for managing data governance and access control.
- Familiarity with Azure cloud services such as Azure Data Lake, Azure Data Factory, and Azure Synapse Analytics.
- Understanding of data modelling concepts and ETL/ELT workflows.
- Excellent problem-solving and analytical skills.
- Solid communication skills, both written and verbal, to collaborate with technical and non-technical stakeholders.
Preferred Qualifications
- Experience with CI/CD pipelines and DevOps practices in a cloud environment.
- Knowledge of streaming data frameworks like Kafka or Spark Streaming.
- Certifications in Azure Data Engineering or related technologies.
📌 Azure Databricks Data Engineer (Noida)
🏢 Topcoder
📍 Noida
Reply to this offer
Impress this employer describing Your skills and abilities, fill out the form below and leave Your personal touch in the presentation letter.