Role Descriptions: Design| develop| and maintain scalable data pipelines using Azure Databricks and PySpark.Build and optimize ETLELT workflows for processing structured and unstructured data.Develop data ingestion frameworks from multiple sources| including databases| APIs| files| and cloud storage.Implement data transformation| cleansing| and validation processes.Optimize Spark jobs for performance| scalability| and cost efficiency.Design and maintain data lakes| data warehouses| and analytical datasets.Ensure data quality| integrity| security| and governance standards are followed.Collaborate with data architects| analysts| and business teams to understand data requirements.Monitor| troubleshoot| and resolve data pipeline issues.Support deployment and automation through CICD and DevOps practices.
Essential Skills:
We are seeking a highly skilled Data Engineer with solid expertise in Azure Databricks and PySpark to design| develop| and maintain scalable data pipelines and modern data platforms. The ideal candidate will have experience working with large-scale datasets| implementing ETLELT processes| and building cloud-based data engineering solutions on Microsoft Azure.The role will involve collaborating with business stakeholders| data analysts| architects| and development teams to enable reliable| high-quality| and efficient data processing solutions that support analytics| reporting| and business intelligence initiatives.