Position Overview We are seeking a highly skilled Senior Data Engineer - Python PySpark Azure Databricks to join our dynamic data engineering team This role focuses on building scalable high-performance data pipelines using Python and PySpark within the Azure Databricks setting While familiarity with broader Azure services is valuable the emphasis is on distributed data processing and automation using modern big data frameworks Prior experience in the Property Casualty P C insurance industry is a strong plus Key Responsibilities Data Pipeline Development Optimization Design develop and maintain scalable ETL ELT data pipelines using Python and PySpark Leverage Azure Databricks to process large volumes of structured and semi-structured data efficiently Implement data quality checks error handling and performance tuning across all stages of data processing Data Architecture Modeling Contribute to the design of cloud-based data architectures that support analytics and reporting use cases Develop and maintain data models that adhere to industry best practices and support business requirements Work with Delta Lake Bronze Silver Gold data architecture patterns and metadata management strategies Cloud Integration Azure Integrate and orchestrate data workflows using Azure Data Factory Azure Blob Storage and Event Hub where applicable Optimize cloud compute resources and manage cost-effective data processing at scale Collaboration Stakeholder Engagement Partner with data analysts data scientists and business users to understand evolving data needs Collaborate with DevOps and platform teams to ensure reliable secure and automated data operations Participate in Agile and contribute to sprint planning demos and retrospectives Documentation Best Practices Maintain clear and comprehensive documentation of code pipelines and architectural decisions Contribute to internal data engineering standards and promote best practices for code quality testing and CI CD Qualifications Position Overview We are seeking a highly skilled Senior Data Engineer - Python PySpark Azure Databricks to join our dynamic data engineering team This role focuses on building scalable high-performance data pipelines using Python and PySpark within the Azure Databricks environment While familiarity with broader Azure services is valuable the emphasis is on distributed data processing and automation using modern big data frameworks Prior experience in the Property Casualty P C insurance industry is a strong plus Key Responsibilities Data Pipeline Development Optimization Design develop and maintain scalable ETL ELT data pipelines using Python and PySpark Leverage Azure Databricks to process large volumes of structured and semi-structured data efficiently Implement data quality checks error handling and performance tuning across all stages of data processing Data Architecture Modeling Contribute to the design of cloud-based data architectures that support analytics and reporting use cases Develop and maintain data models that adhere to industry best practices and support business requirements Work with Delta Lake Bronze Silver Gold data architecture patterns and metadata management strategies Cloud Integration Azure Integrate and orchestrate data workflows using Azure Data Factory Azure Blob Storage and Event Hub where applicable Optimize cloud compute resources and manage cost-effective data processing at scale Collaboration Stakeholder Engagement Partner with data analysts data scientists and business users to understand evolving data needs Collaborate with DevOps and platform teams to ensure reliable secure and automated data operations Participate in Agile and contribute to sprint planning demos and retrospectives Documentation Best Practices Maintain clear and comprehensive documentation of code pipelines and architectural decisions Contribute to internal data engineering standards and promote best practices for code quality testing and CI CD Job Location