Responsibilities
- Analyze the business requirements and needs of the organization.
- Architect and design robust, scalable, and secure enterprise data platform leveraging Databricks and various cloud services on Microsoft Azure.
- Lead implementation of advanced data analytics solutions utilizing Databricks and Delta Lake architectures to extract, transform, and load (ETL) data.
- Leverage Databricks to perform data engineering, artificial intelligence and machine learning tasks.
- Collect, process, and analyze large datasets from various sources within the aviation domain (e.g., flight data, baggage information, weather data, operational logs etc.).
- Perform data quality checks and ensure data accuracy and consistency.
- Identify trends, patterns, and anomalies in data to provide valuable insights and support business decision-making.
- Contribute to the development of data models and database designs.
- Document data sources, data flows, and analytical processes.
- Stay up to date with the latest trends and technologies in data analysis and the aviation industry.
- Create insightful and interactive data visualizations and reports using tools like Power BI to communicate findings to non-technical audiences.
- Define and govern enterprise-wide cloud architecture standards, security protocols, and operational best practices.
- Developing and implementing strategies to connect different systems, ensuring they can share data and functionality
- Defining and managing API that allow different software applications to communicate with each other
- Defining and managing asynchronous data exchange between applications and as a part of data pipelines
- Designing real-time streaming pipelines for IoT time-series data
- Diving deep into the details to solve complex technical challenges
- Ensure the platform is secure and adheres to best practices for data protection and privacy.
- Ensuring high data quality and integrity to avoid operational failures and improve system accur