PySpark, Azure Data Factory Responsibilities Design, develop, and maintain data pipelines using
Azure Data Factory . Manage and optimize data storage using
Azure Data Lake Gen 2 . Build and process large-scale data solutions with
Azure Databricks and
Apache Spark . Create interactive reports and dashboards in
Power BI for business insights. Collaborate with cross-functional IT and business teams to translate requirements into data solutions. Ensure data quality, security, and compliance in line with organizational standards.
Core Skills Required Azure Data Factory : Experience in building and managing data pipelines.
Azure Data Lake
Gen 2 : Proficient in data storage and management within Azure's data lake workplace.
Azure Databricks / Apache Spark : Hands-on skill with distributed data processing, transformations, and analytics.
Power BI : Expertise in data visualization and reporting.