We are seeking an experienced Python and PySpark Developer (Azure Data Engineer) with a strong background in data solution design, Azure Databricks, and ETL pipeline development. The ideal candidate will have hands-on experience with Medallion Architecture, Atlas Framework, and the ability to design and maintain scalable, high-performance data solutions in a cloud environment.
Key Responsibilities
- Design, develop, and maintain data pipelines and data products using Azure Databricks and PySpark.
- Build scalable and efficient ETL workflows integrating data from diverse sources such as Azure Storage Account, SFTP, and Fabric.
- Apply the Medallion Architecture principles to structure and optimize data layers (Bronze, Silver, Gold).
- Ensure data quality, accuracy, and completeness through validation, checks, and automation.
- Collaborate closely with data owners, business analysts, and stakeholders to translate requirements into technical solutions.
- Implement monitoring and alerting mechanisms to proactively detect and resolve data pipeline issues.
- Contribute to solution design and architecture discussions for domain-specific data sets and products.
Required Skills & Qualifications
- 10+ years of relevant experience in data engineering and data pipeline development.
- Strong expertise in Azure Databricks, Apache Spark, PySpark, and ETL frameworks.
- Proven experience with the Atlas Framework and Medallion Architecture.
- Solid programming skills in Python (Scala knowledge is a plus).
- Proficiency in SQL and data transformation logic.
- Hands-on experience with Azure Cloud Platform and its data services.
- Familiarity with Unity Catalog and Databricks-specific tools.
- Strong analytical, problem-solving, and communication skills.
- Ability to work in a team-oriented and fast-paced environment.
Why Join Us
- Opportunity to work on cutting-edge cloud data technologies.
- Collaborative team environment with continuous learning opportunities.
- Be part of data modernization initiatives driving business innovation.
Required Skills for Python & PySpark Developer Job