We are seeking a highly skilled Data Engineer with a strong software engineering mindset and deep expertise in Python-based data engineering. This is not a traditional SQL-centric data engineering role. We are specifically looking for engineers with strong programming, debugging, and distributed data processing experience, who can build scalable and maintainable data platforms using modern cloud and lakehouse technologies.
The ideal candidate will have experience developing data pipelines using Python and Spark, working with modern data lake platforms such as Databricks and Snowflake, and deploying solutions on AWS. Experience with Dagster for data orchestration is highly desirable.
Key Responsibilities
Data Platform Development
• Design, develop, and maintain scalable data pipelines using Python and distributed processing frameworks.
• Build reusable, production-grade data engineering components and frameworks.
• Develop and optimize high-volume batch and streaming data processing solutions.
• Implement robust testing, monitoring, and observability for data pipelines.
• Troubleshoot complex data processing and performance issues.
Lakehouse & Data Engineering
• Build and maintain up-to-date lakehouse architectures using Databricks, Snowflake, and AWS data services.
• Design scalable data models and storage strategies for structured and semi-structured data.
• Develop ingestion, transformation, validation, and enrichment pipelines.
• Support data quality, lineage, and governance initiatives.
Orchestration & Automation
• Develop and manage workflow orchestration solutions using Dagster or similar orchestration frameworks.
• Automate data operations and deployment processes.
• Monitor pipeline performance and optimize reliability and efficiency.
Cloud Engineering
• Build cloud-native data solutions on AWS.
• Integrate data services across storage, processing, orchestration, and analytics platforms.
• Implement security, scalability, and resiliency be