We are seeking an experienced AWS Databricks Engineer to design, develop, and optimize scalable data engineering solutions on AWS using Databricks. The ideal candidate should have robust expertise in Spark, Python, SQL, and cloud-native data services to build modern data pipelines and analytics platforms.
Key Responsibilities:
- Design and develop scalable ETL/ELT pipelines using Databricks on AWS.
- Build and optimize data processing workflows using Apache Spark (PySpark).
- Develop data ingestion frameworks from multiple sources (batch and streaming).
- Work with AWS services such as S3, Glue, Lambda, IAM, CloudWatch, EMR (optional), and Redshift.
- Optimize Spark jobs for performance and cost.
- Implement Delta Lake architecture for reliable data processing.
- Collaborate with data scientists, analysts, and business stakeholders.
- Ensure data quality, governance, security, and compliance.
- Automate deployments using CI/CD pipelines.
- Monitor production jobs and troubleshoot issues.