Design, develop, and optimize scalable data engineering solutions using Databricks and Apache Spark.
Develop and maintain ETL/ELT pipelines and support enterprise data engineering initiatives using Databricks.
Validate Contemporary Data Platform implementations with a focus on Databricks and Snowflake through end-to-end data testing.
Key Responsibilities
- Design and develop ETL/ELT pipelines
- Build Spark applications (PySpark/Scala)
- Optimize Spark jobs
- Work with Delta Lake & Unity Catalog
- Integrate cloud data sources
- Ensure data quality and governance
- Develop ETL/ELT pipelines
- Build PySpark applications
- Write and optimize SQL queries
- Work with Delta Lake
- Monitor production pipelines
- Support data migration and transformations
- Validate ETL/ELT pipelines
- Perform source-to-target data validation
- Test business rules and transformations
Mandatory Skills
- 5+ years Data Engineering experience
- Databricks
- Apache Spark (PySpark/Scala)
- Advanced SQL
- Delta Lake
- Azure/AWS Databricks
- Git & CI/CD
- Airflow/ADF
Preferred Skills
Snowflake, Azure Data Factory, AWS Glue, DevOps, Agile
Snowflake, Azure Data Factory, Airflow, Data Lake Concepts
PySpark, Azure Data Factory, Airflow, Test Automation, Agile
Disclaimer: This job posting has been aggregated from external source. Role details, content, and availability are subject to change. Applicants are advised to confirm the latest information directly on the company website before applying.