Data Engineer – Databricks | PySpark | Azure

Employment Type: Full-Time
Experience: 5–8 Years
Work Mode: Remote

Job Summary

We are looking for a skilled Data Engineer with solid hands-on experience in Databricks, PySpark, Delta Lake, Azure Data Factory (ADF), Azure Logic Apps, and Apache Airflow. The ideal candidate will be responsible for building scalable data pipelines, developing modern Lakehouse solutions, and implementing reliable workflow orchestration on Microsoft Azure.

The role will involve working with large-scale data processing, ETL/ELT development, pipeline optimization, data quality, and cloud-based data engineering solutions. This skill combination is commonly used for modern Azure data platforms and Databricks-based orchestration.

Key Responsibilities
- Design, develop, and maintain scalable ETL/ELT pipelines using Databricks and PySpark.
- Develop data transformation and processing solutions using Delta Lake.
- Build and manage data integration pipelines using Azure Data Factory (ADF).
- Create workflow automation and integrations using Azure Logic Apps.




- Develop and manage Apache Airflow DAGs for scheduling and orchestration.
- Integrate Airflow, ADF, Logic Apps, and Databricks for end-to-end workflow orchestration.
- Work with ADLS Gen2 for data ingestion, storage, and processing.
- Optimize Spark and Databricks workloads for performance, scalability, and cost efficiency.
- Develop reusable frameworks for data ingestion, transformation, validation, and processing.
- Implement data quality checks, monitoring, logging, alerting, and Data Observability practices.
- Support CI/CD pipelines for data engineering code and deployment workflows.
- Collaborate with Data Scientists, BI teams, Architects, and business stakeholders.
- Prepare reliable and high-quality datasets for analytics and AI/ML workloads.
- Support ML workflows and integrate MLflow where required.
- Troubleshoot production issues and ensure data pipeline reliability and SLA adher

📌 Data Engineer (India)
🏢 WebSenor
📍 India