Work Mode: Remote

Employment Type: Part-Time | Freelance

About the Role

We are looking for a skilled Databricks & PySpark Engineer with strong hands-on experience in Python, SQL, data transformations, and data pipeline debugging. The ideal candidate will be responsible for troubleshooting existing Databricks notebooks, validating data-processing logic, identifying data discrepancies, and ensuring reliable data pipeline execution across multiple environments.

Key Responsibilities

- Debug existing Databricks notebooks and troubleshoot data pipeline issues.
- Develop, review, and validate data transformation logic using PySpark, Python, SQL, and DataFrames.
- Analyze conditional logic, thresholds, null handling, and data validation rules.
- Compare input and output datasets to identify discrepancies and ensure data accuracy.
- Work with Databricks catalogs, schemas, tables, job configurations, and access permissions.
- Support data validation and testing across Dev, QA, UAT, and Production environments.




- Create test datasets and validate data-processing scenarios.
- Follow safe data-writing practices, production access controls, and deployment procedures.
- Collaborate with stakeholders to investigate data quality issues and implement reliable fixes.

Mandatory Technical Skills

- Strong hands-on experience with Databricks and PySpark.
- Proficiency in Python, SQL, DataFrames, and data transformation logic.
- Experience debugging notebooks and resolving data pipeline issues.
- Solid understanding of null handling, conditional logic, thresholds, and data validation.
- Experience with input/output dataset reconciliation and discrepancy analysis.
- Knowledge of Databricks catalogs, schemas, tables, jobs, and permissions.
- Understanding of Dev, QA, UAT, and Production environments.
- Experience creating test data and validating data-processing logic.

📌 Freelance Opportunity - Databricks & PySpark Engineer (India)
🏢 IndiTech Consulting
📍 India