Data Engineer – Databricks, Spark, Python & ETL

Experience: 5–8 Years

Location: [Location]

Employment Type: Full time

Job Summary

We are looking for a skilled Data Engineer with strong expertise in Databricks, Apache Spark, Python, SQL, and ETL development. The ideal candidate should have experience building scalable data pipelines, optimizing big data processing workflows, and working with orchestration and CI/CD tools in cloud-based environments.

Key Responsibilities
Design, develop, and maintain scalable ETL/ELT pipelines using Databricks and Apache Spark.
Develop data processing applications using Python and PySpark.
Build and optimize complex SQL queries, stored procedures, and transformations.
Work with structured and unstructured datasets for large-scale data processing.
Implement data integration workflows and orchestration using tools such as Airflow, Azure Data Factory (ADF), or Autosys.
Develop reusable frameworks and automate data engineering workflows.
Monitor and troubleshoot production data pipelines and resolve performance bottlenecks.
Collaborate with business analysts, data scientists, and cross-functional teams for data requirements.
Implement CI/CD pipelines for deployment automation and version control.
Ensure data quality, governance, security, and compliance standards are followed.




Participate in code reviews, testing, and documentation activities.

Required Skills
Robust hands-on experience with Databricks and Apache Spark
Expertise in Python / PySpark
Solid SQL and ETL development experience
Experience with orchestration tools like:
Apache Airflow
Azure Data Factory (ADF)
Autosys
Experience with CI/CD tools and deployment pipelines
Knowledge of data warehousing and big data concepts
Experience with Git/version control systems
Solid debugging and performance optimization skills
Good understanding of distributed data processing systems

Good to Have
Experience with Azure, AWS, or GCP cloud platforms
Knowledge of Delta Lake, Kafka, or streaming technologies
Familiarity with DevOps and Infrastructure as Code (IaC)
Exposure to Snowflake or other cloud data warehouses
Experience working in Agile/Scrum environments

Preferred Qualifications
Bachelor's degree in Computer Science, Information Technology, or related field
Strong analytical and problem-solving abilities
Excellent communication and stakeholder management skills

Technology Stack
Databricks
Apache Spark / PySpark
Python
SQL
ETL/ELT
Airflow / ADF / Autosys
CI/CD
Git
-

📌 Data Engineer Pune (India)
🏢 TalentOla
📍 India