Skill: Data Engineer (Databricks + Pyspark)

Loc: Pune

Mode: 5 days WFO

Shift: UK shift – 2:30 to 11 PM

Exp: 8-13 Years

JD

We are looking for a skilled
Databricks Data Engineer
to design, develop, and optimize scalable data pipelines and data platforms. The ideal candidate will have strong expertise in
Databricks, SQL, Python, and PySpark
, along with experience in implementing
CI/CD practices
for data engineering workflows.

Primary Skills (Mandatory)

1. Databricks
- Strong hands-on experience with Databricks (Data Lakehouse architecture)
- Expertise in Delta Lake, Unity Catalog, and Databricks Workflows
- Experience in performance tuning and cost optimization

1. SQL
- Advanced SQL skills for data transformation, optimization, and querying
- Experience in writing complex joins, window functions, and performance tuning

1. Python
- Proficient in Python for data engineering tasks and scripting
- Experience with modular code development and reusable components

1. PySpark
- Strong experience in building scalable data pipelines using PySpark




- Knowledge of Spark optimization techniques (partitioning, caching, etc.)

1. CI/CD
- Experience implementing CI/CD pipelines for data workflows
- Familiarity with tools such as Azure DevOps, GitHub Actions, Jenkins, or similar
- Knowledge of version control (Git) and automated deployment of data solutions

Secondary Skills
- AWS
- Experience working with AWS services such as S3, EC2, IAM, and EMR
- Apache Airflow
- Experience in workflow orchestration and scheduling using Airflow
- AWS Glue
- Hands-on experience in ETL processing and Glue jobs

Good to Have
- Terraform
- Knowledge of Infrastructure as Code (IaC) for provisioning cloud resources
- Experience in automation and setting setup using Terraform

Key Responsibilities
- Design and develop scalable data pipelines using Databricks and PySpark
- Build and optimize ETL/ELT processes for large-scale data processing
- Collaborate with data architects and stakeholders

📌 Data Engineer (Pune)
🏢 LTIMindtree
📍 Pune