Description
We are looking for a Senior Databricks Engineer to design, develop, and optimize scalable data solutions using Databricks for large enterprise data platforms. The role requires strong hands on expertise in Spark, PySpark, SQL, and cloud ecosystems, with ownership of end to end data pipelines and mentoring of junior team members in an offshore delivery model.
Key Responsibilities
Data Engineering & Development
• Design and develop scalable data pipelines using Azure Databricks / Databricks on GCP
• Build and optimize ETL/ELT pipelines using Spark, PySpark, and SQL
• Implement Delta Lake for ACID transactions, data versioning, and performance optimization
• Handle structured and semi structured data (JSON, Parquet, Avro)
Cloud & Integration
• Integrate Databricks with Cloud Storage, ADLS / GCS, BigQuery, Snowflake
• Implement secure data access using IAM, service principals, secrets
• Work with CI/CD pipelines for Databricks deployments
• Collaborate with Data Scientists, BI teams, and downstream consumers
Delivery & Offshore Ownership
• Own offshore delivery for Databricks workstreams
• Provide technical guidance and code reviews to junior engineers
• Work closely with onshore counterparts and clients across time zones
• Participate in sprint planning, backlog grooming, and technical estimations
Required Skills
Technical Skills
• 6–10 years of experience in Data Engineering
• 3+ years of strong hands on experience with Databricks
• Robust expertise in Apache Spark, PySpark, Spark SQL
• Experience with Delta Lake
• Proficiency in SQL (query optimization, complex joins)
• Experience with Azure or GCP cloud platforms
• Experience with Git, CI/CD, and DevOps practices
Preferred Skills
• Experience with Azure Data Lake / BigQuery / Snowflake
• Exposure to streaming frameworks (Kafka, Spark Structured Streaming)
• Knowledge of data governance, data quality, and security
• Experience supporting production workloads
Soft Skills
•
📌 Databricks, Python (India)
🏢 Clifyx
📍 India