Role Summary
Design, build, and support scalable data pipelines and analytics solutions using Databricks and Azure-native services. The role focuses on end-to-end data ingestion, transformation, and serving for batch and real-time analytics, with strong emphasis on quality, performance, and governance.

Key Responsibilities

- Design and develop data pipelines using Databricks (PySpark) for batch and near real-time processing.
- Build and orchestrate pipelines using Azure Data Factory and Databricks workflows.

- Implement ingestion, cleansing, enrichment, and modeling across Azure Data Lake and Federated Data Lake architectures.
- Develop optimized Spark jobs, manage partitions, and improve performance and cost efficiency.
- Publish curated datasets and analytics-ready tables for downstream reporting and analytics.
- Implement data quality checks, validation frameworks, and monitoring aligned with client benchmarks.
- Apply data security, privacy, and governance standards including PII handling, RBAC, and encryption.
- Collaborate with data governance and platform teams; contribute to design reviews and CI/CD pipelines.

Required Skills & Experience

- Strong hands-on experience with Databricks and PySpark.
- Experience with Azure Data Lake (ADLS Gen2) and Federated Data Lake concepts.




- Proficiency in Azure Data Factory for pipeline orchestration.
- Solid SQL and data modeling experience
- Understanding of batch vs. streaming processing, incremental loads, and data quality frameworks.
- Familiarity with enterprise data governance, security, and compliance practice

- Experience in other data platforms such as AWS Glue, GCP BigQuery, Informatica, Talend, etc. is not considered relevant for these roles.
- Hands-on, real project experience is expected, especially in configuring and managing data pipelines and extracting data from various data sources.
- Strong SQL skills are mandatory: candidates should be able to write and optimize complex SQL queries, handle scenario-based questions, and demonstrate on-the-spot problem solving.
- Good English communication skills are required

Positive To Have:

- Exposure to supply chain

- Experience working with ERP systems (e.g., SAP environments/4HANA)

- Knowledge of Delta Lake concepts

- Exposure to CI/CD practices in Azure DevOps

Disclaimer : This job posting has been aggregated from external source. Role details, content, and availability are subject to change. Applicants are advised to confirm the latest information directly on the company website before applying.

📌 Databricks Data Engineer / Developer (Hubballi)
🏢 SSD Shared Services
📍 Hubballi