Streaming & Real-Time Data Processing
Design and develop real-time streaming pipelines using Databricks Structured Streaming.
Build and maintain Kafka-based ingestion frameworks.
Handle late-arriving events using watermarks, event-time processing, and stateful streaming concepts.
Implement exactly-once processing and checkpointing mechanisms.
Monitor and optimize streaming workloads for performance and reliability.
Spark Declarative Pipelines (SDP) & Delta Live Tables (DLT):
Design and implement Spark Declarative Pipelines using Databricks.
Develop Delta Live Table (DLT) pipelines for scalable data transformations.
Implement data quality expectations, validations, retries, and failure handling within DLT pipelines.
Manage pipeline dependencies and orchestration using declarative approaches.
Understand advantages of SDP over traditional ETL pipelines, including:
Simplified pipeline development
Reduced operational overhead
Automated lineage tracking
Improved maintainability
Enhanced observability
Databricks Asset Bundles:
Develop and deploy Databricks Asset Bundles (DAB) for CI/CD.




Configure mandatory bundle components including:
Resource definitions
Job definitions
Workplace configurations
Source code artifacts
Manage deployment across development, testing, and production environments.
Data Ingestion & Auto Loader
Build scalable ingestion frameworks using Databricks Auto Loader (Cloud Files).
Configure schema inference and schema evolution strategies.
Handle duplicate records and implement deduplication mechanisms.
Design robust ingestion pipelines from ADLS Gen2 and other cloud storage systems.
Work with Cloud Files architecture and incremental processing patterns.
Lakehouse & Medallion Architecture
Design and implement Bronze, Silver, and Gold layer architectures.
Build enterprise-grade data products using Lakehouse principles.
Implement Delta Lake optimization techniques.
Ensure data quality, governance, and lineage across layers.
Azure Data Engineering
Work ext

📌 Senior Data Engineer (New Delhi)
🏢 Xebia
📍 New Delhi