Primary Skills
We are seeking a highly skilled Databricks Data Engineer to design, build, and optimize scalable data pipelines and lakehouse architectures. The ideal candidate will have strong expertise in Apache Spark, PySpark, Delta Lake, and contemporary cloud-based data platforms, with the ability to transform complex business requirements into reliable and performant data solutions.
Location : All Brillio Location:
Experience:4 to 8yrs
? Key Responsibilities
1. Data Engineering & Pipeline Development
Design and develop scalable ETL/ELT pipelines using PySpark and Spark SQL
Build and maintain Databricks workflows (Jobs) for orchestration
Implement Delta Live Tables (DLT) for declarative pipeline development
Develop and manage batch and streaming data pipelines
2. Lakehouse Architecture Implementation
Design and implement Medallion Architecture (Bronze, Silver, Gold layers)
Build curated datasets for analytics and reporting
Optimize storage using Delta Lake best practices
3. Data Ingestion & Integration
Ingest data from multiple sources:
Databases (RDBMS, NoSQL)
APIs and streaming platforms (Kafka, Event Hubs)
Files (CSV, JSON, Parquet)
Handle structured and semi-structured data efficiently
4. Delta Lake & Performance Optimization
Implement:
ACID transactions
Schema enforcement and evolution
Change Data Capture (CDC)
Optimize Spark jobs using:
Partitioning strategies
Caching and broadcast joins
File compaction and indexing (Z-ORDER)
5. Data Quality & Governance
Implement data validation and quality checks
Ensure compliance with data governance standards
Use Unity Catalog for access control and data lineage
Maintain auditability and data traceability
6. Monitoring & Reliability
Build logging, monitoring, and alerting for pipelines
Troubleshoot failures and optimize performance
Ensure high availability and fault tolerance
7. Collaboration & Delivery
Work closely with Data Analysts, Data Scientists, and stakeholders
Translate business requirements
📌 Senior Data Engineer (Bengaluru)
🏢 Brillio
📍 Bengaluru
Reply to this offer
Impress this employer describing Your skills and abilities, fill out the form below and leave Your personal touch in the presentation letter.