5–10 years Data Engineering
Strong Scala programming
Robust Apache Spark
Solid Apache Kafka
Spark Structured Streaming
Advanced SQL
ETL/ELT & Data Pipelines
Distributed Systems
Cloud/Databricks exposure
Required Technical Skills1. Apache Spark – Mandatory
Robust hands-on experience with:
Apache Spark
Spark Core
Spark SQL
Spark DataFrame
Spark Dataset
Spark Structured Streaming
Spark transformations and actions
RDD concepts
Joins and aggregations
Window functions
Partitioning and repartitioning
Caching and persistence
Broadcast joins
Handling data skew
Spark job optimization
Spark cluster execution and troubleshooting
Scala – Mandatory
Solid programming experience in Scala.
Functional programming concepts.
Collections and higher-order functions.
Case classes, traits, objects, pattern matching.
Exception handling and reusable code development.
Development of Spark applications using Scala.
Ability to write clean, modular, scalable, and maintainable Scala code.
Apache Kafka – Mandatory




Strong practical experience with:
Kafka architecture
Kafka brokers
Topics and partitions
Producers and consumers
Consumer groups
Offsets and offset management
Replication factor
Partition strategy
Message retention
Kafka Producer/Consumer APIs
Kafka Streams / Kafka Connect
Schema Registry
Avro / JSON serialization
Consumer lag monitoring
Kafka troubleshooting
Kafka performance tuning
Integration of Kafka with Spark Structured Streaming
SQL – Mandatory
Strong SQL skills including:
Complex SQL queries
Joins
Subqueries
CTEs
Window functions
Aggregations
Query optimization
Data validation and reconciliation
Stored procedures/functions where applicable
Relational database concepts
Data Engineering
Strong understanding of:
ETL / ELT
Batch processing
Real-time/streaming processing
Data pipelines
Data lakes
Data warehouses
Data modeling
D

📌 De Spark,scala,kafka Bengaluru
🏢 Zorba Consulting India
📍 Bengaluru