Apache Spark
Spark Core
Spark SQL
Spark DataFrame
Spark Dataset
Spark Structured Streaming
Spark transformations and actions
RDD concepts
Joins and aggregations
Window functions
Partitioning and repartitioning
Caching and persistence
Broadcast joins
Handling data skew
Spark job optimization
Spark cluster execution and troubleshooting
2. Scala – Mandatory
Strong programming experience in Scala.
Functional programming concepts.
Collections and higher-order functions.
Case classes, traits, objects, pattern matching.
Exception handling and reusable code development.
Development of Spark applications using Scala.
Ability to write clean, modular, scalable, and maintainable Scala code.
3. Apache Kafka – Mandatory
Strong Practical Experience With:
Kafka architecture
Kafka brokers
Topics and partitions
Producers and consumers
Consumer groups
Offsets and offset management
Replication factor
Partition strategy
Message retention
Kafka Producer/Consumer APIs
Kafka Streams / Kafka Connect
Schema Registry
Avro / JSON serialization
Consumer lag monitoring
Kafka troubleshooting
Kafka performance tuning
Integration of Kafka with Spark Structured Streaming
4. SQL – Mandatory
Strong SQL Skills Including:
Complex SQL queries
Joins
Subqueries
CTEs
Window functions
Aggregations
Query optimization
Data validation and reconciliation
Stored procedures/functions where applicable
Relational database concepts
5. Data Engineering
Strong Understanding Of:
ETL / ELT
Batch processing
Real-time/streaming processing
Data pipelines
Data lakes
Data warehouses
Data modeling
Dimensional modeling
Distributed systems
Data partitioning
Data quality
Data governance
Large-scale data processing
Cloud / Big Data
📌 DE-Spark,Scala,kafka (Bengaluru)
🏢 Zorba AI
📍 Bengaluru
Reply to this offer
Impress this employer describing Your skills and abilities, fill out the form below and leave Your personal touch in the presentation letter.