5–10 years Data Engineering Strong Scala programming Strong Apache Spark Strong Apache Kafka Spark Structured Streaming Advanced SQL ETL/ELT & Data Pipelines Distributed Systems Cloud/Databricks exposure Required Technical Skills1.

Apache

Spark – Mandatory Strong Hands-on Experience With: Apache Spark

Spark Core

Spark SQL

Spark DataFrame

Spark Dataset

Spark Structured Streaming

Spark transformations and actions

RDD concepts

Joins and aggregations

Window functions

Partitioning and repartitioning

Caching and persistence

Broadcast joins

Handling data skew

Spark job optimization

Spark cluster execution and troubleshooting 2. Scala – Mandatory Strong programming experience in Scala.

Functional programming concepts.

Collections and higher-order functions.

Case classes, traits, objects, pattern matching.

Exception handling and reusable code development.

Development of Spark applications using Scala.

Ability to write clean, modular, scalable, and maintainable Scala code. 3.

Apache

Kafka – Mandatory Strong Practical Experience With: Kafka architecture

Kafka brokers

Topics and partitions

Producers and consumers

Consumer groups

Offsets and offset management

Replication factor

Partition strategy

Message retention

Kafka Producer/Consumer APIs

Kafka Streams / Kafka Connect

Schema Registry

Avro / JSON serialization

Consumer lag monitoring

Kafka troubleshooting

Kafka performance tuning

Integration of Kafka with Spark Structured Streaming 4. SQL – Mandatory Strong SQL Skills Including: Complex SQL queries

Joins

Subqueries

CTEs

Window functions

Aggregations

Query optimization

Data validation and reconciliation

Stored procedures/functions where applicable

Relational database concepts 5.

Data Engineering Strong Understanding Of: ETL / ELT

Batch processing

Real-time/streaming processing

Data pipelines

Data lakes

Data warehouses

Data modeling

Dimensional modeling

Distributed systems

Data partitioning

Data quality

Data governance

Large-scale data processing Cloud / Big Data Experience Experience With At Least One Cloud Platform Is Preferred: Required Technical Skills1.

Apache

Spark – Mandatory Strong Hands-on Experience With: Apache Spark

Spark Core

Spark SQL

Spark DataFrame





Spark Dataset

Spark Structured Streaming

Spark transformations and actions

RDD concepts

Joins and aggregations

Window functions

Partitioning and repartitioning

Caching and persistence

Broadcast joins

Handling data skew

Spark job optimization

Spark cluster execution and troubleshooting 2. Scala – Mandatory Strong programming experience in Scala.

Functional programming concepts.

Collections and higher-order functions.

Case classes, traits, objects, pattern matching.

Exception handling and reusable code development.

Development of Spark applications using Scala.

Ability to write clean, modular, scalable, and maintainable Scala code. 3.

Apache

Kafka – Mandatory Strong Practical Experience With: Kafka architecture

Kafka brokers

Topics and partitions

Producers and consumers

Consumer groups

Offsets and offset management

Replication factor

Partition strategy

Message retention

Kafka Producer/Consumer APIs

Kafka Streams / Kafka Connect

Schema Registry

Avro / JSON serialization

Consumer lag monitoring

Kafka troubleshooting

Kafka performance tuning

Integration of Kafka with Spark Structured Streaming 4. SQL – Mandatory Strong SQL Skills Including: Complex SQL queries

Joins

Subqueries

CTEs

Window functions

Aggregations

Query optimization

Data validation and reconciliation

Stored procedures/functions where applicable

Relational database concepts 5.

Data Engineering Strong Understanding Of: ETL / ELT

Batch processing

Real-time/streaming processing

Data pipelines

Data lakes

Data warehouses

Data modeling

Dimensional modeling

Distributed systems

Data partitioning

Data quality

Data governance

Large-scale data processing Cloud / Big Data Experience Strong Understanding Of: ETL / ELT

Batch processing

Real-time/streaming processing

Data pipelines

Data lakes

Data warehouses





Data modeling

Dimensional modeling

Distributed systems

Data partitioning

Data quality

Data governance

Large-scale data processing Cloud / Big Data Experience Required Technical Skills1.

Apache

Spark – Mandatory Strong Hands-on Experience With: Apache Spark

Spark Core

Spark SQL

Spark DataFrame

Spark Dataset

Spark Structured Streaming

Spark transformations and actions

RDD concepts

Joins and aggregations

Window functions

Partitioning and repartitioning

Caching and persistence

Broadcast joins

Handling data skew

Spark job optimization

Spark cluster execution and troubleshooting

Scala – Mandatory

Robust programming experience in Scala.

Functional programming concepts.

Collections and higher-order functions.

Case classes, traits, objects, pattern matching.

Exception handling and reusable code development.

Development of Spark applications using Scala.

Ability to write clean, modular, scalable, and maintainable Scala code.

Apache Kafka – Mandatory Strong Practical Experience With: Kafka architecture

Kafka brokers

Topics and partitions

Producers and consumers

Consumer groups

Offsets and offset management

Replication factor

Partition strategy

Message retention

Kafka Producer/Consumer APIs

Kafka Streams / Kafka Connect

Schema Registry

Avro / JSON serialization

Consumer lag monitoring

Kafka troubleshooting

Kafka performance tuning

Integration of Kafka with Spark Structured Streaming

SQL – Mandatory Strong SQL Skills Including: Complex SQL queries

Joins

Subqueries

CTEs

Window functions

Aggregations

Query optimization

Data validation and reconciliation

Stored procedures/functions where applicable

Relational database concepts

Data Engineering Strong Understanding Of: ETL / ELT

Batch processing

Real-time/streaming processing

Data pipelines

Data lakes

Data warehouses

Data modeling

Dimensional modeling

Distributed systems

Data partitioning

Data quality

Data governance

Large-scale data processing Cloud / Big Data Experience Experience With At Least One Cloud Platform Is Preferred: Experience with at least one cloud platform is preferred: Skills: kafka,scala,spark,data

📌 DE-Spark,Scala,kafka (Bengaluru)
🏢 Zorba AI
📍 Bengaluru