Technical Expertise

- Strong hands-on experience with Apache Spark (Spark Core, Spark SQL, Structured Streaming, DataFrames, Dataset APIs).

- Proven experience building and operating large-scale distributed data processing systems.

- Robust understanding of Spark performance tuning, partitioning strategies, joins, shuffles, caching, memory management, and resource optimization.

- Experience with one or more programming languages such as Scala, Java, or Python.

- Experience with modern data ecosystems including Hadoop, Hive, Iceberg, AWS EMR, AWS Glue, GCP Dataproc, BigQuery, or equivalent technologies.

- Experience designing and maintaining batch and streaming data pipelines.

- Strong understanding of distributed systems, cloud-native architectures, and scalability principles.

- Hands-on experience deploying and managing Spark workloads on AWS and/or GCP.

Leadership Experience

- Proven experience leading engineering teams and delivering complex technical initiatives.

- Ability to balance hands-on technical leadership with people management responsibilities.





- Experience driving cross-functional collaboration and influencing technical direction.

- Demonstrated success managing technical roadmaps, execution planning, and delivery commitments.

Scope & Positioning

- Mid-level management role positioned between Senior Engineer/Technical Lead and Senior Engineering Manager.

- Responsible for engineering execution excellence and technical leadership of Spark-based data processing platforms.

- Acts as a key decision-maker for delivery, architecture, operational reliability, and team growth.

Preferred Qualifications

- Experience with AWS services such as EMR, Glue, S3, Lambda, EKS, ECS, Step Functions, and CloudWatch.

- Experience with GCP services such as Dataproc, BigQuery, Cloud Storage, Dataflow, Pub/Sub, and Composer.

- Experience managing terabyte-to-petabyte scale data processing environments.

- Experience with Spark Structured Streaming, real-time analytics, and event-driven data processing.

- Contributions to Spark optimization, platform engineering, or open-source data ecosystem projects are a plus.

📌 Data Engineer Manager (Hyderabad)
🏢 TransUnion
📍 Hyderabad