Staff Data Engineer - Product Engagement(Data Platform Team) Hyderabad

About the Role

We are seeking a Staff Data Engineer to serve as a top technical leader for the Product Engagement Data Engineering team. You will set the technical direction for data systems that process 30+ billion events per day across streaming and batch workloads, and own the architecture of the medallion (bronze, silver, gold) pipelines for sessionization, pathing, user-journey attribution, component performance, and search that produce our canonical engagement datasets.

This is a hands-on technical leadership role. You will make the highest-leverage architectural decisions, raise the engineering bar through standards and mentorship, and drive cross-team initiatives that span our shared Scala and Spark code library and our tenant-specific workflows, configurations, and DDLs. You will partner closely with analytics, data science, and platform teams to ensure the engagement data layer is scalable, reliable, cost-effective, and trustworthy at petabyte scale.

Key Responsibilities

- Own the architecture of the multi-tenant product engagement data platform end to end, from streaming event ingestion and normalization through silver sessionization and the gold attribution, journey, and component-performance layers that downstream analytics depend on.

- Set technical direction and standards for high-performance Spark applications, partitioning and data-modeling conventions, idempotent backfills, schema evolution, and release management across a shared library and multiple tenant pipelines.

- Lead the design and delivery of streaming and batch processing in Scala, Spark, and SQL on Databricks and AWS, optimizing for performance, cost, and reliability at scale.

- Drive engineering excellence in data quality and observability, including monitoring, alerting, validation, and lineage frameworks that protect the integrity of business-critical engagement metrics.





- Resolve the hardest technical problems, including distributed-systems failure modes, session split and late-arrival handling, skew and shuffle optimization, cross-catalog dependencies, and correctness of attribution logic.

- Align engineering, analytics, data science, and platform stakeholders on shared contracts such as event schemas, gold-table grains, and semantic-layer mappings, and shepherd new metrics from requirement to reporting surface.

- Mentor and grow senior and mid-level engineers through design reviews, code review, pairing, and documentation, and build a culture of rigor, ownership, and continuous improvement.

- Evaluate emerging tools and patterns, manage technical debt deliberately, and contribute to the multi-year roadmap for the engagement data platform.

What Youll Bring

- Experience: 10+ years in data engineering or related software engineering, including a track record as a technical leader on large-scale data platforms.

- Programming: Expert-level proficiency in Scala (strongly preferred) and/or Python, with a strong software-engineering foundation including testing, modularity, versioned shared libraries, and CI/CD.

- Distributed computing: Deep expertise with Apache Spark (structured streaming and batch), including performance tuning, skew and shuffle optimization, and operating at petabyte scale.

- Lakehouse and storage: Advanced experience with Delta Lake and lakehouse or medallion architectures, including schema evolution, partitioning strategy, and idempotent and incremental processing.

- Data platforms: Hands-on experience with Databricks (jobs,



workflows, clusters) and AWS (S3 and related services).

- SQL: Advanced SQL for large-scale transformation, validation, and analytical workloads.

- DevOps: Proficiency with CI/CD (GitHub Actions), artifact and version management, and modern build tooling such as sbt.

- Architecture and leadership: Demonstrated ownership of architectural decisions for scaling and reliability, plus a record of mentoring engineers and driving engineering best practices.

Preferred Qualifications

- Background in streaming media, entertainment, or high-traffic consumer applications.

- Experience building clickstream or event-based analytics such as sessionization, pathing, user-journey modeling, or attribution at scale.

- Familiarity with multi-tenant data platforms and semantic layers such as Looker and LookML, and the contracts between engineering and reporting.

- Experience with orchestration and workflow tooling such as Databricks Workflows and Airflow, and infrastructure-as-code practices.

- Familiarity with data quality frameworks such as Deequ, and with data governance, lineage, and catalog tooling.

- Experience designing and executing large historical backfills with validation and abort criteria.

- Exposure to supporting ML and data science consumers such as feature data and experimentation datasets.

Technical Environment

- Data volume: 30+ billion events processed daily.

- Infrastructure: Databricks on AWS (S3, Delta Lake).

- Primary languages: Scala, Python, SQL.

- Processing: Apache Spark (structured streaming and batch) on a medallion bronze, silver, and gold architecture with real-time and batch layers.

Disclaimer : This job posting has been aggregated from external source. Role details, content, and availability are subject to change. Applicants are advised to confirm the latest information directly on the company website before applying.

📌 Staff Data Engineer - Product Engagement(Data Platform Team) (Hyderabad)
🏢 Warner Bros. Discovery
📍 Hyderabad