Role- Lead Data Platform Engineer

Experience- 8 to 15 years

Location- Pune, Hyderabad, Bangalore

Required Skill set- Spark, Kafka, Flink, Airflow, Iceberg, Polaris/Nessie, Kubernetes

Required Skills & Experience

Must Have

- 8-15 years of data engineering or data platform experience, with substantial recent hands-on responsibility for distributed processing platforms.
- Deep hands-on experience with Apache Spark, including DataFrame or SQL workloads, performance tuning, packaging, configuration, and production troubleshooting.
- Strong Apache Kafka experience covering topics, partitions, consumer groups, offsets, delivery semantics, schemas, security, and operations.
- Practical Apache Flink experience covering state, checkpoints, savepoints, event time, recovery, and backpressure.
- Robust understanding of Apache Iceberg, Google Cloud Storage, schema and partition evolution, snapshots, compaction, and table maintenance.
- Experience with Apache Polaris, Project Nessie, or REST-based Iceberg catalogs.
- Hands-on experience with Apache Airflow for distributed data-processing orchestration.




- Working knowledge of GKE-based execution, containers, Helm, service accounts, Workload Identity Federation, secrets, networking, storage, and observability.
- Proficiency in Python, Java, or Scala, SQL, Git, automated testing, CI/CD, and reusable software engineering practices.
- Strong technical leadership, troubleshooting, documentation, mentoring, and stakeholder communication skills.

Good to Have
- Experience with Trino, OpenMetadata, JupyterHub, Dataproc, data-quality frameworks, lineage, or schema registries.
- Exposure to Debezium, Kafka Connect, Datastream, database logs, or comparable CDC patterns.
- Experience with Spark Operator, Flink Kubernetes Operator, Strimzi, KubernetesExecutor, or operator-based GKE integration.
- Understanding of data security, encryption, masking, governance, data contracts, and regulated enterprise controls.

Experience with multi-tenant data platforms, regulated environments, and platform cost optimisation on GCP.

📌 Lead Data Platform Engineer (Hyderabad)
🏢 Tata Consultancy Services
📍 Hyderabad