Senior Apache Spark Technical Lead - Scala, Python (Nagpur)

Senior Apache Spark Technical Lead - Scala, Python (Nagpur)

19 Nov
|
HCLTech
|
Nagpur

19 Nov

HCLTech

Nagpur

Job Summary

Role Overview

We are seeking an experienced Data Engineer to help design, build, and scale our next-generation open Data Lakehouse architecture. In this role, you will build and maintain high-throughput batch and streaming data pipelines using Python and PySpark, manage open-table transactional data with Apache Iceberg, and optimize ultra-fast interactive and federated analytics using Trino.

You will work closely with data scientists, analytics engineers, and platform teams to ensure data reliability, ACID compliance, zero data duplication, and sub-second-to-minute query performance across multiple compute engines.

Key Responsibilities

- Lakehouse Pipeline Development: Design, write, and optimize modular PySpark and Python ETL/ELT pipelines to ingest, transform, and load massive datasets into Apache Iceberg tables on cloud object storage (AWS S3, GCS, or Azure Blob).
- Iceberg Table Management & Optimization: Implement and manage Iceberg table formats (v1/v2), schema evolution, partitioning strategies (hidden partitioning), snapshot expirations, orphan file cleanups, and regular compaction/rewrite jobs (rewrite_data_files).
- Trino Query & Performance Tuning: Administer and tune Trino query execution (concurrency, dynamic filtering, memory limits, and catalog configurations) for low-latency ad-hoc analytics and BI dashboards.
- Catalog & Multi-Engine Interoperability: Manage metadata and catalog integrations (e.g., REST Catalog, AWS Glue, Apache Polaris, or Project Nessie) to guarantee seamless multi-engine read/write access across Spark, Trino, and Python/PyIceberg.
- Data Quality & Observability: Implement automated data validation, schema enforcement, lineage tracking, and monitoring (using tools like Apache Airflow, Great Expectations, or similar orchestration frameworks).
- Cost & Performance Governance: Decouple compute from storage costs, monitor query execution plans, and continuously tune partition/sort layouts to minimize I/O and storage overhead.

Required Qualifications

- Experience: 8+ years of hands-on data engineering experience building large-scale data lakes or lakehouses.
- Programming & Processing: Advanced proficiency in Python and Apache Spark / PySpark for distributed data processing.
- Table Formats: Deep practical experience with Apache Iceberg (ACID transactions, time travel, schema/partition evolution, manifest management, and compaction).
- Query Engines: Strong working knowledge of Trino (or Presto) for federated and distributed SQL analytics.
- Catalogs & Storage: Familiarity with modern Iceberg catalogs (REST Catalog, Nessie, Glue, Polaris) and cloud object storage (S3, GCS, or ADLS).
- Orchestration & DevOps: Experience with workflow orchestrators (Apache Airflow, Dagster, or Prefect) and containerization/IaC (Docker, Kubernetes, Terraform).

Preferred / Nice-to-Have Skills

- Experience with streaming frameworks (Apache Flink or Spark Streaming).
- Knowledge of PyIceberg for lightweight local Python data science workflows.
- Experience with data governance, security, and access control (e.g., Ranger, Hive/OAuth2 credential vending).

Key Responsibilities

Role Overview

We are seeking an experienced Data Engineer to help design, build, and scale our next-generation open Data Lakehouse architecture. In this role, you will build and maintain high-throughput batch and streaming data pipelines using Python and PySpark, manage open-table transactional data with Apache Iceberg, and optimize ultra-fast interactive and federated analytics using Trino.

You will work closely with data scientists, analytics engineers, and platform teams to ensure data reliability, ACID compliance, zero data duplication, and sub-second-to-minute query performance across multiple compute engines.

Key Responsibilities

- Lakehouse Pipeline Development: Design, write, and optimize modular PySpark and Python ETL/ELT pipelines to ingest, transform, and load massive datasets into Apache Iceberg tables on cloud object storage (AWS S3, GCS, or Azure Blob).
- Iceberg Table Management & Optimization: Implement and manage Iceberg table formats (v1/v2), schema evolution, partitioning strategies (hidden partitioning), snapshot expirations, orphan file cleanups, and regular compaction/rewrite jobs (rewrite_data_files).
- Trino Query & Performance Tuning:



Administer and tune Trino query execution (concurrency, dynamic filtering, memory limits, and catalog configurations) for low-latency ad-hoc analytics and BI dashboards.
- Catalog & Multi-Engine Interoperability: Manage metadata and catalog integrations (e.g., REST Catalog, AWS Glue, Apache Polaris, or Project Nessie) to guarantee seamless multi-engine read/write access across Spark, Trino, and Python/PyIceberg.
- Data Quality & Observability: Implement automated data validation, schema enforcement, lineage tracking, and monitoring (using tools like Apache Airflow, Great Expectations, or similar orchestration frameworks).
- Cost & Performance Governance: Decouple compute from storage costs, monitor query execution plans, and continuously tune partition/sort layouts to minimize I/O and storage overhead.

Required Qualifications

- Experience: 8+ years of hands-on data engineering experience building large-scale data lakes or lakehouses.
- Programming & Processing: Advanced proficiency in Python and Apache Spark / PySpark for distributed data processing.
- Table Formats: Deep practical experience with Apache Iceberg (ACID transactions, time travel, schema/partition evolution, manifest management, and compaction).
- Query Engines: Strong working knowledge of Trino (or Presto) for federated and distributed SQL analytics.
- Catalogs & Storage: Familiarity with modern Iceberg catalogs (REST Catalog, Nessie, Glue, Polaris) and cloud object storage (S3, GCS, or ADLS).
- Orchestration & DevOps: Experience with workflow orchestrators (Apache Airflow, Dagster, or Prefect) and containerization/IaC (Docker, Kubernetes, Terraform).

Preferred / Nice-to-Have Skills

- Experience with streaming frameworks (Apache Flink or Spark Streaming).
- Knowledge of PyIceberg for lightweight local Python data science workflows.
- Experience with data governance, security, and access control (e.g., Ranger, Hive/OAuth2 credential vending).

Skill Requirements

Role Overview

We are seeking an experienced Data Engineer to help design, build, and scale our next-generation open Data Lakehouse architecture. In this role, you will build and maintain high-throughput batch and streaming data pipelines using Python and PySpark, manage open-table transactional data with Apache Iceberg, and optimize ultra-fast interactive and federated analytics using Trino.

You will work closely with data scientists, analytics engineers, and platform teams to ensure data reliability, ACID compliance, zero data duplication, and sub-second-to-minute query performance across multiple compute engines.

Key Responsibilities

- Lakehouse Pipeline Development: Design, write, and optimize modular PySpark and Python ETL/ELT pipelines to ingest, transform, and load massive datasets into Apache Iceberg tables on cloud object storage (AWS S3, GCS, or Azure Blob).
- Iceberg Table Management & Optimization: Implement and manage Iceberg table formats (v1/v2), schema evolution, partitioning strategies (hidden partitioning), snapshot expirations, orphan file cleanups, and regular compaction/rewrite jobs (rewrite_data_files).
- Trino Query & Performance Tuning: Administer and tune Trino query execution (concurrency, dynamic filtering, memory limits, and catalog configurations) for low-latency ad-hoc analytics and BI dashboards.
- Catalog & Multi-Engine Interoperability: Manage metadata and catalog integrations (e.g., REST Catalog, AWS Glue, Apache Polaris, or Project Nessie) to guarantee seamless multi-engine read/write access across Spark, Trino, and Python/PyIceberg.
- Data Quality & Observability: Implement automated data validation, schema enforcement, lineage tracking, and monitoring (using tools like Apache Airflow, Great Expectations, or similar orchestration frameworks).
- Cost & Performance Governance: Decouple compute from storage costs, monitor query execution plans, and continuously tune partition/sort layouts to minimize I/O and storage overhead.

Required Qualifications





- Experience: 8+ years of hands-on data engineering experience building large-scale data lakes or lakehouses.
- Programming & Processing: Advanced proficiency in Python and Apache Spark / PySpark for distributed data processing.
- Table Formats: Deep practical experience with Apache Iceberg (ACID transactions, time travel, schema/partition evolution, manifest management, and compaction).
- Query Engines: Strong working knowledge of Trino (or Presto) for federated and distributed SQL analytics.
- Catalogs & Storage: Familiarity with modern Iceberg catalogs (REST Catalog, Nessie, Glue, Polaris) and cloud object storage (S3, GCS, or ADLS).
- Orchestration & DevOps: Experience with workflow orchestrators (Apache Airflow, Dagster, or Prefect) and containerization/IaC (Docker, Kubernetes, Terraform).

Preferred / Nice-to-Have Skills

- Experience with streaming frameworks (Apache Flink or Spark Streaming).
- Knowledge of PyIceberg for lightweight local Python data science workflows.
- Experience with data governance, security, and access control (e.g., Ranger, Hive/OAuth2 credential vending).

Must Have Skill

Python,Scala,Apache Spark

Positive to Have Skill

Other Requirements

Role Overview

We are seeking an experienced Data Engineer to help design, build, and scale our next-generation open Data Lakehouse architecture. In this role, you will build and maintain high-throughput batch and streaming data pipelines using Python and PySpark, manage open-table transactional data with Apache Iceberg, and optimize ultra-fast interactive and federated analytics using Trino.

You will work closely with data scientists, analytics engineers, and platform teams to ensure data reliability, ACID compliance, zero data duplication, and sub-second-to-minute query performance across multiple compute engines.

Key Responsibilities

- Lakehouse Pipeline Development: Design, write, and optimize modular PySpark and Python ETL/ELT pipelines to ingest, transform, and load massive datasets into Apache Iceberg tables on cloud object storage (AWS S3, GCS, or Azure Blob).
- Iceberg Table Management & Optimization: Implement and manage Iceberg table formats (v1/v2), schema evolution, partitioning strategies (hidden partitioning), snapshot expirations, orphan file cleanups, and regular compaction/rewrite jobs (rewrite_data_files).
- Trino Query & Performance Tuning: Administer and tune Trino query execution (concurrency, dynamic filtering, memory limits, and catalog configurations) for low-latency ad-hoc analytics and BI dashboards.
- Catalog & Multi-Engine Interoperability: Manage metadata and catalog integrations (e.g., REST Catalog, AWS Glue, Apache Polaris, or Project Nessie) to guarantee seamless multi-engine read/write access across Spark, Trino, and Python/PyIceberg.
- Data Quality & Observability: Implement automated data validation, schema enforcement, lineage tracking, and monitoring (using tools like Apache Airflow, Great Expectations, or similar orchestration frameworks).
- Cost & Performance Governance: Decouple compute from storage costs, monitor query execution plans, and continuously tune partition/sort layouts to minimize I/O and storage overhead.

Required Qualifications

- Experience: 8+ years of hands-on data engineering experience building large-scale data lakes or lakehouses.
- Programming & Processing: Advanced proficiency in Python and Apache Spark / PySpark for distributed data processing.
- Table Formats: Deep practical experience with Apache Iceberg (ACID transactions, time travel, schema/partition evolution, manifest management, and compaction).
- Query Engines: Strong working knowledge of Trino (or Presto) for federated and distributed SQL analytics.
- Catalogs & Storage: Familiarity with modern Iceberg catalogs (REST Catalog, Nessie, Glue, Polaris) and cloud object storage (S3, GCS, or ADLS).
- Orchestration & DevOps: Experience with workflow orchestrators (Apache Airflow, Dagster, or Prefect) and containerization/IaC (Docker, Kubernetes, Terraform).

Preferred / Nice-to-Have Skills

- Experience with streaming frameworks (Apache Flink or Spark Streaming).
- Knowledge of PyIceberg for lightweight local Python data science workflows.
- Experience with data governance, security, and access control (e.g., Ranger, Hive/OAuth2 credential vending).

📌 Senior Apache Spark Technical Lead - Scala, Python (Nagpur)
🏢 HCLTech
📍 Nagpur

Reply to this offer

Impress this employer describing Your skills and abilities, fill out the form below and leave Your personal touch in the presentation letter.

Subscribe to this job alert:

Get the latest job offers by email for: senior apache spark technical lead - scala, python (nagpur) / nagpur

Subscribe to this job alert:

Get the latest job offers by email for: senior apache spark technical lead - scala, python (nagpur) / nagpur