Key Responsibilities

Lead the design and architecture of scalable big data solutions using Apache Spark and Hadoop ecosystem technologies.

Own end-to-end development of batch and streaming data pipelines.

Design, optimize, and tune Spark applications for performance, scalability, and reliability.

Provide technical leadership for Hadoop components including HDFS, Hive, HBase, and YARN.

Ensure data quality, data governance, security, and compliance standards are met.

Lead real-time streaming solutions using Spark Structured Streaming and Kafka.

Review code, conduct technical design reviews, and enforce engineering best practices.

Collaborate with architects, product owners, and business stakeholders to translate requirements into technical solutions.

Mentor and guide data engineers and developers.

Support production deployments, monitoring, and incident resolution.

Drive continuous improvement, automation, and adoption of current big data technologies.

Required Technical Skills

Robust experience with Apache Spark (Spark SQL, DataFrames, Datasets, Structured Streaming).





Hands-on experience with Hadoop ecosystem tools (HDFS, Hive, HBase, YARN).

Advanced SQL and data modeling knowledge.

Experience with data formats such as Parquet, ORC, and Avro.

Knowledge of distributed systems and performance tuning.

Experience with Kafka or similar messaging platforms.

Robust programming skills in Python, Scala, or Java.

Experience with CI/CD pipelines and DevOps practices.

Cloud & Platform Experience

Experience with cloud-based big data platforms such as AWS EMR/Glue, Azure Databricks/Synapse, or GCP Dataproc.

Experience migrating Hadoop workloads to cloud-based platforms.

Understanding of cloud security, cost optimization, and scalability best practices.

Disclaimer: This job posting has been aggregated from external source. Role details, content, and availability are subject to change. Applicants are advised to confirm the latest information directly on the company website before applying.

📌 Lead Ii Software Engineering Maharashtra
🏢 UST
📍 Maharashtra