Role description:
- Key Responsibilities

Lead the design and architecture of scalable big data solutions using Apache Spark and Hadoop ecosystem technologies.

Own end-to-end development of batch and streaming data pipelines.

Provide technical leadership for Hadoop components including HDFS, Hive, HBase, and YARN.

Ensure data quality, data governance, security, and compliance standards are met.

Lead real-time streaming solutions using Spark Structured Streaming and Kafka.

Review code, conduct technical design reviews, and enforce engineering best practices.

Collaborate with architects, product owners, and business stakeholders to translate requirements into technical solutions.

Mentor and guide data engineers and developers.

Support production deployments, monitoring, and incident resolution.

Drive continuous improvement, automation, and adoption of new big data technologies.

Required Technical Skills

Strong experience with Apache Spark (Spark SQL, DataFrames, Datasets, Structured Streaming).

Hands-on experience with Hadoop ecosystem tools (HDFS, Hive, HBase, YARN).





Advanced SQL and data modeling knowledge.

Experience with data formats such as Parquet, ORC, and Avro.

Knowledge of distributed systems and performance tuning.

Experience with Kafka or similar messaging platforms.

Robust programming skills in Python, Scala, or Java.

Experience with CI/CD pipelines and DevOps practices.

Cloud & Platform Experience

Experience with cloud-based big data platforms such as AWS EMR/Glue, Azure Databricks/Synapse, or GCP Dataproc.

Experience migrating Hadoop workloads to cloud-based platforms.

Understanding of cloud security, cost optimization, and scalability best practices.
Skills

spark,cloud computing,data quality management,databricks,
About UST

UST is a global digital transformation solutions provider. For more than 20 years, UST has worked side by side with the world’s best companies to make a real impact through transformation. Powered by technolog

📌 Lead Ii (Mumbai)
🏢 UST
📍 Mumbai