Job Title: Senior Big Data Engineer

Experience: 7–10 Years

Primary Skills: Scala, Apache Spark, Hive, HBase, Hadoop

Role Overview

We are looking for an experienced Senior Big Data Engineer with 7–10 years of overall experience and solid hands-on expertise in Scala, Apache Spark, Hive, HBase, and Hadoop.

The candidate will be responsible for designing, developing, optimizing, and supporting large-scale data processing solutions in distributed environments. The role requires strong programming skills, deep knowledge of Spark and the Hadoop ecosystem, and proven experience handling performance, scalability, and production challenges.

Key Responsibilities

- Design, develop, and maintain scalable big data applications using Scala and Apache Spark.
- Build and enhance high-volume batch processing and ETL/data transformation pipelines.
- Develop optimized solutions using Spark SQL, DataFrames, Datasets, and RDDs where appropriate.
- Design and optimize Hive tables, partitions, bucketing, and complex HiveQL queries.
- Develop and maintain HBase-based solutions, including efficient row-key, column-family, and access-pattern designs.
- Work extensively with HDFS, YARN, Hive, HBase, and related Hadoop ecosystem components.
- Perform advanced Spark performance tuning, including partition optimization, join strategies, caching, serialization, memory management, skew handling, and shuffle reduction.
- Analyze Spark execution plans, logs, stages, tasks, and cluster utilization to identify performance bottlenecks.
- Troubleshoot complex production issues involving data pipelines, Spark jobs, Hadoop clusters, and data quality.
- Design solutions capable of processing large-scale structured, semi-structured, and unstructured datasets.
- Work with data formats such as Parquet, Avro, ORC, JSON, and CSV.
- Implement robust error handling, logging, monitoring, reconciliation, and data-quality controls.




- Participate in technical design discussions and provide engineering recommendations for scalable data solutions.
- Conduct code reviews and ensure adherence to coding, performance, security, and engineering standards.
- Mentor junior engineers and provide technical guidance to development teams.
- Collaborate with architects, business analysts, data teams, and other stakeholders to translate requirements into technical solutions.
- Support production releases, incident resolution, root-cause analysis, and continuous improvement initiatives.

Mandatory Technical Skills
- 7–10 years of overall IT/data engineering experience.
- Strong hands-on development experience with Scala.
- Extensive experience with Apache Spark and distributed data processing.
- Strong knowledge of Spark SQL, DataFrames, Datasets, and Spark internals.
- Strong experience with Hive/HiveQL, including query and storage optimization.
- Hands-on experience with HBase and NoSQL data-modeling concepts.
- Strong understanding of the Hadoop ecosystem, particularly HDFS and YARN.
- Advanced SQL skills with experience handling large datasets.
- Strong understanding of distributed computing, partitioning, parallel processing, serialization, and data locality.
- Hands-on experience in Spark performance tuning and troubleshooting.
- Experience working in Linux/Unix environments and Shell scripting.
- Experience with source control and build tools such as Git, Maven, or SBT.
- Strong debugging, analytical, and problem-solving capabilities.

Preferred Skills
- Experience with Apache Kafka and streaming/data-ingestion solutions.
- Knowledge of Spark Structured Streaming.
- Experience with workflow orchestration tools such as Airflow or Oozie.
- Exposure to AWS, Azure, or GCP data platforms.
- Experience with CI/CD pipelines and automated deployment processes.
- Knowledge of data security, governance, lineage, and data-quality practices.
- Experience with Hadoop modernization or migration of legacy big data workloads to cloud platforms.

📌 Big Data Developer (Bengaluru)
🏢 CAPCO
📍 Bengaluru