Job Title: Senior Big Data Engineer
Experience: 7–10 Years
Primary Skills: Scala, Apache Spark, Hive, HBase, Hadoop

Role Overview
We are looking for an experienced Senior Big Data Engineer with 7–10 years of overall experience and strong hands-on expertise in Scala, Apache Spark, Hive, HBase, and Hadoop.
The candidate will be responsible for designing, developing, optimizing, and supporting large-scale data processing solutions in distributed environments. The role requires strong programming skills, deep knowledge of Spark and the Hadoop ecosystem, and proven experience handling performance, scalability, and production challenges.
Key Responsibilities
• Design, develop, and maintain scalable big data applications using Scala and Apache Spark.
• Build and enhance high-volume batch processing and ETL/data transformation pipelines.
• Develop optimized solutions using Spark SQL, DataFrames, Datasets, and RDDs where appropriate.
• Design and optimize Hive tables, partitions, bucketing, and complex HiveQL queries.
• Develop and maintain HBase-based solutions,



including effective row-key, column-family, and access-pattern designs.
• Work extensively with HDFS, YARN, Hive, HBase, and related Hadoop ecosystem components.
• Perform advanced Spark performance tuning, including partition optimization, join strategies, caching, serialization, memory management, skew handling, and shuffle reduction.
• Analyze Spark execution plans, logs, stages, tasks, and cluster utilization to identify performance bottlenecks.
• Troubleshoot complex production issues involving data pipelines, Spark jobs, Hadoop clusters, and data quality.
• Design solutions capable of processing large-scale structured, semi-structured, and unstructured datasets.
• Work with data formats such as Parquet, Avro, ORC, JSON, and CSV.
• Implement robust error handling, logging, monitoring, reconciliation, and data-quality controls.
• Participate in technical design discussions and provide engineering recommendation

📌 Big Data Developer (Bengaluru)
🏢 CAPCO
📍 Bengaluru