Role & responsibilities

- Design, develop, and maintain scalable big data pipelines for batch and real-time data processing.
- Build and optimize data processing solutions using Apache Spark, PySpark, Kafka, Hadoop, and Hive.
- Develop robust ETL/ELT pipelines and integrate data from multiple sources.
- Design and maintain data lakes, data warehouses, and distributed data platforms.
- Optimize Spark jobs, SQL queries, storage, and compute resources for performance and cost efficiency.
- Implement real-time/streaming data pipelines using Kafka and Spark Streaming/Structured Streaming.
- Work with cloud platforms such as AWS, Azure, or GCP and cloud-native data services.
- Develop data models and ensure data quality, reliability, security, and governance.
- Automate workflows and data pipelines using Apache Airflow and CI/CD practices.
- Troubleshoot production issues, perform root-cause analysis, and ensure high availability of data systems.
- Collaborate with data scientists, analysts, software engineers, architects, and business stakeholders.
- Participate in technical design, code reviews, documentation, and mentoring of junior engineers.

Preferred candidate profile

- 5 to 10 years of experience in Big Data Engineering,



Data Engineering, or a related role.
- Strong hands-on expertise in Python/PySpark, SQL, and Apache Spark.
- Experience with Kafka, Hadoop, Hive, Airflow, and distributed data processing.
- Strong understanding of ETL/ELT, data modeling, data warehousing, data lakes, and data pipelines.
- Professional experience with at least one major cloud platform: AWS, Azure, or GCP.
- Experience with technologies such as Databricks, Snowflake, AWS S3/EMR/Glue, Redshift, or Azure Data Factory is advantageous.
- Robust understanding of distributed systems, performance optimization, scalability, and fault tolerance.
- Experience developing batch and real-time data processing solutions.
- Proficiency with Git, Linux, CI/CD, Docker, and Kubernetes is preferred.
- Strong problem-solving, analytical, communication, and collaboration skills.
- Bachelor's degree in Computer Science, Information Technology, Engineering, or a related technical discipline; postgraduate qualifications are a plus.
- Experience working in large-scale production data environments and handling high-volume datasets.

📌 Senior Big Data Engineer (India)
🏢 Good Co India
📍 India