[email protected]

JD - Big Data Engineer Key Responsibilities Design, develop, and optimize scalable data pipelines for batch and real-time processing. Build and maintain ETL/ELT workflows using up-to-date data engineering frameworks. Process and analyze large datasets using distributed computing technologies.

Develop and optimize Spark applications for high-volume data processing. Build streaming data solutions using Kafka and related technologies. Implement workflow orchestration and scheduling using Airflow.

Work with data storage systems including HDFS, S3, Hive, and NoSQL databases. Ensure data quality, reliability, and observability across data platforms. Collaborate with cross-functional teams to understand business requirements and translate them into technical solutions.





Monitor and troubleshoot production data pipelines and platform issues. Implement CI/CD practices and automation for data engineering workflows. Contribute to platform modernization initiatives involving containerization and cloud-native technologies.

Required Technical Skills Programming Python

Scala SQL Java is a plus Big Data Technologies Apache Spark (Core, SQL, Structured Streaming) Apache Hive Apache Kafka Apache NiFi Apache Airflow Hadoop Ecosystem (HDFS, YARN) Aerospike is a plus Kibana Grafana Elasticsearch DevOps & Platform Tools Git Jenkins Linux Bitbucket Jira Confluence Kubernetes Podman

📌 Big Data Engineer (Pune)
🏢 Athena Consultancy Services
📍 Pune