18 Nov
|
SWAI TECHNOLOGIES PRIVATE
|
Bengaluru
18 Nov
SWAI TECHNOLOGIES PRIVATE
Bengaluru
Mandatory Skill Set:- Strong hands-on expertise in Apache Spark and Scala for building and optimizing large-scale data pipelines.- Experience working with high-volume batch and streaming workloads across multiple data sources.- Strong experience with Hive and distributed data processing.- Hands-on experience with Kafka and event/streaming-based architectures.- Robust programming experience in Scala and/or Python.- Experience designing, developing and troubleshooting large-scale ETL/data pipelines.- Strong understanding of data modelling, data quality, performance tuning and distributed systems.- Experience working with at least one major cloud platform - GCP, AWS or Azure.- Experience with BigQuery and GCP services is preferred.- Experience with ETL workflows and orchestration tools such as Oozie or equivalent.- Hands-on experience with Git, Jenkins and CI/CD practices, including automated build, testing and deployment pipelines.- Ability to independently troubleshoot complex data, performance and production issues.Preferred / Added Advantage:- Strong Google Cloud Platform (GCP) experience.- Exposure to BigQuery, Google Cloud Storage (GCS),
Dataproc and Pub/Sub.- Experience migrating on-premise Spark/Hive workloads to cloud platforms.- Experience with large-scale data migration and reconciliation.- Knowledge of data quality and observability frameworks.- Experience working with multiple upstream and downstream integrations.- Exposure to multiple cloud environments.What would you do here:- Design, develop and optimize large-scale data pipelines using Apache Spark and Scala.- Work with high-volume batch and streaming workloads across multiple data sources.- Develop and support data processing solutions using Hive, Kafka and GCP services.- Troubleshoot complex data, performance and production issues with minimal supervision.- Contribute to cloud migration and platform modernization initiatives.- Build and maintain ETL workflows and data orchestration pipelines.- Perform performance tuning and optimization of Spark jobs and large datasets.- Develop reusable, maintainable and production-ready data solutions.- Implement and maintain CI/CD pipelines and engineering automation.- Collaborate with engineering, product, analytics and platform teams to deliver end-to-end solutions.- Participate in code reviews, testing, production deployments and operational support.- Quickly understand existing systems and take ownership of assigned deliverables. (ref:hirist.tech)
📌 Data Engineer - Scala/GCP (Bengaluru)
🏢 SWAI TECHNOLOGIES PRIVATE
📍 Bengaluru