Candidates ready to join immediately can share their details via email for quick processing.

? CCTC | ECTC | Notice Period | Location Preference [email protected]

Act fast for immediate attention! ⏳?

Key Responsibilities

- Design, develop, and maintain scalable data pipelines using PySpark
- Build and manage batch and real-time data processing systems
- Develop and integrate Kafka-based streaming solutions
- Optimize Spark jobs for performance, cost, and scalability
- Work with cloud-native services to deploy and manage data solutions
- Ensure data quality, reliability, and security across platforms
- Collaborate with data scientists, analysts, and application teams
- Participate in code reviews, design discussions, and production support

Must-Have Skills

- Strong hands-on experience with PySpark / Apache Spark
- Solid understanding of distributed data processing concepts
- Experience with Apache Kafka (producers, consumers, topics, partitions)
- Hands-on experience with any one cloud platform :
- AWS (S3, EMR, Glue, EC2, IAM) or




- Azure (ADLS, Synapse, Databricks) or
- GCP (GCS, Dataproc, BigQuery)
- Proficiency in Python
- Robust experience with SQL and data modeling
- Experience working with large-scale datasets
- Familiarity with Linux/Unix environments
- Understanding of ETL/ELT frameworks
- Experience with CI/CD pipelines for data applications

Good-to-Have Skills

- Experience with Spark Structured Streaming
- Knowledge of Kafka Connect and Kafka Streams
- Exposure to Databricks
- Experience with NoSQL databases (Cassandra, MongoDB, HBase)
- Familiarity with workflow orchestration tools (Airflow, Oozie)
- Knowledge of containerization (Docker, Kubernetes)
- Experience with data lake architectures
- Understanding of security, governance, and compliance in cloud environments
- Exposure to Scala or Java is a plus
- Prior experience in Agile/Scrum environments

📌 Senior Data Engineer – PySpark, Cloud & Kafka - 5+ YoE - Immediate Joiner - Any UST Location (Bengaluru)
🏢 UST
📍 Bengaluru