Key Responsibilities:

- Design and develop scalable batch and real-time data pipelines using Spark
- Build and maintain distributed streaming systems using Kafka
- Develop ETL/ELT workflows for large-scale structured and unstructured datasets
- Optimize data processing jobs for performance
- Implement data ingestion frameworks for high-volume data sources
- Ensure data quality, integrity, and reliability across pipelines
- Collaborate with cross-functional teams to define data architecture and solutions
- Monitor, troubleshoot, and resolve data pipeline issues
- Implement data security and governance standards

Required Skills & Qualifications

- Bachelors/Master’s degree in Computer Science, Engineering, or related field
- 4+ years of experience in data engineering or big data development
- Solid hands-on experience with Apache Spark (Core, SQL, Structured Streaming)
- Experience with Apache Kafka, Kafka Streams, and connectors

- Proficiency in Python
- Experience with distributed data processing and parallel computing




- Strong knowledge of data structures and algorithms
- Experience with ETL tools and workflow orchestration
- Familiarity with data lake and data warehouse architectures
- Experience with SQL and NoSQL databases

Preferred Qualifications

- Knowledge of modern data formats (Parquet, Avro, ORC)
- Experience with CI/CD pipelines and DevOps practices
- Familiarity with data governance and metadata management tools
- Real-time analytics and event-driven architecture experience
- Experience working in Agile/Scrum environments

Key Competencies

- Strong problem-solving and analytical skills
- Excellent communication and teamwork abilities
- Ability to work with large, complex datasets

- Strong attention to detail
- Self-motivated with ability to work independently

Nice to Have

- Experience with machine learning pipelines
- Knowledge of data visualization tools
- Certification in big data technologies

📌 Data Analyst (Mumbai)
🏢 Dream Solution Indore
📍 Mumbai