Experience: 6-10 Years
Location: Pan India
Interview Mode: Virtual
We are looking for an experienced Python & Kafka Data Engineer to design, develop, and maintain scalable data pipelines and real-time streaming solutions. The ideal candidate should have strong expertise in Python, Apache Kafka, Data Engineering, ETL processes, and cloud technologies.
Key Responsibilities
• Design, develop, and optimize batch and real-time data pipelines.
• Build scalable streaming applications using Apache Kafka.
• Develop data processing frameworks and ETL workflows using Python.
• Integrate data from multiple sources, including APIs, databases, and cloud platforms.
• Ensure data quality, performance, reliability, and security.
• Work closely with Data Scientists, Analysts, and Business teams to deliver data solutions.
• Monitor and troubleshoot production data pipelines.
• Implement best practices for data governance and data management.
Required Skills
• Robust hands-on experience in Python Development.
• Expertise in Apache Kafka, Kafka Streams, and Kafka Connect.
• Experience with PySpark/Spark and Big Data technologies.
• Strong knowledge of SQL and database systems.
• Experience in building ETL/ELT pipelines.
• Hands-on experience with AWS/Azure/GCP cloud platforms.
• Knowledge of REST APIs and Microservices.
• Experience with Git, CI/CD tools, and Agile methodologies.
• Strong analytical and problem-solving skills.