We are looking for an experienced Python & Kafka Data Engineer to design, develop, and maintain scalable data pipelines and real-time streaming solutions. The ideal candidate should have strong expertise in Python, Apache Kafka, Data Engineering, ETL processes, and cloud technologies.
Key Responsibilities
- Design, develop, and optimize batch and real-time data pipelines.
- Build scalable streaming applications using Apache Kafka.
- Develop data processing frameworks and ETL workflows using Python.
- Integrate data from multiple sources, including APIs, databases, and cloud platforms.
- Ensure data quality, performance, reliability, and security.
- Work closely with Data Scientists, Analysts, and Business teams to deliver data solutions.
- Monitor and troubleshoot production data pipelines.
- Implement best practices for data governance and data management.
Required Skills
- Robust hands-on experience in Python Development.
- Expertise in Apache Kafka, Kafka Streams, and Kafka Connect.
- Experience with PySpark/Spark and Big Data technologies.
- Strong knowledge of SQL and database systems.
- Experience in building ETL/ELT pipelines.
- Hands-on experience with AWS/Azure/GCP cloud platforms.
- Knowledge of REST APIs and Microservices.
- Experience with Git, CI/CD tools, and Agile methodologies.
- Strong analytical and problem-solving skills.