STS-DATA ENGINEERING-1

Posted a few seconds ago

Erode

Job Summary:
Responsibilities
- Design and implement data models to support analytical and reporting needs, ensuringefficient schema design for warehouse and transactional systems
- Build, maintain, and optimize ETL/ELT pipelines using Python, Apache Kafka, Apache Flink for batch and real-time data processing and PySpark
- Set up and manage data warehouse infrastructure (Doris), including table design,partitioning strategies, and performance tuning
- Develop and schedule data workflows using Apache Airflow, ensuring reliability andmonitoring of pipeline jobs
- Integrate data from multiple sources including PostgreSQL, MS SQL, and MinIO into the data warehouse
- Containerize data pipeline components using Docker for consistent deployment across environments
- Ensure data quality, consistency, and integrity across pipelines through validation and testing
- Collaborate with analysts, data scientists, and other engineering teams to understand data requirements and deliver solutions




- Troubleshoot and resolve issues related to data pipelines, storage, and processing performance
- Document data flows, schemas, and pipeline architecture for team reference

Required Qualifications
- Bachelor's degree in Computer Science, Information Technology, Engineering, or a related field
- 1 - 3 years of hands-on experience in data engineering
- Proficiency in Python and SQL for data processing and transformation
- Experience with Apache Kafka for stream processing and messaging
- Experience with Apache Flink or similar stream/batch processing frameworks
- Hands-on experience with PySpark for large-scale data processing
- Familiarity with Apache Airflow for workflow orchestration and scheduling
- Working knowledge of relational databases (PostgreSQL, MS SQL)
- Exposure to OLAP/data warehouse systems (Doris or similar columnar databases)
- Experience with Docker for containerization of data pipelines
- Familiarity with object stor

📌 Data Engineering (Erode)
🏢 SPAN Technology Services
📍 Erode