STS-DATA ENGINEERING-1

Posted 20 hours ago

Erode

Job Summary:
We are looking for motivated Data Engineers to join our team and help design, build, andmaintain scalable data pipelines and warehouse solutions. The ideal candidates will havehands-on experience with modern data engineering tools and a strong interest in real-time and batch data processing.

Responsibilities

- Design and implement data models to support analytical and reporting needs, ensuringefficient schema design for warehouse and transactional systems
- Build, maintain, and optimize ETL/ELT pipelines using Python, Apache Kafka, Apache Flink for batch and real-time data processing and PySpark
- Set up and manage data warehouse infrastructure (Doris), including table design,partitioning strategies, and performance tuning
- Develop and schedule data workflows using Apache Airflow, ensuring reliability andmonitoring of pipeline jobs
- Integrate data from multiple sources including PostgreSQL, MS SQL, and MinIO into the data warehouse
- Containerize data pipeline components using Docker for consistent deployment across environments
- Ensure data quality, consistency, and integrity across pipelines through validation and testing
- Collaborate with analysts, data scientists, and other engineering teams to understand data requirements and deliver solutions
- Troubleshoot and resolve issues related to data pipelines, storage, and processing performance
- Document data flows, schemas, and pipeline architecture for team reference

Required Qualifications





- Bachelor's degree in Computer Science, Information Technology, Engineering, or a related field
- 1 - 3 years of hands-on experience in data engineering
- Proficiency in Python and SQL for data processing and transformation
- Experience with Apache Kafka for stream processing and messaging
- Experience with Apache Flink or similar stream/batch processing frameworks
- Hands-on experience with PySpark for large-scale data processing
- Familiarity with Apache Airflow for workflow orchestration and scheduling
- Working knowledge of relational databases (PostgreSQL, MS SQL)
- Exposure to OLAP/data warehouse systems (Doris or similar columnar databases)
- Experience with Docker for containerization of data pipelines
- Familiarity with object storage solutions (MinIO or similar S3-compatible storage)
- Understanding of data warehousing concepts, data modeling, and ETL/ELT pipeline

design

Preferred Skills

- Experience building real-time data pipelines using Kafka + Flink
- Knowledge of distributed computing concepts (partitioning, parallelism, fault tolerance)
- Familiarity with CI/CD practices for deploying data pipelines
- Experience with version control (Git)
- Understanding of data quality, validation, and monitoring practices
- Basic knowledge of cloud platforms (AWS, Azure, or GCP) is a plus

Soft Skills

- Strong problem-solving and debugging skills
- Good communication and collaboration abilities
- Eagerness to learn and adapt to recent tools in a fast-paced environment

📌 Data Engineering (India)
🏢 SPAN Technology Services
📍 India