- Design and implement data pipelines to cleanse and transform data in preparation for feeding to AI/Forecasting, Price Optimization, and related Supply Chain SaaS softwareproducts.
- Collaborate closely with Data Scientists to support feature engineering and operationalize AI/ML and mathematical optimization models.
- Develop efficient and robust data transformation code in PySpark.
- Build and maintain workflows to automate data pipelines using tools such as Databricks and Airflow.
- Optimize performance of data pipelines to ensure minimal cost.
- Develop data validation tests to assess the quality of the input data and recommend remediation measures.
- Conduct performance testing and profiling of the code using a variety of tools and techniques.
- Create and maintain comprehensive technical documentation for data pipelines and architecture.
Qualifications:
Job Requirements:
- Minimum Education: Bachelor's degree in computer science, Engineering, or a related field (or equivalent work experience).
• Minimum Work Experience (years):
o 3 - 6 years of general Data Engineering experience.
o Within that timeframe, experience is expected in: Python/PySpark, SQL, and relational or NoSQL databases.
• Key Skills and Competencies:
o Experience working with AWS, Azure, or GCP cloud environments.
o Experience building end-to-end data pipelines using PySpark or SQL. Strong understanding of parallel processing, clusters and compute management in the cloud.
o Experience with data orchestration tools like Databricks, Airflow (or similar tools like Snowflake, Dagster, etc.).
o Excellent verbal and written communication skills, especially in technical communications.
o Demonstrated ability to learn recent technologies quickly and independently.
o Preferred (but not required): Familiarity with Data Science concepts (e.g., Machine Learning) and a desire to grow into hybrid engineering/data science tasks.
o Ability to work and achieve stretch goals in a very innovative and fast-paced environment.
• Licenses/Certifications, special qualifications: N/A
📌 Senior Data Engineer (Pune)
🏢 Kairos Technologies Private
📍 Pune
Reply to this offer
Impress this employer describing Your skills and abilities, fill out the form below and leave Your personal touch in the presentation letter.