Data Engineer :

Job Description :
- Develop and maintain data pipelines, ELT processes, and workflow orchestration using Apache Airflow, Python and PySpark to ensure the efficient and reliable delivery of data.
- Design and implement custom connectors to facilitate the ingestion of diverse data sources into our platform, including structured and unstructured data from various document formats .
- Collaborate closely with cross-functional teams to gather requirements, understand data needs, and translate them into technical solutions.
- Implement DataOps principles and best practices to ensure robust data operations and efficient data delivery.
- Design and implement data CI/CD pipelines to enable automated and productive data integration, transformation, and deployment processes.
- Monitor and troubleshoot data pipelines, proactively identifying and resolving issues related to data ingestion, transformation, and loading.
- Conduct data validation and testing to ensure the accuracy, consistency, and compliance of data.




- Stay up-to-date with emerging technologies and best practices in data engineering.
- Document data workflows, processes, and technical specifications to facilitate knowledge sharing and ensure data governance.

Responsibilities:
- Bachelors degree in Computer Science, Engineering, or a related field
- 5 + years experience in data engineering, ELT development, and data modeling.
- Proficiency in using Apache Airflow and Spark for data transformation, data integration, and data management.
- Experience implementing workflow orchestration using tools like Apache Airflow, SSIS or similar platforms.
- Demonstrated experience in developing custom connectors for data ingestion from various sources.
- Strong understanding of SQL and database concepts, with the ability to write efficient queries and optimize performance.
- Experience implementing DataOps principles and practices, including data CI/CD pipelines.
- Excellent problem-solving and troubleshooting

📌 Data Engineer (Bengaluru)
🏢 Photon
📍 Bengaluru