- Develop and maintain data pipelines using Apache Spark and Scala to process large volumes of data efficiently.
- Implement data transformation logic to cleanse, normalize, and aggregate data from various sources, ensuring high data quality and consistency.
- Integrate data from different systems and sources, including structured and unstructured data, databases, and Mainframe systems, ensuring seamless data flow and compatibility.
- Implement data validation and quality checks within data pipelines to ensure data accuracy, consistency, and integrity, and handle exception handling and error logging.
- Develop and execute unit tests, integration tests, and end-to-end tests to ensure the reliability and correctness of Spark applications.
- Debug and troubleshoot issues during development and production phases.
📌 Data Enginner (India)
🏢 Infosys
📍 India
Reply to this offer
Impress this employer describing Your skills and abilities, fill out the form below and leave Your personal touch in the presentation letter.