Lead data engineering strategy across multiple projects and teams. Drive innovation in data engineering practices and model optimization. Represent the data engineering team in executive meetings. Develop long-term plans for data engineering and data infrastructure.

Responsibilities
Key Responsibilities

• Ability to design, build and unit test applications on Spark framework on Python.

• Build Python and PySpark based applications based on data in both Relational (Oracle, SQL Server) and NoSQL database e.g. Redis, Valkey

• Build ingestion pipelines to load data into in-memory caching database.

• Build data pipelines to load and export data to/from the Master Data Management or Reference Data Management Platform using PySpark

• Hands on experience doing at least one MDM implementation using TIBCO, Informatica, GoldenSource or custom-built MDM/RDM platforms

• Experience with designing/building Data APIs and its interaction with data consumers

• Optimize performance of the built Spark applications using configurations around Spark Context, Spark-SQL and Data Frame.





• Build Python programs using module libs e.g. pandas, requests, json, flask, pickle

• Experience in processing large amounts of structured data, including integrating data from multiple sources.

• Ability to design & build real-time applications using REST API, JSON, XML

• Ability to build solutions on AWS services using Glue ETL, Lambda functions on Python

• Create and maintain integration and regression testing framework on Jenkins integrated with BitBucket and/or GIT repositories

• Participate in the agile development process, and document and communicate issues and bugs relative to data standards in scrum meetings

• Work collaboratively with onsite and offshore team.

• Develop & review technical documentation for artifacts delivered.

• Ability to solve complex data-driven scenarios and triage towards defects and production issues

• Participate in code release and production deployment.

📌 Manager (Chennai)
🏢 EXL
📍 Chennai