Job Summary

Design, build and optimise robust data ingestion pipelines to acquire, transform and load data. Develop scalable data engineering solutions using Databricks, PySpark, SparkSQL and associated cloud technologies. Build and maintain secure, reliable and automated data ingestion processes from external vendor systems, including SFTP-based file transfers and other integration methods. Ensure data is landed, structured, governed and accessible to support reporting, analytics and business use cases.

Responsibilities

- Work with Business Analysts, Product Owners, Architects and delivery squads to translate business requirements into technical data solutions.
- Support data discovery, profiling and validation activities to understand source data structures, data quality issues and data gaps.
- Develop and maintain data transformations, curated datasets and data models required to support reporting and analytical use cases.
- Monitor, troubleshoot and resolve data ingestion issues, defects and enhancements identified during development, testing, UAT and production support.
- Ensure solutions comply with data architecture standards, engineering best practices, security requirements and governance frameworks.
- Produce clear technical documentation for data ingestion processes, data flows and operational support requirements.
- Provide comprehensive handover documentation and knowledge transfer to the Data Support team following delivery of data ingestion pipelines.
- Collaborate within Agile delivery teams, actively contributing to sprint planning, stand-ups, retrospectives and continuous improvement activities.




- Identify opportunities to improve pipeline performance, automation, scalability and maintainability through process and technology enhancements.
- Support knowledge sharing and contribute to Engineering and Data Communities of Practice.

Requirements

- Essential Proven experience designing, building and supporting enterprise-scale data ingestion pipelines and ETL/ELT solutions.
- Essential Strong hands-on experience with Databricks, PySpark and SparkSQL.
- Essential Experience developing and supporting secure data integrations using SFTP and other file-based or API-driven ingestion mechanisms.
- Essential Experience ingesting and processing structured, semi-structured and unstructured data from internal and third-party source systems.
- Essential Strong understanding of data modelling, transformation techniques and data warehousing principles.
- Essential Experience working with cloud-based data lake and analytics platforms.
- Essential Strong understanding of batch and near real-time data processing patterns.
- Essential Experience conducting data profiling, discovery and validation activities to assess data quality, completeness and suitability for business requirements.
- Essential Experience implementing data quality checks,



reconciliations and monitoring processes.
- Essential Ability to investigate and resolve ingestion, transformation and data quality issues identified during testing, UAT or production support.
- Essential Understanding of data governance, security, data lineage and documentation standards.
- Essential Experience producing technical documentation and operational handover materials.
- Essential Robust stakeholder engagement skills with the ability to work effectively across business, architecture, engineering and analytics teams.
- Essential Experience working within Agile delivery environments.
- Essential Knowledge of source control, CI/CD practices and release management processes.
- Essential Ability to work independently while collaborating effectively within cross-functional squads.

Desirable

- Experience integrating data from retail technology platforms, IoT devices or third-party vendor systems.
- Experience working with AI Camera, Computer Vision or Electronic Shelf Edge Label (eSEL) technologies.
- Knowledge of Azure Data Lake, Azure Data Factory and related Azure data services.
- Experience supporting reporting, analytics or BI solutions through the creation of trusted and governed data assets.
- Experience working within large-scale retail or data transformation programmes.

Disclaimer: This job posting has been aggregated from external source. Role details, content, and availability are subject to change. Applicants are advised to confirm the latest information directly on the company website before applying.

📌 Lead II - Data Engineering (Kerala)
🏢 UST
📍 Kerala