Job Description

We are seeking an experienced Senior Data Engineer to join our Data and Analytics Platform team. In this role, you will collaborate with data quality engineers and business data analysts to understand data requirements and models, build robust data ingestion pipelines, and develop complex business-specific use cases that enable seamless information consumption from our platform.

You will be part of a dynamic and enthusiastic team driving agile development to build an enterprise-level data analytics platform as part of Taylor Franciss product and technology-led initiatives. The ideal candidate will possess strong hands-on experience with diverse data management methodologies and a proven track record of implementing them across a wide range of data projects to generate highly relevant data products.

Building Scalable Data Pipelines

Design and develop high-quality, scalable ETL/ELT pipelines for processing big data using AWS analytical services. Leverage no-code tools and reusable Python libraries to ensure efficiency, maintainability, and reusability. Ensure data pipelines and platform components are scalable, performant, and cost-efficient.

Platform Engineering

Build and enhance reusable platform capabilities across ingestion, transformation, orchestration, data quality, cataloguing, and monitoring. Develop scalable and reliable AWS data pipelines and data products using standardised data architecture patterns such as medallion architecture, lakehouse, and other modern design paradigms.

Data-Related Automation

Drive configuration-driven, declarative, and automated engineering approaches that reduce bespoke development and improve productivity. Champion automation across the data engineering lifecycle to minimize manual intervention and accelerate delivery.

Collaborating Aligning with Project Goals

Work closely with cross-functional teams, including Tech Leads, Engineering Managers, and Business Analysts, to understand project objectives and deliver robust data solutions. Follow Agile/Scrum principles to drive consistent progress and iterative delivery.

Data Discovery Root Cause Analysis

Perform data discovery and analysis to uncover data anomalies. Identify and resolve data quality issues through root cause analysis, and provide informed recommendations for data quality improvement and remediation.

AI-Augmented Development

Champion the integration of Claude Code and other LLM tools into our software development lifecycle (SDLC). Lead the team in using AI to accelerate coding, debugging, automated testing, and documentation.

CI/CD Operational Excellence

Manage the automated deployment of code and ETL workflows within cloud infrastructure (AWS preferred) using tools such as GitHub Actions, AWS CodePipeline, or other modern CI/CD systems. Implement Infrastructure as Code (IaC), automated testing frameworks, observability solutions, security best practices, and operational reliability measures to ensure robust and resilient data platform operations.

Effective Time Management

Demonstrate strong organizational and time management skills.



Prioritize tasks effectively and ensure the timely delivery of key project milestones.

Technical Mentorship

Set the gold standard for the team by leading code reviews, defining CI/CD patterns, and enforcing data governance standards via AWS Lake Formation.

The AI Frontier (SageMaker Bedrock)

Architect the infrastructure for our AI initiatives. Implement our initial AWS SageMaker footprint (Data Wrangler, Feature Store) and manage AWS Bedrock integrations (Knowledge Bases, RAG pipelines) to support downstream Data Science and GenAI initiatives.

Documentation Data Mapping

Develop and maintain comprehensive data catalogs, including data mapping and documentation, to ensure data governance, transparency, and accessibility for all stakeholders.

Learning Contributing to Best Practices

Continuously improve your skills by learning and implementing data engineering best practices. Stay updated on industry trends and contribute to team knowledge-sharing and codebase optimization.

What You Will Be Doing (Daily Execution Process)

Data Engineering

Build and evolve reusable data platform capabilities. Develop scalable ingestion, ETL/ELT and data processing solutions using AWS. Implement and support medallion data processing and data product patterns. Build automation around data quality, metadata, cataloguing, orchestration and monitoring.

Agile Orchestration

Function as an integral part of an Agile engineering team, defining delivery phases, sub-activities, and milestones while ensuring development standards are rigorously enforced.

Project Delivery

Collaborate with Engineering Managers and Business Analysts to produce accurate delivery estimates and oversee the transition from analysis through to final delivery.

Infrastructure as Code (IaC)

Create and maintain resilient infrastructure using Terraform or AWS CloudFormation and strong deployment practices enabled by automated tests to ensure environment parity and automationstability.

Quality Testing

Implement comprehensive unit testing strategies and automated data quality checks to identify and resolve issues prior to deployment.

Observability Monitoring

Establish and maintain observability practices including logging, monitoring, and alerting to ensure platform health, performance optimization, and proactive issue detection.

Cross-Functional Collaboration

Work closely with cross-functional teams to gather requirements and develop cloud-based applications, analytical services, and platforms.

Security Compliance

Ensure all data solutions strictly adhere to company security guidelines and regulatory requirements. Implement security best practices throughout the data engineering lifecycle.

Knowledge Management

Document technical solutions and maintain a robust knowledge base for the team. Qualifications

3. Technical Requirements (Qualifications)





Experience

6+ years in Data Engineering, with deep expertise in architecting scalable cloud-native data solutions, optimizing complex ETL/ELT workflows, and implementing modern data platform patterns including lakehouse and medallion architectures.

Data Expertise

Robust analytical skills in handling and processing structured and semi-structured datasets, with hands-on experience in designing and implementing scalable and cost-effective data engineering solutions on AWS.

Cloud Technologies

Proficiency in building data pipelines and working with data warehousing solutions on AWS (Redshift, S3, Glue, Lambda, etc.). Experience with alternative cloud platforms (Google Cloud, Azure) is a plus.

Programming Skills

Strong programming proficiency in Python, with additional experience in Java/Scala being advantageous. Ability to write efficient, reusable, and scalable code to process large datasets. PySpark experience is a plus.

Data Warehousing

Proven experience with modern data warehousing tools such as AWS Redshift, Snowflake, or equivalent platforms, with a focus on performance optimization and query tuning.

Platform Engineering Architecture

Experience building reusable platform capabilities and implementing standardised data architecture patterns including medallion architecture, lakehouse, and modern data mesh principles.

AI-Enabled Engineering

Proficiency in using Claude Code or similar AI assistants to streamline complex engineering tasks.

MLOps

Familiarity with setting up SageMaker or Bedrock environments to build the infrastructure where models operate.

Data Modeling

Advanced knowledge of Star/Snowflake schemas and modern Data Lake design principles.

Version Control Automation

Hands-on experience with version control systems such as GitHub, GitLab, or Bitbucket, and with CI/CD pipelines using tools like GitHub Actions, AWS CodePipeline, or Jenkins to ensure smooth, automated deployments. Strong understanding of Infrastructure as Code principles and automated testing frameworks.

Observability Operational Reliability

Experience implementing observability solutions (CloudWatch, Datadog, Grafana, etc.) and ensuring operational reliability through monitoring, alerting, and incident response practices.

Data Governance Security

Knowledge of data governance practices, compliance standards, and security protocols in a cloud environment.

Automation Efficiency

Demonstrated ability to drive configuration-driven and declarative engineering approaches that reduce manual effort and improve team productivity.

Optional Skills

Experience with business intelligence (BI) tools such as Tableau, Power BI, or QuickSight, and exposure to data visualization techniques will be an advantage.

Education

Bachelors or Masters degree in Computer Science, Data Engineering, or a related technical field.

Disclaimer: This job description has been sourced from a public domain and may have been modified by Naukri.com to improve clarity for our users. We encourage job seekers to verify all details directly with the employer via their official channels before applying.

📌 Senior Data Engineer (Bengaluru)
🏢 Taylor and Francis
📍 Bengaluru