Your role

We, at Khatabook, are looking to hire a Data Engineer II for our Data Engineering team.

As a Data Engineer II, you will build, maintain, and scale reliable, secure, and cost-efficient data infrastructure that enables data-driven decision-making across the organisation. You will work on production-grade data pipelines, cloud data warehouses, CDC systems, streaming platforms, and large-scale data processing.

You will play a key role in ensuring data reliability, quality, security, and performance while contributing to AI-ready data infrastructure that supports AI/ML and Generative AI use cases.

What would you do at Khatabook?

- Design, develop, and maintain scalable batch and streaming data pipelines using Python and Apache Airflow.
- Build reliable data ingestion workflows with appropriate error handling, retries, monitoring, and recovery mechanisms.
- Develop incremental data processing workflows and Change Data Capture (CDC) pipelines to ensure timely and consistent data availability.
- Build, maintain, and optimize cloud data warehouses, preferably Snowflake, with a focus on scalability, performance, and cost efficiency.
- Develop and maintain SQL-based data models and transformation workflows using tools such as DBT.
- Build event-driven data workflows using Kafka or other messaging and queuing technologies.
- Develop and optimize Apache Spark jobs for large-scale data processing.
- Build and operate data engineering workloads on AWS and work with Kubernetes and containerised environments where appropriate.
- Optimize SQL queries, data models, Spark jobs, warehouse utilisation, and pipeline execution to improve performance and reduce infrastructure costs.
- Implement data quality checks, validation frameworks, monitoring, alerting, and failure-recovery mechanisms to ensure data accuracy, freshness, and reliability.
- Implement data access controls, role-based access control (RBAC), secure data sharing,



and appropriate security practices for sensitive data.
- Support data governance, metadata management, and data lineage to improve data discoverability and trust.
- Build and maintain data pipelines that prepare structured and unstructured data for AI/ML and Generative AI applications.
- Support AI-powered search and analytics use cases involving Large Language Models (LLMs), embeddings, vector databases, and Retrieval-Augmented Generation (RAG).
- Collaborate with engineering, product, and analytics teams to understand data requirements and deliver trusted, actionable data.
- Take ownership of production workloads, troubleshoot pipeline failures, investigate incidents, and continuously improve data platform reliability and operational efficiency.
- Explore AI-assisted development tools to improve Python and SQL development, debugging, and code quality while validating generated code for correctness and security.
- Participate in a shared production on-call rotation approximately once every three weeks to ensure the reliability of critical data workflows.

What are we looking for?

- You have 3 - 7 years of experience in Data Engineering or a related technical role.
- You have robust Python programming skills and advanced SQL knowledge, including complex joins, window functions, query optimisation, and performance tuning.
- You have hands-on experience building and maintaining production-grade data pipelines using Apache Airflow.




- You have experience working with modern data warehouses such as Snowflake or Amazon Redshift.
- You have experience with CDC pipelines, incremental data ingestion, or database replication.
- You have working knowledge of AWS cloud services and experience operating data engineering workloads.
- You have experience with Apache Spark or PySpark for large-scale data processing.
- You have experience with Kafka or other messaging and queuing technologies.
- You have a strong understanding of relational databases such as PostgreSQL or MySQL.
- You understand data modeling, data quality, pipeline reliability, monitoring, troubleshooting, and production support.
- You have strong analytical, problem-solving, and debugging skills.
- You understand distributed systems, data security, and performance optimisation.
- You have a strong sense of ownership and can independently troubleshoot and improve production systems.
- You can collaborate effectively with engineering, product, and analytics teams to deliver scalable data solutions.

Good to have:

- Strong knowledge of Snowflake security, RBAC, secure data sharing, and compute and storage cost optimisation.
- Familiarity with DBT for data transformation, testing, documentation, and data model management.
- Understanding of dimensional modeling and modern data warehouse architecture.
- Familiarity with Docker, Kubernetes, CI/CD, and automated testing.
- Experience with data observability, metadata management, and data lineage.
- Understanding of Generative AI concepts, including LLMs, embeddings, vector databases, and RAG.
- Exposure to AI-ready data pipelines for structured and unstructured data.
- Familiarity with AI-assisted development tools for Python and SQL.
- Exposure to AI application platforms such as AWS Bedrock or Snowflake Cortex.
- Familiarity with BI and analytics platforms such as Tableau, Redash, or similar tools.

📌 Data Platform Engineer - II (Bengaluru)
🏢 Khatabook
📍 Bengaluru