We are looking for a Python / PySpark Data Engineer to design, build, and optimize large-scale data pipelines and analytics solutions on cloud infrastructure. This role is pivotal in enabling data-driven decisions across our portfolio of SaaS businesses. The ideal candidate will partner closely with product, analytics, and engineering teams to deliver reliable, high-performance data platforms.

Responsibilities

- Design and develop scalable data pipelines using Python and PySpark on AWS (EMR, Glue, or Databricks).

- Build and maintain ETL/ELT workflows to process large volumes of structured and semi-structured data across multiple data sources.

- Collaborate with product managers and data analysts to translate business requirements into robust data models and analytical pipelines.

- Design efficient data schemas and manage data persistence using SQL/NoSQL stores including Redshift, Athena, S3, and DynamoDB.

- Implement data quality frameworks, monitoring, and alerting to ensure pipeline reliability and data integrity.

- Lead CI/CD integration for data pipelines using Bitbucket Pipelines or equivalent DevOps tooling for continuous delivery and deployment.

- Promote engineering best practices including pair programming, code reviews, unit testing, and documentation.

- Operate within Scrum teams, contributing to agile ceremonies and driving data engineering domain best practices.

- Champion an AI-first development approach by integrating AI tools and LLM-powered capabilities into data engineering workflows and pipeline automation.

- Leverage Claude Code and Anthropics Claude API to accelerate development velocity, automate repetitive data tasks, perform AI-assisted code reviews,



and generate high-quality transformation logic and test code.

- Mentor the team on responsible and effective use of AI pair-programming tools, prompt engineering best practices, and AI-augmented data pipeline development.

Desired Qualifications

- B.E./B.Tech in Computer Science, Data Engineering, or a related field; Masters degree preferred.

- Minimum 4 to 6 years of experience in data engineering with strong hands-on expertise in Python and PySpark.

- Proven experience building and optimizing large-scale data pipelines on AWS (EMR, Glue, Databricks, or equivalent).

- Strong proficiency in SQL and experience with columnar/analytical databases such as Redshift, Athena, or Snowflake.

- Experience with streaming data architectures using Apache Kafka, Spark Streaming, or AWS Kinesis.

- Familiarity with data orchestration tools such as Apache Airflow.

- Working knowledge of data lake and lakehouse architectures (Delta Lake, Apache Iceberg, or AWS Lake Formation).

- Strong understanding of DevOps principles and experience with CI/CD tools, specifically Bitbucket Pipelines.

- Demonstrated hands-on experience with an AI-first development approach, including using AI coding assistants, LLM APIs, or generative AI tools to accelerate engineering delivery and product innovation.

- Proficiency with Claude Code (Anthropics agentic CLI coding tool) or equivalent AI-powered development environments (GitHub Copilot, Cursor, etc.) for daily development tasks, code generation, refactoring, and debugging.

- Familiarity with prompt engineering, RAG (Retrieval-Augmented Generation), and integrating LLM APIs (Anthropic Claude, OpenAI, etc.) into data pipelines or analytical workflows is a robust plus.

📌 PySpark Data Engineer (Bengaluru)
🏢 DigiRecruitx
📍 Bengaluru