Job Purpose
Build and maintain reliable, scalable batch and real-time data pipelines on the Enterprise Data Platform to enable analytics, reporting, and downstream applications. The role delivers high-quality data engineering solutions using SQL, Python, and PySpark, with solid focus on streaming, Change Data Capture (CDC), and database mirroring to ensure timely, trusted data delivery.
Duties and Responsibilities
• Design, develop, and optimize data pipelines using SQL, Python, and PySpark on cloud data platforms.
• Implement and operate real-time/streaming data ingestion (e.g., Spark Structured Streaming/Kafka) including schema evolution and late-arriving data handling.
• Set up and manage CDC frameworks and database mirroring for near?real?time replication and minimal-latency updates.
• Build robust data models and curated datasets for analytics, dashboards, and application consumption.
• Ensure data quality, lineage, and observability (validation, alerting, SLAs/SLOs) across batch and streaming workloads.
• Drive performance tuning and cost optimization (partitioning, file formats, caching, autoscaling).
• Harden solutions with security best practices (access controls, PII handling), governance, and compliance standards.
• Contribute to CI/CD using Git/GitHub and DevOps pipelines; automate testing and deployments.
• Partner with Data Platform, BI/Analytics, and Application teams to translate requirements into technical solutions.
• Provide L2/L3 support for pipelines and jobs; troubleshoot incidents, perform RCA, and implement preventive fixes.




• Create and maintain technical documentation and runbooks; participate in code reviews and knowledge sharing.
Key Decisions / Dimensions
• Select appropriate ingestion patterns (batch vs. streaming), CDC/mirroring approaches, and storage formats.
• Define partitioning, indexing, and optimization strategies to meet SLAs.
• Recommend tooling and frameworks for orchestration, testing, and observability.
• Prioritize defect fixes and enhancements based on impact and risk.
Major Challenges
• Maintaining reliability and low latency for mission?critical streaming and CDC pipelines.
• Managing schema changes and data drift across diverse source systems.
• Balancing feature delivery with production support within tight timelines.
• Optimizing performance and cost at scale across environments.
Required Qualifications and Experience
Educational Qualifications:
• Graduate or Post?Graduate in Computer Science, Information Technology, or Data Science/Technologies.
Work Experience:
• 3–4 years of hands?on data engineering experience.
Technical Expertise / Skills Keywords:
• SQL, Python, PySpark
• Data streaming (e.g., Spark Structured Streaming, Kafka), CDC (e.g., Debezium/Log-based), Database Mirroring
• Data modeling, performance tuning, and optimization
• Version control (Git/GitHub) and DevOps pipelines (e.g., Azure DevOps)
• Preferred: Azure Databricks, Azure Data Factory, Data Lake Storage; experience with orchestration and observability tools.

📌 Senior Data Engineer (Western India)
🏢 Bajaj Finance
📍 Western India