We are looking for a Lead Data Engineer to own the data foundation for a high-volume client engagement where accuracy, reconciliation, reliability, and data integrity are critical.
Roles & Responsibilities
• Own ingestion, transformation, and serving pipelines for high-volume event data using AWS, Databricks, and PySpark.
• Make build-versus-buy decisions balancing cost, latency, and reliability.
• Maintain lakehouse architecture as data sources, volumes, and schemas evolve.
• Manage pipeline infrastructure as code, CI/CD for data jobs, and orchestration using Airflow or Databricks Workflows.
• Optimize cloud costs through partitioning, file formats, and compute sizing; track cost per pipeline and query.
• Diagnose and resolve bottlenecks in Spark jobs and downstream serving layers.
• Implement reconciliation and data quality checks to prevent discrepancies in reporting and billing.
• Monitor and alert on pipeline failures, schema drift, and silent data loss.
• Ensure data accuracy for downstream business logic.
• Define schema contracts, naming conventions,



and documentation standards.
• Establish access and data-handling practices aligned with client and regulatory requirements.
• Review shared dataset changes and communicate potential breaking changes to downstream consumers.
• Guide engineers without formal reporting authority; mentor through design reviews, debugging, and code reviews.
• Represent data engineering with clients, explain trade-offs, defend technical decisions, and flag risks early.

Key Skills
• 8–12 years of data engineering experience with ownership of large-scale production pipelines.
• Deep hands-on expertise in AWS, Databricks, and PySpark, including job troubleshooting.
• Experience managing schema evolution and upstream changes without disrupting downstream consumers.
• Experience in high-volume, high-stakes environments where accuracy and timeliness have business consequences.
• Cloud cost optimization, including reducing costs and resolv

📌 Lead Data Engineer | 8–12 Years | Bengaluru
🏢 Nasugroup.com
📍 Bengaluru