We are looking for a Lead Data Engineer to own the data foundation for a high-volume client engagement where accuracy, reconciliation, reliability, and data integrity are critical.

Roles & Responsibilities

- Own ingestion, transformation, and serving pipelines for high-volume event data using AWS, Databricks, and PySpark.
- Make build-versus-buy decisions balancing cost, latency, and reliability.
- Maintain lakehouse architecture as data sources, volumes, and schemas evolve.
- Manage pipeline infrastructure as code, CI/CD for data jobs, and orchestration using Airflow or Databricks Workflows.
- Optimize cloud costs through partitioning, file formats, and compute sizing; track cost per pipeline and query.
- Diagnose and resolve bottlenecks in Spark jobs and downstream serving layers.
- Implement reconciliation and data quality checks to prevent discrepancies in reporting and billing.
- Monitor and alert on pipeline failures, schema drift, and silent data loss.
- Ensure data accuracy for downstream business logic.
- Define schema contracts, naming conventions, and documentation standards.
- Establish access and data-handling practices aligned with client and regulatory requirements.




- Review shared dataset changes and communicate potential breaking changes to downstream consumers.
- Guide engineers without formal reporting authority; mentor through design reviews, debugging, and code reviews.
- Represent data engineering with clients, explain trade-offs, defend technical decisions, and flag risks early.

Key Skills

- 8–12 years of data engineering experience with ownership of large-scale production pipelines.
- Deep hands-on expertise in AWS, Databricks, and PySpark, including job troubleshooting.
- Experience managing schema evolution and upstream changes without disrupting downstream consumers.
- Experience in high-volume, high-stakes environments where accuracy and timeliness have business consequences.
- Cloud cost optimization, including reducing costs and resolving runaway pipelines.
- Hands-on Infrastructure as Code (Terraform or equivalent), workflow orchestration, and CI/CD for data pipelines.
- Strong consulting skills to manage client scrutiny, shifting priorities, and high communication standards.
- Robust technical leadership and problem-solving skills, with the ability to lead without formal authority.

Interested candidates, please share your profile or reach out via DM.

📌 Lead Data Engineer | 8–12 Years | Bengaluru (Hybrid)
🏢 Nasugroup.com
📍 Bengaluru