About the Role

We are looking for a Data Engineering Manager (or a senior Tech Lead ready to step into management) to lead the teams building and running our data lake platform for enterprise clients. This is a player-coach role: you will manage 10–15 data engineers while staying deep enough in the stack to review pipeline designs, debug the ugly production incident, and set the engineering standards the platform is held to.

The work spans two distinct modes, and you will own both:

- Implementation projects — greenfield and migration builds: standing up data lake environments for new clients, building ingestion and transformation pipelines, and taking them through UAT to production
- Maintenance and support engagements — running live, SLA-bound production environments: pipeline reliability, data quality, batch-window adherence, incident response, and continuous enhancement

Hands-on Scala and production data pipeline experience are mandatory. This is a non-negotiable screening bar — our core processing layer is Scala/Spark, and you cannot lead this team, review its work, or earn its respect from behind a status report.

What You'll Own

1. Team Leadership: Implementation and Maintenance Squads

Manage 10–15 data engineers across build and run workstreams. Own hiring, onboarding, one-on-ones, performance management, and growth paths. Rotate engineers deliberately between implementation and maintenance so the team builds full-lifecycle judgment — engineers who build pipelines they'll never have to operate build worse pipelines. Grow senior engineers into tech leads who can own a client workstream independently.

2. Pipeline Architecture and Engineering Quality

Own the technical quality of the data platform:

- Ingestion pipelines — batch and file-based ingestion from heterogeneous client source systems, handling late arrivals, reprocessing, and upstream format drift
- Transformation layers — Scala/Spark processing jobs: design reviews, performance tuning (partitioning, shuffle behavior, skew handling, memory management), and code quality standards
- Orchestration — dependency-aware scheduling, retry and recovery semantics, idempotent job design, and batch-window management
- Data modeling — lake zoning (raw/curated/consumption), file formats and layout strategy, schema evolution, and downstream consumption contracts
- Data quality engineering — validation frameworks, reconciliation checks, anomaly detection, and quality gates that stop bad data before clients see it

3. Implementation Delivery

Own the engineering side of new client implementations: estimation, technical planning, environment setup, pipeline build, data migration and historical loads, UAT support, and cutover to production. Work with delivery and program management on committed timelines, and flag technical risk early with options rather than at the deadline.

4. Production Ownership and SLA Management

Own live environments for maintenance clients:
- Batch SLA adherence — pipelines complete inside committed windows,



and misses are caught and communicated before the client's business day starts
- Incident management — triage, root cause, permanent fixes, and post-incident reviews that reduce recurrence
- Monitoring and observability — pipeline health, data freshness, quality metrics, and alerting that catches issues before downstream consumers do
- Capacity and performance management as client data volumes grow
- Disciplined change management in production environments — releases, hotfixes, and reprocessing runs that don't create new incidents

5. Multi-Client Platform Discipline

Keep the platform coherent across clients: config-over-fork as the default, client-specific customization contained and documented, and a core codebase that improvements flow back into. Resist the entropy of every client engagement drifting into its own snowflake — the economics of the practice depend on it.

6. Standards and Enforcement

Define and enforce the engineering standards for the data practice:
- Scala coding standards, review discipline, and shared libraries
- Pipeline design patterns — templates for ingestion, transformation, and reconciliation that new workstreams start from
- Testing expectations for data pipelines: unit tests on transformations, data validation suites, regression testing on reprocessing
- CI/CD for data workloads — versioned, promotable, rollback-capable deployments
- Documentation standards: pipeline runbooks, data dictionaries, and operational handover packs

Standards are measured in the codebase and the incident trends, not in the wiki.

7. Client and Stakeholder Engagement

Be the senior technical face of the data team to client counterparts: design walkthroughs, technical governance calls, incident explanations, and enhancement scoping. Give delivery leadership honest engineering status. Translate client asks into technical plans, and push back with alternatives when asks threaten platform coherence or committed SLAs.

What You'll Need

Must-Haves

- 8–12+ years in data engineering, with 2–4+ years leading data engineering teams as a Tech Lead or Engineering Manager
- Mandatory: deep, hands-on Scala expertise — production Scala/Spark development you have written, reviewed, and tuned yourself, not overseen from a distance
- Mandatory: production data pipeline experience at scale — you have built, operated, and debugged batch pipelines processing large volumes under committed time windows
- Demonstrated experience leading 8+ data engineers, including performance management and technical mentoring
- Solid command of the Spark internals that matter in production: partitioning strategy, shuffle and skew behavior, memory tuning, join strategies,



and job troubleshooting from the Spark UI down
- Data lake / lakehouse implementation experience — lake zoning, file format and layout decisions (Parquet/ORC, Delta/Iceberg/Hudi or equivalent), schema evolution, and metadata management
- Experience with orchestration frameworks (Airflow or equivalent) and designing dependency-aware, restartable, idempotent job flows
- Experience running SLA-bound production data environments — batch windows, incident response, reconciliation, and the operational discipline live clients demand
- Strong SQL and data modeling fundamentals across relational and lake paradigms
- Experience with cloud data platforms (at least one of the major hyperscalers' data stacks; Databricks, Snowflake, or BigQuery exposure valued) and comfort with on-premise or hybrid deployments
- Ability to review a pull request in the morning and walk a client through an incident root cause in the afternoon — credible in both rooms

Strong-to-Have

- Experience in BFSI — ideally Mutual Fund / Asset Management — familiarity with NAV cycles, T+1 processing pressure, regulatory reporting, and reconciliation-heavy data flows
- Experience in multi-tenant or multi-client platform environments where one codebase serves many deployments
- Background spanning both implementation projects and long-running support engagements
- Hands-on cloud experience on AWS and/or GCP, with working knowledge of their core data engineering services — e.g. S3/GCS, Glue/Dataproc, EMR/Dataflow, Redshift/BigQuery, IAM and networking basics for data workloads — and comfort designing pipelines that run natively on cloud-managed infrastructure rather than only on self-managed clusters
- Exposure to modern lakehouse and transformation tooling — Snowflake, dbt, and/or Databricks — including how these fit alongside or replace hand-rolled Scala/Spark transformation layers for parts of the platform
- Experience with real-time streaming architectures — Kafka and/or Flink — for use cases needing continuous ingestion or low-latency processing, beyond the batch-dominant core of this platform
- Experience with data governance tooling — lineage, cataloging, access control — in regulated environments
- Infrastructure-as-code and containerization familiarity (Terraform, Kubernetes) for environment provisioning

How You'll Work

- Player-coach — reviews Scala, tunes the stubborn Spark job, joins the 6 a.m. incident bridge when it matters — but measures success by what the team delivers without you
- Operationally paranoid, in a good way — assumes pipelines fail, designs for restart and reconciliation, and treats monitoring as a first-class deliverable
- Standards with teeth — enforces quality through review gates, templates, and CI, not reminders
- Direct and low-drama — straight feedback to engineers, straight status to leadership and clients
- Pragmatic over purist — knows when the elegant refactor is worth it and when the client's batch window says otherwise

📌 Data Engineering Manager (Kolkata)
🏢 KFintech
📍 Kolkata