It's fun to work in a company where people truly BELIEVE in what they are doing!
Role Overview
Cogentiq I2C is Fractal’s agentic AI Product for invoice-to-cash, covering Collections, Cash Application, Deductions, Invoice Management and Credit Risk. It runs on a Next.js front end, a FastAPI service layer, the Cogentiq agentic runtime, and a data tier on Azure Databricks with PostgreSQL.
We are looking for a hands-on Data Engineering Leader with strong experience in building scalable, configurable, and cloud-agnostic data engineering solutions. The role requires a combination of deep technical expertise, product engineering mindset, and proven ownership of enterprise client deliveries.
A critical requirement is hands-on experience across at least two cloud data engineering platforms, with the ability to design portable PySpark-based pipelines and migrate workloads between platforms with minimal rework
Key Responsibilities
- Data Engineering & Architecture: Design and develop scalable ETL/ELT pipelines, distributed data processing frameworks, Lakehouse solutions, and Medallion architectures (Bronze/Silver/Gold).
- Cloud-Agnostic Engineering: Build reusable, metadata-driven, configuration-based data pipelines that can operate across different cloud environments and processing platforms.
- Multi-Platform Portability: Architect and migrate PySpark workloads across platforms such as Databricks, Microsoft Fabric, AWS Glue/EMR, and GCP Dataproc, managing platform-specific dependencies and optimizations.
- Product Engineering: Develop reusable frameworks, connectors, orchestration capabilities, and accelerators that support multiple enterprise clients with minimal customization.
- Enterprise Integration:
Design upstream and downstream integrations with SAP ECC/S/4HANA and other enterprise systems using APIs, BAPIs, IDocs, OData, CDC, and event-driven interfaces.
- Data Governance: Implement data quality, cataloging, metadata management, lineage, security, and reconciliation capabilities.
- Dev Ops & Data Ops: Establish CI/CD, automated testing, infrastructure automation, observability, monitoring, and production support practices.
- Engineering & Delivery Leadership: Lead engineering teams, own architecture and technical decisions, drive end-to-end client delivery, and ensure engineering quality, scalability, and production readiness.
Required Skills & Experience
- 12+ years of engineering experience, including significant hands-on Data Engineering experience and technical leadership.
- Strong hands-on expertise in Python, PySpark, Spark, SQL, distributed processing, and performance optimization.
- Hands-on experience with at least TWO data engineering platforms, such as: Azure Databricks Microsoft FabricAWS Glue / EMRGoogle Cloud Dataproc / Dataflow
- Demonstrated ability to migrate or adapt PySpark pipelines between platforms, understanding differences in execution environments, storage, orchestration, dependencies, and deployment.
- Experience building configurable, metadata-driven ETL/ELT frameworks, reusable components, and cloud-agnostic data pipelines.
- Strong understanding of Lakehouse architecture, Delta Lake, Medallion Architecture, data modeling,
cataloging, lineage, and data quality.
- Experience with enterprise systems integration, preferably SAP (upstream ingestion and downstream write-back).
- Experience with Dev Ops/Data Ops, CI/CD, automated testing, and production deployments.
- Proven experience in product engineering, engineering team leadership, and end-to-end enterprise client delivery.
Valuable-to-Have
- Experience with additional data platforms, including Snowflake, Big Query, or Azure Synapse.
- Experience with data governance tools such as Unity Catalog, Microsoft Purview, AWS Glue Data Catalog, or Collibra.
- Experience with streaming pipelines, Kafka, CDC, and event-driven architectures.
- Exposure to ML/AI data pipelines using Sage Maker, Vertex AI, or Databricks ML.
- Experience building reusable enterprise data products or accelerators for multiple clients.
Ideal Candidate
A hands-on engineering leader, not just a delivery manager or solution architect, who has built production-grade data pipelines, worked across multiple data engineering platforms, understands how to make PySpark workloads portable, and can lead both product development and complex enterprise client deliveries.
Critical Hiring Criteria: Strong PySpark engineering + hands-on experience across at least two DE platforms + configurable pipeline development + engineering delivery ownership.
If you like wild growth and working with happy, enthusiastic over-achievers, you'll enjoy your career with us!
Not the right fit? Let us know you're interested in a future opportunity by clicking in the top-right corner of the page or create an account to set up email alerts as new job postings become available that meet your interest!
📌 Lead Data Engineering - Product Engineering & Delivery (India)
🏢 Fractal
📍 India