Job Summary (List Format): Data Engineering Lead

- Lead the design, build, and operation of a scalable, cloud-native enterprise data platform on AWS.
- Architect and implement AWS-based enterprise data lake frameworks for structured and unstructured data.
- Develop standards for data ingestion, transformation, storage, and secure access.
- Integrate seamlessly with Databricks for analytics and downstream data consumption.
- Design and develop real-time and batch data ingestion pipelines from internal systems (CRM, ERP, KYC, OMS, PMS) and external providers.
- Leverage AWS services including AppFlow, Lambda, Glue, S3, and Athena for data integration.
- Build event-driven pipelines that support near real-time analytics for trading, operations, and compliance use cases.
- Implement platform security, encryption, and access control using AWS KMS, Secrets Manager, Security Hub, Config, and CloudTrail.
- Set up monitoring and alerting for pipeline health, performance, and infrastructure using CloudWatch and Grafana.




- Develop CI/CD pipelines with GitLab to automate testing and deployment of data pipelines.
- Establish data validation, reconciliation, and monitoring frameworks to ensure data quality.
- Manage metadata and data lineage via AWS Glue Data Catalog.
- Provide technical leadership through code reviews, architecture discussions, and team mentorship.
- Collaborate with analytics teams and business stakeholders within an Agile delivery model.
- Utilize robust programming skills in Python, PySpark, and SQL.
- Apply best practices in cloud (AWS), DevOps (GitLab), and analytics platforms (Databricks).
- Demonstrate problem-solving, analytical thinking, and an ownership mindset.
- Bring 7–10 years of experience in data/platform engineering, with a preference for financial services backgrounds and expertise in both batch and real-time processing.

📌 Data Engineering Lead (India)
🏢 Sparix Global
📍 India