15 Nov
|
WebSenor®
|
Mumbai
Job highlights
- Shift: General IST
- Develop real-time and batch ingestion pipelines.
- Proven experience in building cloud-native data platforms. Hands-on experience with batch and real-time data processing.
- Experience Required
- Experience in financial services / wealth management preferred
- Experience: 7+ Years
Job Summary
Job Title: Data Engineering Lead; Location: Mumbai;
Experience: 7+ Years; Shift: General IST. We are looking for a highly skilled Data Engineering Lead to design, build, and operate a scalable, cloud-native enterprise data platform on AWS. This is a hands-on leadership role combining architecture design, data pipeline development, and team mentorship in an Agile environment. The role focuses on enabling secure and reliable ingestion of structured and unstructured data for analytics and downstream consumption on Databricks.
Key Responsibilities
A.
Enterprise Data Platform
Architecture
- Design and implement AWS-based enterprise data lake architecture
- Build scalable frameworks for all data types (structured, semi-structured, unstructured)
- Define standards for ingestion, transformation, storage, and access
- Ensure seamless integration with analytics workloads
B.
Data Ingestion
Integration
- Develop real-time and batch ingestion pipelines
- Integrate internal systems (CRM, ERP, KYC, OMS, PMS)
- Integrate external data sources (market data, research feeds, documents)
- Utilize AWS services like AppFlow, Lambda, Glue, S3, and Athena
C. Real-Time Data Processing
- Build event-driven pipelines for near real-time analytics
- Support use cases such as:
- Trading operational analytics
- Compliance and surveillance monitoring
D. Security, Governance Compliance
- Implement security using AWS KMS, Secrets Manager, and Security Hub
- Use AWS Config and CloudTrail for governance and audit
- Ensure strong data protection, encryption, and access control
E.
Monitoring
Observability
- Implement monitoring using CloudWatch and Grafana
- Track pipeline performance, failures, data freshness, and system health
F. DevOps Automation
- Build and manage CI/CD pipelines using GitLab
- Automate testing and deployment processes
- Maintain code quality and version control standards
G.
Data Quality
Metadata
- Implement validation, reconciliation, and monitoring frameworks
- Manage metadata and lineage using AWS Glue Data Catalog
H.
Technical Leadership
Collaboration
- Lead architecture discussions and troubleshooting
- Mentor team members through code reviews and guidance
- Collaborate with analytics teams and business stakeholders
Skills Competencies
Technical Skills
- Programming: Python, PySpark, SQL
- Cloud: AWS (S3, Glue, Athena, Lambda, AppFlow)
- Analytics Platform: Databricks
- Monitoring: CloudWatch, Grafana
- DevOps: GitLab
Behavioral Competencies
- Robust analytical and problem-solving skills
- Ownership mindset with delivery focus
- Ability to lead in fast-paced Agile environments
Experience Required
- 7-10 years in Data Engineering / Platform Engineering
- Proven experience in building cloud-native data platforms
- Hands-on experience with batch and real-time data processing
- Experience in financial services / wealth management preferred
About the team
You will work with WebSenor project managers, UI/UX designers, QA engineers, cloud specialists and senior developers focused on quality delivery for enterprise clients.
Disclaimer: This job posting has been aggregated from external source. Role details, content, and availability are subject to change. Applicants are advised to confirm the latest information directly on the company website before applying.
📌 Data Engineering Lead (Mumbai)
🏢 WebSenor®
📍 Mumbai