19 Nov
|
Su0026P Global Market Intelligence
|
Hyderabad
19 Nov
Su0026P Global Market Intelligence
Hyderabad
About the Role
Grade Level (for internal use): 11
Responsibilities
Data Pipeline Transition and Platform Delivery
- Partner with the core Databricks team to support the transition of existing data pipelines to the target enterprise data platform.
- Provide hands-on technical guidance for enhancing, stabilizing, and optimizing pipelines after transition, with a focus on performance, scalability, maintainability, operational reliability, and secure AWS deployment patterns.
- Implement and promote repeatable engineering patterns for ingestion, transformation, testing, deployment, monitoring, and support across onboarded datasets.
- Ensure pipelines are designed and operated in a way that supports long-term platform consistency, reliability, observability, and ease of maintenance.
- Support the design and operation of cloud-native data pipelines using AWS services such as:
- Amazon S3 for durable data lake storage
- AWS Glue for data integration, cataloging, and processing
- AWS Lambda for event-driven processing
- Amazon Kinesis for streaming use cases
- AWS Lake Formation for governed data lake controls
- AWS IAM for secure access control
- Promote the use of encryption, environment-level controls, access policies, and platform guardrails to support secure access to platform resources and data products.
- Apply lakehouse technologies and concepts such as Delta Lake, Apache Iceberg, Databricks Unity Catalog, metadata-driven pipelines, and governed data access patterns.
AI-Assisted Engineering and Context Engineering
- Use AI-assisted engineering tools such as Claude, GitHub CoPilot, LLMs, and similar platforms to accelerate data engineering delivery while maintaining enterprise quality and security standards.
- Apply AI tooling to support common engineering activities, including:
- Code generation
- Code refactoring
- SQL and PySpark optimization
- Unit test generation
- Documentation creation
- Pipeline analysis
- Troubleshooting and debugging
- Data quality rule suggestions
- Runbook and operational support documentation
- Apply prompt engineering and context engineering practices to improve the accuracy, relevance, and reliability of AI-generated outputs.
- Structure technical context for AI-assisted work, including requirements, schemas, data contracts, architecture constraints, coding standards, logs, error messages, acceptance criteria, and governance requirements.
- Create or contribute to reusable prompt templates, context packs, and engineering playbooks for common data platform activities such as Databricks migration, AWS data engineering, pipeline troubleshooting, and test generation.
- Review, validate, and refine AI-generated code, recommendations,
documentation, and analysis before applying them to production-grade engineering work.
- Ensure responsible use of AI tooling by following enterprise standards for security, privacy, sensitive data protection, compliance, and human review.
Data Onboarding and Asset-Agnostic Enablement
- Support asset-agnostic data onboarding patterns that can accommodate a broad range of data assets, domains, and source systems without requiring bespoke platform redesign for each use case.
- Work with platform, data engineering, architecture, governance, and business-aligned teams to simplify and standardize how data is ingested, transformed, governed, and published to the enterprise platform.
- Help offshore and global teams adopt common onboarding frameworks, technical standards, and delivery practices that improve speed, consistency, and reduce friction.
- Establish reusable ingestion and processing patterns across batch, streaming, and event-driven use cases.
- Create or contribute to reusable technical assets such as:
- Design patterns
- Reference implementations
- Onboarding templates
- Pipeline frameworks
- Technical documentation
- Operational runbooks
- AI-assisted engineering templates
- Support metadata-driven and configuration-driven pipeline patterns to improve scalability, consistency, and maintainability.
Data Mastering Platform Integration
- Support integration of platform pipelines and datasets with the enterprise data mastering platform.
- Collaborate with upstream and downstream stakeholders to ensure mastered data can be consumed reliably through standardized interfaces and governed data flows.
- Help implement data quality controls, reconciliation processes, metadata alignment, and stewardship workflows required to support trusted mastered data in the platform.
- Contribute to issue resolution and continuous improvement related to mastering-related ingestion, enrichment, and distribution workflows.
- Support data mastering capabilities aligned with platforms such as NeoXam DataHub, including:
- Data acquisition
- Cleansing
- Enrichment
- Mastering
- Reconciliation
- Golden copy generation
- Downstream distribution of trusted data products
- Work with business and platform stakeholders to support golden record / golden copy generation and maintenance for key financial datasets, including reference data, instruments, business entities, market data, investment data, risk data, and related mastered domains.
Technical Leadership and Engineering Excellence
- Serve as a senior technical individual contributor for data platform engineering, with hands-on involvement across pipeline migration, lakehouse architecture, AWS-native data engineering, governance, AI-assisted engineering, and mastering integrations.
- Provide technical guidance to engineers, delivery partners, and platform stakeholders without direct people management responsibility.
- Contribute to architecture discussions, design reviews, code reviews, implementation planning, technical standards, and production readiness reviews.
- Translate architectural direction into practical engineering tasks, implementation patterns, and reusable technical assets.
- Promote engineering best practices including:
- Version control
- Automated testing
- CI/CD
- Infrastructure as code
- Release automation
- Monitoring and alerting
- Incident response
- Documentation
- Production support readiness
- Help establish cloud engineering standards for infrastructure automation, release management, and workplace promotion using tools and services such as AWS CodePipeline, AWS CodeBuild, and infrastructure automation frameworks.
- Drive operational discipline across production data pipelines, including observability, logging, telemetry, support models, service ownership, and incident management.
Collaboration, Governance, and Platform Standards
- Partner effectively with global platform, architecture, governance, security, Databricks-aligned, and data mastering teams to ensure offshore delivery aligns with enterprise standards.
- Act as a technical bridge between platform direction and engineering execution.
- Support governance requirements through appropriate controls around:
- Data lineage
- Schema consistency
- Data quality
- Metadata
- Retention
- Access control
- Encryption
- Auditability
- Secure data distribution
- Ensure monitoring and operational health practices are in place using logging, alerting, telemetry, dashboards, and AWS-native operational tooling where appropriate.
- Help improve consistency and reusability across data onboarding and pipeline delivery by contributing to platform guardrails, standards, engineering playbooks, and reusable implementation patterns.
Required Qualifications
Professional Experience
- 6-8+ years of experience in data engineering, data platforms, cloud data architecture, software engineering, or related engineering domains.
📌 Senior Data Engineer (Hyderabad)
🏢 Su0026P Global Market Intelligence
📍 Hyderabad