This job is with S&P; Global, an inclusive employer and a member of myGwork – the largest global platform for the LGBTQ+ business community. Please do not contact the recruiter directly.

About the Role:

Grade Level (for internal use):

13

Key Responsibilities

Data Pipeline Transition and Platform Delivery

- Partner with the core Databricks team to plan and execute the transition of existing data pipelines to the target enterprise data platform.

- Drive implementation of repeatable engineering patterns for ingestion, transformation, testing, deployment, and monitoring across onboarded datasets.

- Guide the design and operation of cloud-native data pipelines leveraging relevant AWS services such as:

- Amazon S3 for durable storage

- AWS Glue for integration and catalog-driven processing

- AWS Lambda for event-driven processing

- Amazon Kinesis for streaming use cases

- AWS Lake Formation for governed data lake controls

- Promote the use of AWS IAM, encryption, and workplace-level controls to enforce secure access to platform resources and data products in line with enterprise governance expectations.

- Influence architectural decisions related to Databricks, Delta Lake, Apache Iceberg, Unity Catalog, metadata-driven processing, and governed data lake design.

AI-Assisted Engineering and Context Engineering

- Champion the adoption of AI-powered development tools, including Claude, GitHub CoPilot, LLMs, and other AI-assisted engineering solutions , to increase engineering velocity, code quality, and operational effectiveness.





- Apply LLM-based tools to support common data engineering activities such as:

- Code generation

- Code refactoring

- SQL optimization

- PySpark optimization, etc.

- Develop and implement context engineering practices that provide AI tools with the necessary technical background, architectural constraints, data schemas, coding standards, platform patterns, security requirements, and governance expectations.

- Create reusable context packs , prompt libraries, and AI-enabled engineering playbooks for common data platform tasks, including Databricks pipeline migration, ingestion framework development, test generation, code review support, and operational troubleshooting.

- Use LLMs to accelerate understanding and modernization of legacy pipelines, including dependency analysis, code explanation, transformation logic interpretation, metadata extraction, and refactoring recommendations.

Data Onboarding and Asset-Agnostic Enablement

- Provide technical direction for offshore teams supporting data onboarding to the enterprise platform in an asset-agnostic manner.

- Define and operationalize onboarding patterns that can support a broad range of data assets, domains,



and source systems without requiring bespoke platform redesign for each use case.

- Work with partner teams to simplify and standardize how data is ingested, transformed, governed, and published to the platform.

- Establish reusable ingestion and processing patterns across batch and streaming use cases using technologies such as AWS Glue, AWS Lambda, Amazon Kinesis, or event-driven integrations where appropriate.

Data Mastering Platform Integration

- Support integration of platform pipelines and datasets with the enterprise data mastering platform.

- Collaborate with upstream and downstream stakeholders to ensure mastered data can be consumed reliably through standardized interfaces and governed data flows.

- Help establish data quality controls, reconciliation processes, metadata alignment, and stewardship workflows required to support trusted mastered data in the platform.

- Contribute to issue resolution and continuous improvement related to mastering-related ingestion and distribution workflows.

- Support data mastering capabilities aligned with platforms such as NeoXam DataHub, including: Data acquisition, Cleansing, Enrichment, Mastering, Reconciliation, Golden copy generation & Downstream distribution of trusted data products.

Semantic Modeling and Data Discoverability

- Support semantic modeling practices that help create consistent business definitions, reusable data concepts, and governed consumption patterns across the enterprise data platform.

- Partner with business, architecture, da

📌 Director, Distinguished Engineer (Hyderabad)
🏢 S&P Global
📍 Hyderabad