About the Role:

Grade Level (for internal use):
13Key Responsibilities
Data Pipeline Transition and Platform Delivery
Partner with the core Databricks team to plan and execute the transition of existing data pipelines to the target enterprise data platform.
Drive implementation of repeatable engineering patterns for ingestion, transformation, testing, deployment, and monitoring across onboarded datasets.
Guide the design and operation of cloud-native data pipelines leveraging relevant AWS services such as:
Amazon S3 for durable storage
AWS Glue for integration and catalog-driven processing
AWS Lambda for event-driven processing
Amazon Kinesis for streaming use cases
AWS Lake Formation for governed data lake controls
Promote the use of AWS IAM, encryption, and setting-level controls to enforce secure access to platform resources and data products in line with enterprise governance expectations.
Influence architectural decisions related to Databricks, Delta Lake, Apache Iceberg, Unity Catalog, metadata-driven processing, and governed data lake design.





AI-Assisted Engineering and Context Engineering
Champion the adoption of AI-powered development tools, including Claude, GitHub CoPilot, LLMs, and other AI-assisted engineering solutions, to increase engineering velocity, code quality, and operational effectiveness.
Apply LLM-based tools to support common data engineering activities such as:
Code generation
Code refactoring
SQL optimization
PySpark optimization, etc.
Develop and implement context engineering practices that provide AI tools with the necessary technical background, architectural constraints, data schemas, coding standards, platform patterns, security requirements, and governance expectations.
Create reusable context packs, prompt libraries, and AI-enabled engineering playbooks for common data platform tasks, including Databricks pipeline migration, ingestion framework development, test generation, code review suppo

📌 Distinguished Engineer, Data Platforms Hyderabad (India)
🏢 S&P Global
📍 India