About the Role:

Grade Level (for internal use):
13Key Responsibilities
Data Pipeline Transition and Platform Delivery
- Partner with the core Databricks team to plan and execute the transition of existing data pipelines to the target enterprise data platform.
- Drive implementation of repeatable engineering patterns for ingestion, transformation, testing, deployment, and monitoring across onboarded datasets.
- Guide the design and operation of cloud-native data pipelines leveraging relevant AWS services such as:
- Amazon S3 for durable storage
- AWS Glue for integration and catalog-driven processing
- AWS Lambda for event-driven processing
- Amazon Kinesis for streaming use cases
- AWS Lake Formation for governed data lake controls
- Promote the use of AWS IAM, encryption, and setting-level controls to enforce secure access to platform resources and data products in line with enterprise governance expectations.
- Influence architectural decisions related to Databricks, Delta Lake, Apache Iceberg, Unity Catalog, metadata-driven processing, and governed data lake design.





AI-Assisted Engineering and Context Engineering
- Champion the adoption of AI-powered development tools, including Claude, GitHub CoPilot, LLMs, and other AI-assisted engineering solutions, to increase engineering velocity, code quality, and operational effectiveness.
- Apply LLM-based tools to support common data engineering activities such as:
- Code generation
- Code refactoring
- SQL optimization
- PySpark optimization, etc.
- Develop and implement context engineering practices that provide AI tools with the necessary technical background, architectural constraints, data schemas, coding standards, platform patterns, security requirements, and governance expectations.
- Create reusable context packs, prompt libraries, and AI-enabled engineering playbooks for common data platform tasks, including Databricks pipeline migration, ingestion framework development, test generation, code review suppo

📌 Distinguished Engineer, Data Platforms (Hyderabad)
🏢 S&P Global
📍 Hyderabad