Key Responsibilities

Architecture & Design

- Lead the design and implementation of end-to-end Medallion Architecture (Bronze, Silver, Gold layers) using Delta Lake
- Design scalable and maintainable data solutions aligned with enterprise standards

Pipeline Development & Optimization

- Build scalable batch and real-time streaming pipelines using PySpark, SQL, and Delta Live Tables (DLT)
- Perform advanced performance tuning including Z-Ordering, Liquid Clustering, partitioning, and Spark UI debugging to reduce compute costs and improve latency

Data Governance & Security

- Implement and manage enterprise-grade security and data lineage using Unity Catalog
- Establish data governance frameworks and best practices across the organization

Orchestration & DevOps

- Manage complex workflows using Databricks Workflows or Apache Airflow
- Implement CI/CD using Databricks Asset Bundles (DABs) and Git integration

Migration & Integration

- Lead migrations from legacy systems (Hadoop, On-prem SQL, SAP HANA) to the Databricks Lakehouse

Technical Leadership & Mentoring

- Act as the primary point of contact for Databricks architecture decisions
- Conduct code reviews and mentor junior staff
- Participate in the development of standards, processes, and frameworks to enhance overall data integration capabilities




- Continuously stay updated with the latest trends and technologies in the data integration and cloud data platform space

Core Competencies Required

Databricks & Platform Expertise

- Expert-level knowledge of Databricks (Photon engine, Unity Catalog, Serverless SQL, Lakeflow)
- Deep understanding of Delta Lake ACID transactions and Medallion Architecture patterns

Programming & Data Processing

- Mastery of Python/PySpark and Advanced SQL
- Deep understanding of Apache Spark internals and Structured Streaming
- Scala knowledge is a plus

Cloud & Infrastructure

- Extensive experience with at least one major cloud platform (Azure, AWS, or GCP)
- Proficiency with cloud-native storage solutions (ADLS Gen2, S3, GCS)
- Proficiency with Terraform and infrastructure-as-code practices

DevOps & CI/CD

- Proficiency with GitHub Actions or Azure DevOps
- Experience implementing CI/CD pipelines for data engineering workflows

Data Engineering Specialization

- Expertise in ELT (Extract, Load, Transform) patterns and reporting frameworks
- Solid understanding of data ecosystem design and data governance
- Experience with performance optimization and cost management in cloud data platforms

Soft Skills

- Strong communication skills with the ability to explain complex technical concepts to non-technical business leaders

📌 Data Engineer | Trianz | Bengaluru (Hyderabad)
🏢 Trianz
📍 Hyderabad