Role & responsibilities

Key Responsibilities

Lakehouse & Medallion Architecture

- Design and implement Medallion architecture (Bronze, Silver, Gold layers) for scalable data processing with focus on schema enforcement, audit trails, versioning, time travel and incremental processing strategies.
- Design and maintain optimized data models for Lakehouse and warehouse consumption

Spark Engineering & Performance Optimization

- Develop high-performance distributed data pipelines using Apache Spark
- Optimize Spark workloads through partitioning, caching, broadcast joins, and query tuning, execution plans analysis and memory optimization
- Implement efficient incremental data processing and change data capture strategies
- Monitor and troubleshoot Spark job failures, latency, and resource bottlenecks

CI/CD & Data Pipeline Automation

- Design and implement CI/CD pipelines for data engineering workflows. Automate build, test, and deployment of data pipelines across environments
- Enable infrastructure-as-code and reproducible deployments

Data Pipeline Engineering

- Build scalable batch and streaming pipelines, Implement incremental data loading and watermark-based processing
- Design resilient ETL/ELT workflows with monitoring and retry mechanisms
- Develop reusable data transformation frameworks
- Implement pipeline observability, logging,



and alerting

Preferred candidate profile

Technical Skills (Revised)

Core Data Engineering

- 6+ years building distributed data pipelines
- Strong Python and SQL expertise
- Data modeling and data lifecycle management

Spark & Distributed Processing (Mandatory)

- Strong experience with Apache Spark (PySpark/Scala Spark)
- Spark performance tuning and optimization
- Partitioning strategies and file optimization
- Execution plan analysis and query optimization
- Distributed computing concepts

Lakehouse & Medallion Architecture

- Experience implementing Bronze, Silver, Gold data layers
- Delta Lake / Lakehouse architectures
- Incremental processing and time travel
- Data versioning and schema evolution

CI/CD & DevOps for Data

- Git-based version control workflows
- CI/CD tools (Azure DevOps / GitHub Actions / Jenkins or similar)
- Deployment automation and workplace management
- Infrastructure-as-code concepts

Data Platform & Cloud

- Experience with Azure/AWS/GCP data platforms
- Data lakes and analytical warehouses
- Workflow orchestration tools (Airflow or similar)

Good to Have

- Microsoft Fabric / Databricks experience
- Streaming frameworks
- Data observability tools

Insurance or financial domain experience

📌 Senior Data Engineer (Telangana)
🏢 Independent Recruitment Consultant (freelancer)
📍 Telangana