Core Technologies
Apache Spark (Core, Structured Streaming)
PySpark
Databricks (AWS/Azure)
Advanced SQL DevOps &
CI/CD
Jenkins
Git/GitHub/Bitbucket

Programming
Python
SQL

Cloud &
Infrastructure
AWS (S3, EMR, EC2, IAM, CloudWatch)
Databricks Runtime &
Cluster Management- Preferred

Streaming &
Integration
Apache Kafka
Snowflake integration (preferred)
Airflow (preferred)

Preferred Qualification
Experience in financial services, payouts, or enterprise data platforms.
Hands-on experience in Delta Lake optimization and incremental processing strategies.
Experience with Snowflake data warehousing.
Solid understanding of distributed computing principles. Key Responsibilities
Design and develop scalable batch and near-real-time ETL/ELT pipelines using Snowflake (AWS) and Apache Spark (PySpark, Spark SQL, Structured Streaming).
Build structured streaming pipelines using Kafka and Spark Structured Streaming.
Design dimensional data models (Fact/Dimension, SCD Type 2).
Orchestrate pipelines using Databricks Workflows / Apache Airflow.
Integrate CI/CD pipelines using Jenkins, Git, Bitbucket/GitHub for automated deployment across DEV/UAT/PROD.

PySpark, Jenkins

📌 Technical Leader Software Pune
🏢 ALTIMETRIK
📍 Pune