Job highlights

- . MUST HAVE. Azure Cloud Architecture (Expert Level - Required)
- . Advanced PySpark . (Expert Level - Required)
- Expertise (Expert Level - Required)
- . PostgreSQL . (Required)
- Performance & Scalability (Expert Level - Required)
- . VALUABLE TO HAVE

DETAILED Roles & Responsibilities

- 17 to 22 years of experience in cloud-based systems and data platforms management with 8 to 10 years of experience as a Data Architect.
- Design, develop, and implement robust and scalable data architectures on Azure, considering performance, cost-efficiency, and security.
- Identify performance bottlenecks, analyse system behaviour under load, and develop strategies to improve efficiency and scalability.
- Provide guidance on system design and architecture to optimize performance, including database configurations, caching mechanisms, and code optimization.
- Develop and maintain scalable and efficient streaming frameworks.
- Integrate Salesforce solutions with other system components to meet business requirements.
- Utilize scripting languages such as Python for automation and data processing tasks.
- Collaborate with cross-functional teams, including development, operations, and security, to ensure effective cloud architecture and integration.
- Manage a team of 6 to 8 members with below skills.

Detailed

MUST HAVE

Azure Cloud Architecture (Expert Level - Required)

Advanced PySpark (Expert Level - Required)

- Mastery of DataFrame APIs and Spark SQL
- Performance tuning: broadcast joins, caching strategies, shuffle reduction
- Handling skewed data and optimizing wide transformations
- Writing scalable, testable PySpark code with modular design





Databricks Expertise (Expert Level - Required)

- Cluster sizing, autoscaling, and spot instance optimization
- Job orchestration using Workflows with retries, alerts, and dependencies

- Delta Lake
- OPTIMIZE with Z-Ordering
- MERGE operations and schema evolution

- Time travel and versioning
- Use of Unity Catalog for data governance and access control
- Monitoring via Databricks metrics and audit logs

PostgreSQL (Required)

- Query profiling and execution plan analysis
- Indexing strategies for analytical workloads
- Table partitioning and parallel query execution

Performance & Scalability (Expert Level - Required)

- Designing horizontally scalable pipelines
- Spark memory management: executor sizing, GC tuning
- Fault tolerance and retry logic in distributed systems
- Monitoring pipeline health and latency metrics

GOOD TO HAVE

Kafka (Nice to have)

- Designing high-throughput Kafka consumers with Spark Structured Streaming
- Managing consumer groups and offset strategies
- Handling schema evolution with Avro/Protobuf and Schema Registry
- Backpressure handling and watermarking in Spark

Structured Streaming in Spark (Nice to have)

- Stateful aggregations and windowed operations
- Exactly-once semantics with checkpointing and write-ahead logs
- Integration with Delta Lake for streaming upserts

Machine Learning & MLOps (Nice to have)

- MLflow integration for model tracking and deployment

- Deploying Spark ML pipelines
- Feature store design and versioning

- Unity Catalog

Node.js - (Nice to have)

- Building REST APIs for data access and control

- Integrating with PostgreSQL
- Typescript Node and/or React experience

📌 Sr. Data Architect (Hubballi)
🏢 Han Digital Solution
📍 Hubballi