Job Type: Full-time

Title: Data Architect-Pyspark Location: Bangalore Exp: 8 Yrs Job Description: Required Skills PySpark and Python Apache Spark internals and distributed systems architecture Delta Lake Parquet and Trino Linux and vi commands Workflow orchestration using Airflow MinIO Containerized deployment environments using Kubernetes Git and Azure DevOps (ADO) Visual Studio Code (VS Code) GitHub Copilot Preferred / Nice-to-Have Skills Trade surveillance domain experience Financial services data engineering experience Time-series analytics Market data processing Ideal Candidate Profile The ideal candidate is a Spark architecture expert with proven real-world experience in: Distributed computing Memory and execution optimization Performance tuning at scale Framework development Scalable data platform engineering The candidate should thrive in high-scale environments combine deep technical execution with leadership and deliver robust Spark systems for mission-critical workloads. Key Responsibilities 1. Platform and Application Engineering Design and develop enterprise-scale distributed data processing applications using PySpark. Build reusable Spark frameworks and libraries that can be shared across multiple projects. Architect scalable batch and near-real-time processing pipelines. Develop modular configurable and highly maintainable Spark applications. Establish coding standards and Spark engineering best practices across the team. 2. Performance Engineering Own end-to-end Spark performance optimization including: Memory and CPU utilization optimization Shuffle reduction Stage and DAG optimization Data skew handling Job parallelism tuning Executor sizing Garbage Collection (GC)



optimization Identify and resolve performance bottlenecks using: Spark UI Explain plans Event logs Executor- and stage-level metrics 3. Distributed Processing Expertise Demonstrate robust expertise in: Spark execution model Driver and Executor architecture Lazy evaluation Catalyst Optimizer Lineage and fault tolerance Adaptive Query Execution (AQE)4. Data Partitioning and Data Layout Design and implement effective partitioning strategies. Optimize partition pruning. Implement static and dynamic partitioning. Improve data layout for efficient distributed processing. 5. Join and Window Optimization Implement and optimize large-to-large dataset joins. Work with window joins time-series joins time-based joins and interval joins. Select appropriate join strategies based on data volume data skew cluster behavior and resource profiles. Implement advanced window processing including: Running aggregations Sliding and rolling windows Ranking Look-back calculations Time-based analytics Complex business-rule implementation 6. Spark Optimization Techniques Apply and optimize the following techniques: Cache and Persist Checkpointing Compression Predicate pushdown Bucketing Z-ordering 7. Data Engineering Practices Design and optimize Delta Lake solutions. Implement schema evolution and data governance practices. 8. Technical Leadership Lead architecture discussions for distributed Spark platforms. Conduct Spark code reviews and design reviews. Establish and enforce engineering standards and quality practices. Mentor junior and senior engineers. Drive technical decisions across the data platform. Troubleshoot critical production incidents and lead their resolution. Required Experience: Staff IC

📌 Data Architect-Pyspark (Bengaluru)
🏢 NR Consulting
📍 Bengaluru