CGEMJP00359372- Brokerage Pyspark

Location - PUNE

Brokerage is seeking a highly motivated **Data Engineer** with strong expertise in **PySpark, SQL, and large-scale data processing**. The ideal candidate will be responsible for building scalable data pipelines, optimizing data processing frameworks, and supporting critical financial data platforms. The role requires hands-on experience with big data technologies, ETL development, performance tuning, and data quality management. This JD is aligned with enterprise Data Engineering requirements emphasizing PySpark and SQL expertise.

### Key Responsibilities

- Design, develop, and maintain scalable data pipelines using PySpark and SQL.
- Process and transform large volumes of structured and semi-structured data.
- Develop and optimize ETL workflows for high-performance data processing.
- Perform data cleansing, reconciliation, validation, and quality checks.
- Tune Spark jobs and SQL queries for optimal performance and resource utilization.
- Collaborate with business analysts, data consumers, and upstream/downstream application teams.
- Analyze production issues and support batch processing activities.
- Develop reusable frameworks and utilities for data ingestion and transformation.
- Ensure compliance with data governance, security, and regulatory requirements.
- Participate in code reviews and contribute to engineering best practices.

### Required Technical Skills

Primary Skills (Must Have)

- Solid experience in **PySpark**
- Advanced SQL development and query optimization
- Python programming




- Data Engineering and ETL development
- Large-scale data processing and distributed computing
- Data validation, reconciliation, and data quality frameworks

PySpark Expertise

- Spark DataFrames and Spark SQL
- Joins, aggregations, window functions
- Partitioning and bucketing
- Performance tuning and optimization
- Handling large datasets (millions of records)
- Error handling and data recovery mechanisms

SQL Expertise

- Complex joins and subqueries
- CTEs and Window Functions
- Query tuning and performance optimization
- Stored procedures and data modeling concepts
- Data reconciliation and validation queries

Database Technologies

- Oracle / SQL Server / DB2 / Snowflake
- Data warehousing concepts
- Structured and unstructured data processing

Tools & Technologies

- Git/GitHub
- Unix/Linux Shell Scripting
- Autosys/Airflow (preferred)
- CI/CD pipelines
- Agile/Scrum methodology

### Experience

- **5–8 years of overall IT experience**
- Minimum **3–5 years of hands-on PySpark development**
- Strong experience in SQL development and optimization
- Experience working on enterprise-grade data platforms

### Interview Focus Areas

1. PySpark Transformations & Actions
2. Spark Performance Tuning
3. SQL Query Optimization
4. Data Modeling & ETL Design
5. Partitioning, Caching, and Memory Management
6. Data Reconciliation & Validation
7. Python Coding
8. Batch Processing & Scheduling
9. Financial Data Processing Concepts
10. Problem Solving & Debugging

📌 CGE372- Morgan stanley Pyspark (Pune)
🏢 Diverse Lynx
📍 Pune