Job Title: Big Data Engineer

Job Locatiob: Bengaluru, Chennai

Experience: 4 - 15 Years of Experience

Key Responsibility:

- Analyze source system data models, schemas, and business requirements to create effective migration strategies.
- Design, develop, and maintain scalable PySpark-based data migration pipelines for moving data from source to Apache Iceberg.
- Perform data extraction, transformation, cleansing, standardization, and loading activities while ensuring data integrity.
- Develop and optimize complex SQL queries for data extraction, validation, reconciliation, and reporting.
- Perform SQL query tuning and performance optimization to improve execution efficiency on large datasets.
- Conduct detailed data profiling and identify data quality issues prior to migration.
- Define and execute data validation and reconciliation processes to ensure completeness and accuracy of migrated data.
- Debug and troubleshoot PySpark jobs, identifying root causes for failures, performance bottlenecks, and data inconsistencies.
- Analyze Spark execution plans and optimize jobs through partitioning, caching,



and effective transformation techniques.
- Monitor and troubleshoot Azure DevOps (ADO) pipeline executions, resolve failures, and coordinate deployments.
- Work with Git repositories to manage code changes through commits, pull requests, code reviews, branching, and merge strategies.
- Investigate production issues, perform Root Cause Analysis (RCA), and implement preventive measures.
- Review job logs, application logs, and execution metrics to identify and resolve operational issues.
- Collaborate with business users, architects, data engineers, QA teams, and project stakeholders to ensure successful project delivery.
- Create and maintain technical documentation, migration runbooks, mapping documents, and operational procedures.
- Support SIT, UAT, deployment, hypercare, and production support activities.
- Ensure adherence to coding standards, development best practices, and data governance policies.
- Participate in estimation, planning, sprint activities, and technical discussions.

📌 Big Data Engineer (Bengaluru)
🏢 SAN Engineering Solutions
📍 Bengaluru