Senior Data QA Engineer – Big Data & Azure Databricks

Experience: 5–10 Years
Location: Remote
CTC: Up to ₹22 LPA
Position: Senior Engineer
Environment: 60% On-Premise Big Data | 40% Azure Cloud

Role Overview

We are looking for an experienced Senior Data QA Engineer / Big Data Test Engineer with strong hands-on expertise in Hadoop, Hive, PySpark, Advanced SQL, Python, and Azure Databricks.

The role involves validating large-scale data pipelines, performing source-to-target reconciliation, testing ETL/data transformations, and automating data validation across on-premise Hadoop ecosystems and Azure cloud environments.

Key Responsibilities

- Perform end-to-end ETL/Data Warehouse and Big Data testing across source, transformation, and target layers.
- Validate large datasets within Hadoop/HDFS, including CSV and Parquet files.
- Write complex HiveQL and Advanced SQL queries for data validation, reconciliation, and transformation testing.
- Use PySpark/Spark for large-scale data processing and validation.
- Perform data testing and validation within Azure Databricks.
- Navigate Databricks workspace, manage files using commands such as %fs, and work with Unity Catalog.
- Validate data movement and consistency between on-prem Hadoop/Hive and Azure/cloud environments.
- Develop Python-based automation scripts for data validation and reconciliation.
- Perform source-to-target, record-level, aggregate-level, and apple-to-apple data comparison.
- Work with Snowflake or similar cloud data warehouse platforms.
- Use UNIX/Linux shell scripting for file validation, log analysis, and ETL testing activities.
- Monitor and validate scheduled ETL/data pipeline jobs using Control-M.




- Work with database environments including SQL Server/SSMS.
- Track requirements, user stories, defects, and test activities using Jira and Confluence.

Must-Have Skills

- Hadoop / HDFS – Strong hands-on experience
- Apache Hive / HiveQL – Strong hands-on experience
- PySpark / Apache Spark – Strong hands-on experience
- Advanced SQL – Complex queries and data validation
- Python – Strong scripting/automation experience; critical skill
- ETL / Data Warehouse Testing – Strong practical experience
- Azure Databricks – Mandatory
- Snowflake – Important
- Robust source-to-target data validation and reconciliation experience

Good-to-Have Skills

- Unity Catalog
- Databricks %fs and file management
- Python data-comparison/reconciliation libraries such as DataComPy
- Control-M
- UNIX/Linux Shell Scripting
- SQL Server / SSMS
- PyCharm
- Jira & Confluence
- Cross-platform Hadoop–Databricks connectivity
- Hive Metastore integration

Pay: ₹1,000,000.00 - ₹2,200,000.00 per year

Benefits:

- Flexible schedule
- Health insurance
- Life insurance
- Paid sick time
- Paid time off
- Provident Fund
- Work from home

Application Question(s):

- Are you based in Noida?
- How many years of experience do you have in Hadoop?
- How many years of experience do you have in Hive?
- How many years of experience do you have in Pysprak?
- How many years of experience do you have in Advanced SQL?
- How many years of experience do you have in Python?
- How many years of experience do you have in Data warehouse - Snowflake?
- What's your current CTC?

Education:

- Bachelor's (Preferred)

Experience:

- Data QA Engineer: 5 years (Required)

Work Location: Remote

📌 Senior Data QA Engineer – Big Data & Azure Databricks (India)
🏢 Aigen Technologies
📍 India