CHENNAI LOCATION IS AVAILABLE

NO FRESHER WILL BE PREFERRED

We are looking for Senior Data Engineer (Databricks Pyspark Python) as a Full Time for our IT Company.

Location- Chennai
Experience- 6+ years
Budget- As per company standard
Notice Period- Immediate Joiner

Data Engineer

Project Description

Horizon is a data and AI platform designed to collect, process, and analyze travel-market signals from multiple external sources, including social-listening platforms, travel-data providers, SEO/AEO tools, RSS feeds, web sources, and public APIs.

The platform will use Databricks as its central data-processing and governance workplace. It will ingest and normalize structured, semi-structured, and unstructured data, apply data-quality and privacy controls, and provide reliable datasets for signal scoring, travel-corridor analysis, automated reporting, and downstream AI services.

Responsibilities

Design, build, and maintain scalable data pipelines in Databricks.

Develop ingestion pipelines for REST APIs, RSS feeds, web sources, files, and third-party data providers.





Transform heterogeneous source data into standardized travel-signal and corridor data models.

Implement batch and scheduled data-processing workflows using Python, SQL, PySpark, and Databricks Workflows.

Design and maintain Delta Lake tables and appropriate Bronze, Silver, and Gold data layers.

Configure and use Unity Catalog for data governance, access control, metadata management, and lineage.

Implement data-quality controls, schema validation, deduplication, error handling, and reconciliation mechanisms.

Build processing logic for rolling time windows, signal aggregation, scoring inputs, rankings, and reporting datasets.

Implement data-retention and automated data-cleanup processes.

Support PII detection, filtering, masking, and other data-privacy requirements.

Optimize Spark jobs, queries, storage layouts, cluster utilization, and pipeline performance.

Implement monitoring, logging, alert

📌 Senior Data Engineer (Chengalpattu)
🏢 Haparz
📍 Chengalpattu