We are looking for a Data Engineer to work on a cloud-based data pipeline that automates the ingestion, validation, transformation, and standardization of data coming from multiple legacy systems.
? About the Project
The existing pipeline processes data artifacts from different legacy systems, including Finance and SSOT. Each source produces files with different formats, structures, and data characteristics.
The pipeline automatically identifies the source system and applies the appropriate parsing, cleansing, transformation, and data-quality rules before producing a standardized dataset for downstream analytical and forecasting teams.
The architecture has evolved from a primarily GCP-based pipeline into a hybrid GCP + Azure setting:
• GCP – Data ingestion and transformation
• Azure – Data quality rule checks
• Automated processing of incoming files
• Source-specific parsing and transformation
• Data validation and cleansing
• Standardized output for downstream analytics
? Key Responsibilities
• Develop and maintain automated data ingestion and transformation pipelines




• Work with data arriving from multiple legacy systems and file formats
• Build source-specific parsing and transformation logic
• Implement data quality and validation checks
• Clean, standardize, and transform raw data into analytical-ready datasets
• Work across GCP and Azure cloud environments
• Troubleshoot pipeline and data-quality issues
• Handle varying file sizes and high-volume file ingestion
• Improve automation and reduce manual data handling
• Collaborate with data and analytics teams to ensure reliable downstream datasets
? Required Skills
✅ Data Engineering / ETL
✅ Python
✅ SQL
✅ Data Pipeline Development
✅ Data Quality & Validation
✅ Data Transformation & Cleansing
✅ Data Integration
✅ GCP
✅ Azure
✅ Experience working with heterogeneous file formats and legacy data systems
? Good to Have
• BigQuery / GCP data services
• Azure data services
• Apache Beam or similar data-processing f

📌 Freelance Data Engineer – Cloud Data Pipeline (India)
🏢 ThreatXIntel
📍 India