10 Oct
|
Virtusa
|
Hyderabad
⃣ Sr. Data Engineers (2) – Databricks Pipeline Developers
Role Overview
Build and maintain data pipelines in Databricks for ingestion, cleansing, enrichment, and storage.
Key Responsibilities
• Develop PySpark pipelines for
• API ingestion
• FTP file ingestion
• Data cleansing
• Enrichment transformations
• Implement incremental & batch loads
• Develop Bronze → Silver → Gold transformations
• Translate legacy SQL stored procedures into Spark jobs
• Optimize Delta Lake tables
• Implement validation checks
• Support testing & reconciliation
Required Skills
• Strong PySpark / Spark SQL
• Solid SQL (read & interpret legacy procedures)
• Experience with API integration
• Experience handling flat files (FTP ingestion)
• Experience with Delta Lake
• Knowledge of partitioning & optimization
📌 Databricks (Hyderabad)
🏢 Virtusa
📍 Hyderabad