Remote
Experience: 8+ Years
Qualification: BE/MS in Computer Science or a related field
Employment Type: Contract (6 Months)

About the Role

We are looking for an experienced Senior Data Engineer to lead the modernization of legacy Hadoop-based data platforms and drive the transition to a scalable Databricks Lakehouse architecture. The ideal candidate will have deep expertise in Data Engineering, Databricks, PySpark, ETL/ELT pipelines, distributed systems, and cloud-based data platforms.

This role offers an opportunity to work on cutting-edge AI and SaaS products while collaborating with global customers across North America, Australia, Europe, and the UAE.

Key Responsibilities
Assess existing Hadoop-based data infrastructure and execute a structured migration to Databricks and Delta Lake.
Modernize legacy architectures and define scalable target-state data platforms.
Design and implement Medallion Architecture (Bronze, Silver, Gold) using Delta Lake and Databricks components such as Delta Live Tables, Delta Sharing, and Workflows.
Develop scalable, reliable, and production-ready ETL/ELT pipelines using PySpark, SQL, and Databricks notebooks.
Build and manage workflow orchestration using Databricks Workflows, Airflow, or similar tools.
Optimize Spark workloads for performance, scalability, and cost efficiency through partitioning, caching, query tuning, and cluster optimization.
Implement data quality frameworks and governance using Unity Catalog, including access control, lineage, and PII masking.
Design and develop real-time and event-driven data pipelines using Kafka or equivalent technologies.
Establish monitoring, observability, and alerting mechanisms to ensure high data reliability.
Collaborate with engineering, business, and customer stakeholders to design and review scalable data architectures.
Required Skills & Experience
8+ years of experience in Data Engineering and distributed data systems.




Strong hands-on experience with the Hadoop ecosystem, including HDFS, YARN, Hive, and Spark on Hadoop.
Proven expertise in Databricks, Delta Lake, and modern Lakehouse architectures.
Advanced proficiency in PySpark, SQL, and large-scale data processing.
Experience designing and building ETL/ELT pipelines and data models.
Working knowledge of Java or Scala for understanding and migrating legacy Hadoop pipelines.
Experience with workflow orchestration tools such as Airflow or Apache NiFi.
Solid understanding of event streaming platforms such as Kafka.
Experience working with cloud platforms including AWS, Azure, or GCP.
Strong foundation in event-driven architectures and distributed systems.
Ability to design, review, and communicate technical architectures effectively.
Experience leveraging AI-assisted development tools to improve engineering productivity.
Strong understanding of Agile delivery methodologies and estimation practices.
Experience working directly with global customers and client-facing teams.
Excellent communication and stakeholder management skills.
High ownership, accountability, and execution focus.
Preferred Qualifications
Experience working with Conversational AI, NLP, or Large Language Model (LLM) platforms.
Knowledge of data pipelines supporting AI/ML model training, fine-tuning, and evaluation workflows.
Experience with vector databases, embedding pipelines, and semantic search infrastructure.
Expertise in handling unstructured and semi-structured conversational data at scale.
Understanding of real-time data processing challenges in AI-driven systems.
Familiarity with data privacy, security, and compliance standards such as GDPR and CCPA

Disclaimer : This job posting has been aggregated from external source. Role details, content, and availability are subject to change. Applicants are advised to confirm the latest information directly on the company website before applying.

📌 Senior Data Engineer - PM (India)
🏢 Sparix Global
📍 India