Experience:- 8-10 Years
Work Type- Hybrid Duration:6-12 Months
Shift Timing: 1:00 PM 10:00 PM

Job Locations:--Bengaluru, India / Pune, India / Noida, India / Gurgaon, India

Skills Required

Primary Skills

PySpark and Databricks

Job Description

Shift Timing: 1:00 PM 10:00 PM

Key Responsibilities A. Data Engineering & Pipeline Development

Design, develop, and optimize large-scale ETL/ELT pipelines using PySpark on Databricks, processing structured and unstructured data at scale.

Build and maintain Lakehouse architecture (Bronze/Silver/Gold medallion layers) using Delta Lake, ensuring reliability, scalability, and schema evolution support.

Develop reusable, parameterized, metadata-driven pipeline frameworks for ingestion, transformation, and curation of data from diverse source systems.

Optimize Spark jobs for performance and cost (partitioning, caching, cluster sizing/auto-scaling, Photon engine, Z-ordering, file compaction).

Implement data quality checks, validation rules, and monitoring/alerting to ensure pipeline reliability and data trust.

B. Databricks Platform & Cloud Engineering

Configure and manage Databricks workspaces, clusters, jobs, and workflows; tune cluster policies for cost and performance.

Implement data governance, access control, and lineage using Unity Catalog.

Integrate Databricks with cloud services such as Azure Data Lake Storage (ADLS Gen2), Azure Data Factory, Event Hub/Kafka, and Key Vault (or AWS S3/Glue/EMR equivalents).

Build and maintain CI/CD pipelines for Databricks notebooks, jobs, and Delta Live Tables using Azure DevOps/GitHub Actions and Databricks Repos.

C. Data Modeling, Architecture & Governance

Design and maintain data models (dimensional, medallion, canonical) to support analytics, reporting, and downstream consumption.

Partner with Data Architects on target-state data platform design,



migration strategy, and platform standards.

Ensure adherence to data governance, security, and privacy requirements (data masking, encryption, access controls, regulatory compliance).

D. Leadership & Collaboration

Lead and mentor a team of data engineers; conduct code reviews and enforce engineering best practices and coding standards.

Partner with Business Analysts, Data Scientists, and Product Owners to translate requirements into scalable technical solutions.

Participate in Agile ceremonies, provide effort estimates, and manage delivery timelines and technical risks.

Tools & Environment (Typical)

Big Data & Processing: PySpark, Databricks, Delta Lake, Spark SQL, Apache Airflow, Delta Live Tables.

Cloud: Azure (ADLS Gen2, Data Factory, Synapse, Event Hub, Key Vault) or AWS (S3, Glue, EMR).

Languages: Python, SQL; Scala (nice to have).

DevOps: Git, Azure DevOps/GitHub Actions, Databricks CLI/API, Terraform (nice to have).

Orchestration & Monitoring: Databricks Workflows, Airflow, Prometheus/Grafana, Datadog.

Required Qualifications

8 10 years of overall Data Engineering experience, including at least 4 5 years of hands-on work with PySpark and Databricks.

Strong programming skills in Python and expert-level SQL.

Proven experience building and optimizing large-scale ETL/ELT pipelines and Delta Lake/Lakehouse architectures.

Solid understanding of Spark internals (partitioning, shuffling, joins, caching)



and hands-on performance tuning experience.

Hands-on experience with cloud data platforms (Azure preferred; AWS/GCP acceptable).

Experience with CI/CD, version control (Git), and orchestration tools (Airflow/Databricks Workflows).

Experience leading or mentoring a small team of data engineers.

Strong communication skills and ability to work effectively with cross-functional and global teams.

Preferred Qualifications

Databricks Certified Data Engineer Associate/Professional.

Azure Data Engineer Associate (DP-203) or equivalent cloud certification.

Experience in the Insurance or Financial Services domain.

Exposure to Unity Catalog, Delta Live Tables, and MLflow.

Familiarity with data quality and governance tooling (e.g., Great Expectations, Collibra).

Core Competencies & Behaviors

Strong problem-solving and analytical thinking.

Ownership mindset with close attention to data quality and reliability.

Ability to balance hands-on delivery with technical leadership.

Collaborative, with clear communication to both technical and business stakeholders.

Adaptability and effectiveness in a rapid-paced, agile environment.

Key Deliverables

Scalable PySpark/Databricks pipelines (Bronze/Silver/Gold) meeting agreed SLAs.

Pipeline documentation, runbooks, and monitoring/alerting dashboards.

Optimized Spark jobs with defined and measured performance benchmarks.

Data quality and governance framework implementation (Unity Catalog, validation rules).

Technical mentorship and guidance provided to junior and mid-level engineers.

Disclaimer : This job posting has been aggregated from external source. Role details, content, and availability are subject to change. Applicants are advised to confirm the latest information directly on the company website before applying.

📌 Lead Data Engineer PySpark & Databricks (Noida)
🏢 Sparix Global
📍 Noida