Experience:- 8-10 Years

Work Type- Hybrid

Duration:6-12 Months

Shift Timing: 1:00 PM – 10:00 PM

Job Locations:--Bengaluru, India / Pune, India / Noida, India / Gurgaon, India

Skills Required

Primary Skills

PySpark and Databricks

Shift Timing: 1:00 PM – 10:00 PM

Key Responsibilities

A. Data Engineering & Pipeline Development

• Design, develop, and optimize large-scale ETL/ELT pipelines using PySpark on Databricks, processing structured and unstructured data at scale.

- Build and maintain Lakehouse architecture (Bronze/Silver/Gold medallion layers) using Delta Lake, ensuring reliability, scalability, and schema evolution support.
- Develop reusable, parameterized, metadata-driven pipeline frameworks for ingestion, transformation, and curation of data from diverse source systems.
- Optimize Spark jobs for performance and cost (partitioning, caching, cluster sizing/auto-scaling, Photon engine, Z-ordering, file compaction).
- Implement data quality checks, validation rules, and monitoring/alerting to ensure pipeline reliability and data trust.

B. Databricks Platform & Cloud Engineering

• Configure and manage Databricks workspaces, clusters, jobs, and workflows; tune cluster policies for cost and performance.
- Implement data governance, access control, and lineage using Unity Catalog.
- Integrate Databricks with cloud services such as Azure Data Lake Storage (ADLS Gen2), Azure Data Factory, Event Hub/Kafka, and Key Vault (or AWS S3/Glue/EMR equivalents).
- Build and maintain CI/CD pipelines for Databricks notebooks, jobs, and Delta Live Tables using Azure DevOps/GitHub Actions and Databricks Repos.

C. Data Modeling, Architecture & Governance

• Design and maintain data models (dimensional, medallion, canonical) to support analytics, reporting, and downstream consumption.




- Partner with Data Architects on target-state data platform design, migration strategy, and platform standards.
- Ensure adherence to data governance, security, and privacy requirements (data masking, encryption, access controls, regulatory compliance).

D. Leadership & Collaboration

• Lead and mentor a team of data engineers; conduct code reviews and enforce engineering best practices and coding standards.
- Partner with Business Analysts, Data Scientists, and Product Owners to translate requirements into scalable technical solutions.
- Participate in Agile ceremonies, provide effort estimates, and manage delivery timelines and technical risks.

Tools & Environment (Typical)

• Big Data & Processing: PySpark, Databricks, Delta Lake, Spark SQL, Apache Airflow, Delta Live Tables.
- Cloud: Azure (ADLS Gen2, Data Factory, Synapse, Event Hub, Key Vault) or AWS (S3, Glue, EMR).
- Languages: Python, SQL; Scala (nice to have).
- DevOps: Git, Azure DevOps/GitHub Actions, Databricks CLI/API, Terraform (nice to have).
- Orchestration & Monitoring: Databricks Workflows, Airflow, Prometheus/Grafana, Datadog.

Required Qualifications

• 8–10 years of overall Data Engineering experience, including at least 4–5 years of hands-on work with PySpark and Databricks.
- Strong programming skills in Python and expert-level SQL.




- Proven experience building and optimizing large-scale ETL/ELT pipelines and Delta Lake/Lakehouse architectures.
- Solid understanding of Spark internals (partitioning, shuffling, joins, caching) and hands-on performance tuning experience.
- Hands-on experience with cloud data platforms (Azure preferred; AWS/GCP acceptable).
- Experience with CI/CD, version control (Git), and orchestration tools (Airflow/Databricks Workflows).
- Experience leading or mentoring a small team of data engineers.
- Strong communication skills and ability to work effectively with cross-functional and global teams.

Preferred Qualifications

• Databricks Certified Data Engineer Associate/Professional.
- Azure Data Engineer Associate (DP-203) or equivalent cloud certification.
- Experience in the Insurance or Financial Services domain.
- Exposure to Unity Catalog, Delta Live Tables, and MLflow.
- Familiarity with data quality and governance tooling (e.g., Great Expectations, Collibra).

Core Competencies & Behaviors

• Strong problem-solving and analytical thinking.
- Ownership mindset with close attention to data quality and reliability.
- Ability to balance hands-on delivery with technical leadership.
- Team-oriented, with clear communication to both technical and business stakeholders.
- Adaptability and effectiveness in a fast-paced, agile environment.

Key Deliverables

• Scalable PySpark/Databricks pipelines (Bronze/Silver/Gold) meeting agreed SLAs.
- Pipeline documentation, runbooks, and monitoring/alerting dashboards.
- Optimized Spark jobs with defined and measured performance benchmarks.
- Data quality and governance framework implementation (Unity Catalog, validation rules).
- Technical mentorship and guidance provided to junior and mid-level engineers.

📌 Lead Data Engineer – PySpark & Databricks (India)
🏢 Sparix Global
📍 India