Senior Data Engineer - Databricks (PySpark)

We are seeking a Databricks Consultant with robust expertise in PySpark and Python to design, develop, and optimize large-scale data processing solutions on cloud-based data platforms.

The ideal candidate will build scalable data pipelines, ETL workflows, and analytics solutions using Databricks and Apache Spark, supporting enterprise data platforms in IQVIA.

Key Responsibilities

Data Engineering & Databricks Development
Design and develop pipelines using Databricks and PySpark
Build and maintain distributed data processing workflows using Spark
Develop reusable data transformation frameworks and components
Work with Delta Lake for optimized data storage and versioning
PySpark & Python Development
Implement Spark jobs using PySpark for large-scale data processing
Develop Python-based data processing, automation, and orchestration scripts
Optimize Spark transformations (joins, aggregations, partitioning)
Data Pipeline & Architecture
Build and optimize:
Data pipelines
Data ingestion frameworks
Data lake architectures
Ensure high scalability,



fault tolerance, and performance
Performance Tuning & Optimization
Tune Spark jobs for:
Performance
Cost efficiency
Resource utilization
Identify bottlenecks in data pipelines and optimize workloads
Monitor cluster and workload performance
Integration & Cloud Enablement
Integrate Databricks pipelines with:
Azure Data Factory / AWS Glue
Data lakes, APIs, databases
Work with cloud platforms:
Azure / AWS / GCP
Best Practices & Governance
Implement best practices for:
Logging
Monitoring
Error handling
Ensure data quality, security, and governance standards
Follow CI/CD and DevOps practices using Git and pipelines
Collaboration & Delivery
Collaborate with:
Data engineers and architects
Product and business teams
Translate requirements into technical solutions and pipelines
Participate in Agile (Scrum) delivery model
Required Skills & Experience

Core Skills

Strong experience in Databrick

📌 Senior Data Engineer � Databricks/Pyspark (Bengaluru)
🏢 PRI INDIA IT SERVICES PRIVATE
📍 Bengaluru