below :Senior Data Engineer Databricks / PySpark

Role: Databricks Developer / Senior Data Engineer

Experience: 6+ years in Data Engineering, with strong hands-on experience in Databricks and PySpark

Domain: Banking / Financial ServicesRole OverviewWe are looking for an experienced Data Engineer with strong expertise in Databricks, PySpark and Python to design, develop and optimize scalable, reliable and secure data pipelines.The ideal candidate should have hands-on experience in processing large datasets, integrating multiple source systems, implementing end-to-end data engineering solutions and working with enterprise-scale data architectures.The candidate should possess strong technical problem-solving skills, experience in production data engineering and the ability to independently design and implement complex data pipelines.Key Responsibilities

- Design, develop and maintain scalable end-to-end data pipelines using Databricks, PySpark and Python.
- Build and optimize data ingestion frameworks to process large volumes of structured, semi-structured and unstructured data.
- Implement Medallion Architecture comprising Bronze, Silver and Gold layers.
- Develop metadata-driven and reusable ingestion frameworks to support multiple source systems.
- Integrate data from databases, REST APIs, cloud storage, Oracle, SAP and various file formats such as CSV, JSON and Parquet.
- Implement incremental data loading, Change Data Capture (CDC), schema evolution and data quality validation.
- Handle duplicate records, late-arriving data, schema mismatches and data drift.
- Optimize Spark jobs using partitioning, Adaptive Query Execution (AQE), broadcast joins and Delta Lake optimization techniques.
- Implement robust error handling, retry mechanisms, logging, monitoring and production troubleshooting.
- Develop modular,



reusable and maintainable Python code for data engineering applications.
- Implement CI/CD pipelines and manage code deployment across development, QA and production environments.
- Ensure data security through access control, encryption, data masking and role-based security mechanisms.
- Collaborate with business stakeholders, architects and technical teams to understand data requirements and deliver reliable data solutions.

Required Technical SkillsDatabricks & Apache Spark

- Strong hands-on experience with Databricks and PySpark.
- Good understanding of Spark architecture, Driver, Executors and memory management.
- Experience with Delta Lake, Delta tables and Medallion Architecture.
- Knowledge of Spark optimization, AQE, partitioning and performance tuning.
- Experience with Databricks Workflows and orchestration.

Python Programming

- Strong proficiency in Python for data engineering applications.
- Experience developing modular, reusable and production-grade Python code.
- Understanding of multithreading, multiprocessing and asynchronous execution.
- Experience with exception handling, logging and testing frameworks.

Data Engineering & ETL

- Experience building end-to-end ETL/ELT pipelines.
- Strong understanding of metadata-driven ingestion frameworks.
- Experience with CDC, incremental loading and data quality checks.
- Knowledge of schema evolution, deduplication and late-arriving data.




- Experience handling large datasets and integrating multiple source systems.

Cloud & Data Integration

- Experience with cloud platforms such as Azure, AWS or OCI.
- Knowledge of cloud storage solutions such as Azure Blob Storage, ADLS or Amazon S3.
- Experience integrating databases, APIs and enterprise source systems.
- Exposure to Oracle, SAP or banking data systems is desirable.

DevOps & Production Engineering

- Experience with Git, GitHub or Azure DevOps.
- Knowledge of CI/CD pipelines and branching strategies.
- Experience with unit testing, integration testing and production deployments.
- Ability to troubleshoot production failures and perform root cause analysis.

Security & Governance

- Understanding of role-based access control (RBAC).
- Experience with Databricks Unity Catalog and access management.
- Knowledge of PII data protection, encryption and data masking.
- Understanding of data governance and enterprise security practices.

Preferred Qualifications

- Experience in banking, financial services or other regulated industries.
- Experience handling high-volume transactional datasets.
- Exposure to Oracle Fusion, SAP, OCI or enterprise data integration.
- Experience implementing data security and governance frameworks.
- Exposure to AI-assisted development tools such as GitHub Copilot.

Candidate ExpectationsThe candidate should demonstrate:

- Strong hands-on technical expertise.
- Experience working on complex, enterprise-scale data engineering projects.
- Ability to independently design and implement scalable data pipelines.
- Solid analytical and problem-solving skills.
- Ability to explain technical architecture and implementation decisions.
- Ownership of development, deployment and production support activities.

📌 Senior Data Engineer Databricks / PySpark (Bengaluru)
🏢 wissen technology
📍 Bengaluru