Responsibilities:
Design, develop, and optimize Apache Spark and PySpark workloads within Databricks for large-scale batch and streaming data processing.
Build and maintain robust ETL/ELT pipelines supporting Bronze, Silver, and Gold layer data movement and transformation.
Implement Medallion Architecture patterns, ensuring proper separation of raw, conformed, and certified data layers.
Develop and support Delta Lake solutions utilizing ACID transactions, schema enforcement, incremental processing, and SCD Type 1 and Type 2 methodologies.
Translate conceptual and logical data models into scalable physical Databricks tables, views, and analytical structures.
Design metadata-driven and schema-aware frameworks that support reusable, parameterized pipeline execution.
Implement orchestration strategies using Databricks Workflows, Airflow, Azure Data Factory, or similar tools.
Embed automated data quality, validation, reconciliation, and monitoring controls into engineering solutions.
Develop and execute unit, integration,



and regression testing for pipelines and data products.
Implement logging, observability, alerting, and operational monitoring frameworks.
Support production deployments, stabilization activities, warranty periods, issue remediation, and BAU transition.
Collaborate with architects, modelers, and business SMEs to validate data definitions, canonical keys, and entity-level acceptance criteria.
Ensure compliance with governance, security, access control, and enterprise data management standards.
Contribute to agile delivery activities including sprint planning, backlog refinement, code reviews, and continuous improvement initiatives.

Must-Have Skills:
Advanced Databricks engineering experience.
Solid expertise with Apache Spark, PySpark, Spark SQL, and DataFrame APIs.
Advanced SQL development skills supporting transformations, joins, SCD processing, and analytical workloads.
Solid Python development experience for reu

📌 Senior Databricks Data Engineer Bengaluru
🏢 Everforth Apex Systems
📍 Bengaluru