Responsibilities:
- Design, develop, and optimize Apache Spark and PySpark workloads within Databricks for large-scale batch and streaming data processing.
- Build and maintain robust ETL/ELT pipelines supporting Bronze, Silver, and Gold layer data movement and transformation.
- Implement Medallion Architecture patterns, ensuring proper separation of raw, conformed, and certified data layers.
- Develop and support Delta Lake solutions utilizing ACID transactions, schema enforcement, incremental processing, and SCD Type 1 and Type 2 methodologies.
- Translate conceptual and logical data models into scalable physical Databricks tables, views, and analytical structures.
- Design metadata-driven and schema-aware frameworks that support reusable, parameterized pipeline execution.
- Implement orchestration strategies using Databricks Workflows, Airflow, Azure Data Factory, or similar tools.
- Embed automated data quality, validation, reconciliation, and monitoring controls into engineering solutions.
- Develop and execute unit, integration,



and regression testing for pipelines and data products.
- Implement logging, observability, alerting, and operational monitoring frameworks.
- Support production deployments, stabilization activities, warranty periods, issue remediation, and BAU transition.
- Collaborate with architects, modelers, and business SMEs to validate data definitions, canonical keys, and entity-level acceptance criteria.
- Ensure compliance with governance, security, access control, and enterprise data management standards.
- Contribute to agile delivery activities including sprint planning, backlog refinement, code reviews, and continuous improvement initiatives.

Must-Have Skills:
- Advanced Databricks engineering experience.
- Solid expertise with Apache Spark, PySpark, Spark SQL, and DataFrame APIs.
- Advanced SQL development skills supporting transformations, joins, SCD processing, and analytical workloads.
- Strong Python development experience for reu

📌 Senior Databricks data Engineer (Bengaluru)
🏢 Everforth Apex Systems
📍 Bengaluru