Description

Role purpose

The Senior Data Engineer will migrate and validate batch data workloads from AWS Databricks to Azure Databricks. The role focuses on notebook and job conversion, Databricks Asset Bundles, data movement, Unity Catalog integration and automated deployment across environments.

Key responsibilities

- Assess source notebooks, jobs, libraries, SQL scripts, schedules, parameters, cluster configurations and upstream or downstream dependencies.

- Refactor Python, PySpark and SQL workloads for Azure Databricks while preserving business logic, data contracts and operational behavior.

- Convert deployment assets to Databricks Asset Bundles and implement environment specific configuration for development, test and production.

- Migrate data and metadata from S3 based patterns to ADLS Gen2 and Unity Catalog aligned structures using approved migration methods.

- Update secrets, service connections, file paths, storage access, libraries and integrations to use Azure platform services and managed identities.

- Implement or migrate scheduled batch workflows, job dependencies, retries, notifications and operational logging.

- Support Unity Catalog object creation, ownership, grants, external locations, storage credentials and workspace bindings under architect guidance.

- Build unit checks and reconciliation utilities for schema, row counts, aggregates, checksums and business rules.

- Resolve defects found during system integration testing and user acceptance testing. Provide evidence for release and cutover decisions.

- Document migrated components, operating instructions,



known constraints and support procedures.

Required experience and skills

- 5+ or more years in data engineering with robust hands on delivery experience in Databricks and Spark based platforms.

-

Experience designing and implementing Genie Spaces in Databricks, including semantic modeling, governance, and optimization of natural language query responses.

Ability to collaborate with business stakeholders to leverage Databricks Genie for AI-driven reporting, data discovery, and decision-making across large-scale data platforms.

- Advanced Python, PySpark and SQL skills, including debugging and optimization of batch workloads.

- Experience with Azure Databricks, ADLS Gen2, Delta Lake, Azure Key Vault and Entra based access patterns.

- Working knowledge of AWS Databricks, S3, IAM roles and cross cloud migration considerations.

- Practical experience with Databricks Asset Bundles, Git based development and CI CD pipelines.

- Knowledge of Unity Catalog objects, privileges, external locations and governed data access.

- Experience with automated data validation and source to target reconciliation.

Preferred experience

- Experience migrating Databricks workloads between cloud providers.





- Experience with Delta Sharing and downstream Power BI or Fabric consumption.

- Exposure to Terraform, Azure Dev Ops, Git Hub Actions or similar automation tools.

- Databricks Data Engineer or Azure Data Engineer certification.

Measures of success

- Migrated workloads reproduce approved source business results and meet agreed acceptance measures.

- Deployment packages promote consistently through environments with traceable evidence.

- Migration defects are resolved quickly without creating avoidable design exceptions.

- Technical documentation supports cutover, operations and knowledge transfer.

Qualifications and professional expectations

- Bachelor degree in computer science, engineering, information systems or a related discipline. Equivalent relevant experience may be considered.

- Clear written and verbal communication with the ability to work across customer, partner and delivery teams.

- Strong ownership, attention to detail and disciplined documentation of decisions, evidence and unresolved risks.

- Ability to work effectively in a distributed delivery model and support planned release activities when required.

We may use artificial intelligence (AI) tools to support parts of the hiring process, such as reviewing applications, analyzing resumes, or assessing responses. These tools assist our recruitment team but do not replace human judgment. Final hiring decisions are ultimately made by humans. If you would like more information about how your data is processed, please contact us.

📌 Data Engineer for S&P EDO (India)
🏢 Ness Digital Engineering
📍 India