Senior / Principal Data Engineer (Databricks)

Experience: 10+ Years

Summary

Senior Data Engineer responsible for reverse engineering Tableau-based reporting solutions and rebuilding underlying data platforms in Databricks.

Ability to work independently, investigate existing dashboards, document data lineage, and define modernization requirements with minimal direction.

Must Have Skills

Databricks: Python, PySpark, Clustering, Jobs, Pipelines, Lakeflow Connectors, SDP.

SQL Server: Stored Procedures, SQL Jobs, Functions, Triggers, query optimization.

Tableau: Workbooks, Data Sources, Views, Flows, lineage analysis.

Robust Data Modeling expertise.

VS Code, Git, CI/CD pipelines.

Experience with GitHub Copilot Agents and AI-assisted development.

Key Responsibilities

Analyze Tableau dashboards and identify upstream data sources and dependencies.

Reverse engineer complex SQL logic and fragmented data flows.

Document data lineage, transformations, and business rules.

Re-engineer legacy data solutions into Databricks using Python and PySpark.

Create scalable data models and pipelines supporting analytics and reporting.





Work from high-level objectives rather than detailed backlogs.

What to have:

- Platform setup: "set up Databricks workspaces", "configured Unity Catalog", or "Databricks Asset Bundles", "Private Link" or "VNet injection", "cluster policies".
- Hands-on pipelines: PySpark DataFrame code (not only Spark SQL), Databricks Jobs/Workflows, Delta Lake, MERGE and incremental loads, and ideally DLT, SDP or Lakeflow.
- Legacy migration: "migrated SQL Server stored procedures to PySpark", "converted SSIS packages to Databricks", or reconciliation between old and new outputs.
- SQL Server depth: stored procedures, SQL Agent jobs, triggers, and query tuning with execution plans.
- Tableau beyond dashboards: data sources, Tableau Prep flows, Metadata API or Tableau Catalog.
- Data modeling: star schemas, SCD Type 2, and medallion layer design.
- Credentials that help: Databricks Certified Data Engineer Professional, or the Databricks platform administrator and architect credentials.
- Results with numbers: for example "migrated 120 stored procedures" or "cut pipeline runtime 60%".

📌 Principal Data Engineer (Hyderabad)
🏢 Qentelli
📍 Hyderabad