Role Description Required skills for Senior Data Engineer (Azure Databricks)
• Spark and PySpark ‣ 5+ years building batch and/or streaming pipelines with Spark ‣ Deep understanding of Spark fundamentals: lazy evaluation, transformations vs. actions, parti tions, shuffles, joins, and caching ‣ (Nice to have) Performance tuning: broadcast joins, AQE, skew mitigation, shuffle optimization ‣ (Nice to have) Structured Streaming and Auto Loader for incremental ingestion
• Databricks platform on Azure ‣ Unity Catalog: catalogs/schemas/tables, permissions, external locations, storage credentials, lineage ‣ Databricks Jobs/Workflows: task orchestration, cluster policies, retries, parameters ‣ Git integration (Repos), CI/CD, and setting promotion ‣ (Nice to have) Declarative pipelines with Delta Live Tables (DLT): expectations, CDC, quality rules, event logs
• Cloud storage and access (Azure) ‣ Secret management via Databricks secret scopes and Key Vault integration ‣ (Nice to have) ADLS Gen2: ABFS access, ACLs vs. RBAC, mounts vs. direct access, OAuth/ service principals/managed identity
• Delta Lake ‣ ACID transactions, time travel, MERGE/UPDATE/DELETE ‣ Schema enforcement and evolution ‣ (Nice to have) OPTIMIZE, ZORDER, VACUUM and retention policies ‣ (Nice to have) Compaction and smallfiles mitigation strategies
• SQL and data querying ‣ Complex SQL with CTEs and window functions ‣ (Nice to have) Query optimization and execution plan analysis in Databricks SQL
• Data modeling and architecture ‣ Medallion (bronze/silver/gold) patterns for data lakes ‣ Partitioning and file layout to match read patterns ‣ Designing tables for CDC and incremental processing ‣ (Nice to have) Lakehouse design tradeoffs and table constraints
• Largescale data processing ‣ Cluster sizing, autoscaling, and cost optimization (spot, pools) ‣ Observability: Spark UI, metrics, event logs, Delta logs ‣ (Nice to have) Handling TBscale datasets, throughput vs. latency tradeoffs
• Governance, security, and compliance ‣ (Nice to have) Privilege model in Unity Catalog; table, row, and columnlevel security patterns ‣ (Nice to have) PII handling, masking/anonymization, auditability, lineage
• Software engineering practices ‣ Python packaging, code quality, testing (pytest, chispa), code reviews ‣ CI/CD for Databricks (Repos, asset bundles/wheels), artifact versioning ‣ Documentation and stakeholder communication
• Nice to have (optional) ‣ Scala, dbt, Kafka/Event Hubs, ADF/Airflow, Terraform, Power BI, data mesh concepts, SCD patterns Skills PySpark, Azure Data Factory, SQL, Data Lakes
📌 Lead I (Pune)
🏢 UST
📍 Pune