Role Description
Required skills for Senior Data Engineer (Azure Databricks)
- Spark and PySpark ‣ 5+ years building batch and/or streaming pipelines with Spark ‣
- Deep understanding of Spark fundamentals: lazy evaluation, transformations vs. actions, parti tions, shuffles, joins, and caching ‣ (Nice to have) Performance tuning: broadcast joins, AQE, skew mitigation, shuffle optimization ‣ (Nice to have) Structured Streaming and Auto Loader for incremental ingestion
- Databricks platform on Azure ‣
- Unity Catalog: catalogs/schemas/tables, permissions, external locations, storage credentials, lineage ‣
- Databricks Jobs/Workflows: task orchestration, cluster policies, retries, parameters ‣
- Git integration (Repos), CI/CD, and setting promotion ‣ (Nice to have) Declarative pipelines with Delta Live Tables (DLT): expectations, CDC, quality rules, event logs
- Cloud storage and access (Azure) ‣
- Secret management via Databricks secret scopes and Key Vault integration ‣ (Nice to have) ADLS Gen2: ABFS access, ACLs vs. RBAC, mounts vs. direct access, OAuth/ service principals/managed identity
- Delta Lake ‣
- ACID transactions, time travel, MERGE/UPDATE/DELETE ‣
- Schema enforcement and evolution ‣ (Nice to have) OPTIMIZE, ZORDER, VACUUM and retention policies ‣ (Nice to have) Compaction and smallfiles mitigation strategies
- SQL and data querying ‣
- Complex SQL with CTEs and window functions ‣ (Nice to have) Query optimization and execution plan analysis in Databricks SQL
- Data modeling and architecture ‣
- Medallion (bronze/silver/gold) patterns for data lakes ‣
- Partitioning and file layout to match read patterns ‣
- Designing tables for CDC and incremental processing ‣ (Nice to have) Lakehouse design tradeoffs and table constraints
- Largescale data processing ‣
- Cluster sizing, autoscaling, and cost optimization (spot, pools) ‣
- Observability: Spark UI, metrics, event logs, Delta logs ‣ (Nice to have) Handling TBscale datasets, throughput vs. latency tradeoffs
- Governance, security, and compliance ‣ (Nice to have) Privilege model in Unity Catalog; table, row, and columnlevel security patterns ‣ (Nice to have) PII handling, masking/anonymization, auditability, lineage
- Software engineering practices ‣
- Python packaging, code quality, testing (pytest, chispa), code reviews ‣
- CI/CD for Databricks (Repos, asset bundles/wheels), artifact versioning ‣
- Documentation and stakeholder communication
- Nice to have (optional) ‣
- Scala, dbt, Kafka/Event Hubs, ADF/Airflow, Terraform, Power BI, data mesh concepts, SCD patterns
Skills PySpark, Azure Data Factory, SQL, Data Lakes
📌 Lead I - Data Engineering (Pune)
🏢 UST
📍 Pune