Role Description
Required skills for Senior Data Engineer (Azure Databricks)
- Spark and PySpark ‣ 5+ years building batch and/or streaming pipelines with Spark ‣ Deep understanding of Spark fundamentals: lazy evaluation, transformations vs. actions, parti tions, shuffles, joins, and caching ‣ (Nice to have) Performance tuning: broadcast joins, AQE, skew mitigation, shuffle optimization ‣ (Nice to have) Structured Streaming and Auto Loader for incremental ingestion
- Databricks platform on Azure ‣ Unity Catalog: catalogs/schemas/tables, permissions, external locations, storage credentials, lineage ‣ Databricks Jobs/Workflows: task orchestration, cluster policies, retries, parameters ‣ Git integration (Repos), CI/CD, and setting promotion ‣ (Nice to have) Declarative pipelines with Delta Live Tables (DLT): expectations, CDC, quality rules, event logs
- Cloud storage and access (Azure) ‣ Secret management via Databricks secret scopes and Key Vault integration ‣ (Nice to have) ADLS Gen2: ABFS access, ACLs vs. RBAC, mounts vs. direct access, OAuth/ service principals/managed identity
- Delta Lake ‣ ACID transactions, time travel, MERGE/UPDATE/DELETE ‣ Schema enforcement and evolution ‣ (Nice to have) OPTIMIZE, ZORDER, VACUUM and retention policies ‣ (Nice to have) Compaction and smallfiles mitigation strategies
- SQL and data querying ‣ Complex SQL with CTEs and window functions ‣ (Nice to have) Query optimization and execution plan analysis in Databricks SQL
- Data modeling and architecture ‣ Medallion (bronze/silver/gold) patterns for data lakes ‣ Partitioning and file layout to match read patterns ‣ Designing tables for CDC and incremental processing ‣ (Nice to have) Lakehouse design tradeoffs and table constraints
- Largescale data processing ‣ Cluster sizing, autoscaling, and cost optimization (spot, pools) ‣ Observability: Spark UI, metrics, event logs, Delta logs ‣ (Nice to have) Handling TBscale datasets, throughput vs. latency tradeoffs
- Governance, security, and compliance ‣ (Nice to have) Privilege model in Unity Catalog; table, row, and columnlevel security patterns ‣ (Nice to have) PII handling, masking/anonymization, auditability, lineage
- Software engineering practices ‣ Python packaging, code quality, testing (pytest, chispa), code reviews ‣ CI/CD for Databricks (Repos, asset bundles/wheels), artifact versioning ‣ Documentation and stakeholder communication
- Nice to have (optional) ‣ Scala, dbt, Kafka/Event Hubs, ADF/Airflow, Terraform, Power BI, data mesh concepts, SCD patterns
Skills
PySpark, Azure Data Factory, SQL, Data Lakes
📌 Lead I - Data Engineering (Pune)
🏢 UST
📍 Pune