Lead I (Pune)

Lead I (Pune)

19 Nov
|
UST
|
Pune

19 Nov

UST

Pune

Role Description Required skills for Senior Data Engineer (Azure Databricks)
• Spark and PySpark ‣ 5+ years building batch and/or streaming pipelines with Spark ‣ Deep understanding of Spark fundamentals: lazy evaluation, transformations vs. actions, parti tions, shuffles, joins, and caching ‣ (Nice to have) Performance tuning: broadcast joins, AQE, skew mitigation, shuffle optimization ‣ (Nice to have) Structured Streaming and Auto Loader for incremental ingestion
• Databricks platform on Azure ‣ Unity Catalog: catalogs/schemas/tables, permissions, external locations, storage credentials, lineage ‣ Databricks Jobs/Workflows: task orchestration, cluster policies, retries, parameters ‣ Git integration (Repos), CI/CD, and setting promotion ‣ (Nice to have) Declarative pipelines with Delta Live Tables (DLT): expectations, CDC, quality rules, event logs
• Cloud storage and access (Azure) ‣ Secret management via Databricks secret scopes and Key Vault integration ‣ (Nice to have) ADLS Gen2: ABFS access, ACLs vs. RBAC, mounts vs. direct access, OAuth/ service principals/managed identity
• Delta Lake ‣ ACID transactions, time travel, MERGE/UPDATE/DELETE ‣ Schema enforcement and evolution ‣ (Nice to have) OPTIMIZE, ZORDER, VACUUM and retention policies ‣ (Nice to have) Compaction and smallfiles mitigation strategies




• SQL and data querying ‣ Complex SQL with CTEs and window functions ‣ (Nice to have) Query optimization and execution plan analysis in Databricks SQL
• Data modeling and architecture ‣ Medallion (bronze/silver/gold) patterns for data lakes ‣ Partitioning and file layout to match read patterns ‣ Designing tables for CDC and incremental processing ‣ (Nice to have) Lakehouse design tradeoffs and table constraints
• Largescale data processing ‣ Cluster sizing, autoscaling, and cost optimization (spot, pools) ‣ Observability: Spark UI, metrics, event logs, Delta logs ‣ (Nice to have) Handling TBscale datasets, throughput vs. latency tradeoffs
• Governance, security, and compliance ‣ (Nice to have) Privilege model in Unity Catalog; table, row, and columnlevel security patterns ‣ (Nice to have) PII handling, masking/anonymization, auditability, lineage
• Software engineering practices ‣ Python packaging, code quality, testing (pytest, chispa), code reviews ‣ CI/CD for Databricks (Repos, asset bundles/wheels), artifact versioning ‣ Documentation and stakeholder communication
• Nice to have (optional) ‣ Scala, dbt, Kafka/Event Hubs, ADF/Airflow, Terraform, Power BI, data mesh concepts, SCD patterns Skills PySpark, Azure Data Factory, SQL, Data Lakes

📌 Lead I (Pune)
🏢 UST
📍 Pune

Reply to this offer

Impress this employer describing Your skills and abilities, fill out the form below and leave Your personal touch in the presentation letter.

Subscribe to this job alert:

Get the latest job offers by email for: lead i (pune) / pune