20 Nov
|
UST
|
Maharashtra
Role description
Required skills for Senior Data Engineer (Azure Databricks) • Spark and PySpark ‣ 5+ years building batch and/or streaming pipelines with Spark ‣ Deep understanding of Spark fundamentals: lazy evaluation, transformations vs. actions, parti tions, shuffles, joins, and caching ‣ (Nice to have) Performance tuning: broadcast joins, AQE, skew mitigation, shuffle optimization ‣ (Nice to have) Structured Streaming and Auto Loader for incremental ingestion • Databricks platform on Azure ‣ Unity Catalog: catalogs/schemas/tables, permissions, external locations, storage credentials, lineage ‣ Databricks Jobs/Workflows: task orchestration, cluster policies, retries, parameters ‣ Git integration (Repos), CI/CD, and workplace promotion ‣ (Nice to have) Declarative pipelines with Delta Live Tables (DLT): expectations, CDC, quality rules, event logs • Cloud storage and access (Azure) ‣ Secret management via Databricks secret scopes and Key Vault integration ‣ (Nice to have) ADLS Gen2: ABFS access, ACLs vs. RBAC, mounts vs. direct access, OAuth/ service principals/managed identity • Delta Lake ‣ ACID transactions, time travel, MERGE/UPDATE/DELETE ‣ Schema enforcement and evolution ‣ (Nice to have) OPTIMIZE, ZORDER, VACUUM and retention policies ‣ (Nice to have) Compaction and smallfiles mitigation strategies • SQL and data querying ‣ Complex SQL with CTEs and window functions ‣ (Nice to have) Query optimization and execution plan analysis in Databricks SQL • Data modeling and architecture ‣ Medallion (bronze/silver/gold)
patterns for data lakes ‣ Partitioning and file layout to match read patterns ‣ Designing tables for CDC and incremental processing ‣ (Nice to have) Lakehouse design tradeoffs and table constraints • Largescale data processing ‣ Cluster sizing, autoscaling, and cost optimization (spot, pools) ‣ Observability: Spark UI, metrics, event logs, Delta logs ‣ (Nice to have) Handling TBscale datasets, throughput vs. latency tradeoffs • Governance, security, and compliance ‣ (Nice to have) Privilege model in Unity Catalog; table, row, and columnlevel security patterns ‣ (Nice to have) PII handling, masking/anonymization, auditability, lineage • Software engineering practices ‣ Python packaging, code quality, testing (pytest, chispa), code reviews ‣ CI/CD for Databricks (Repos, asset bundles/wheels), artifact versioning ‣ Documentation and stakeholder communication • Nice to have (optional) ‣ Scala, dbt, Kafka/Event Hubs, ADF/Airflow, Terraform, Power BI, data mesh concepts, SCD patterns
Skills
PySpark, Azure Data Factory, SQL, Data Lakes
About UST
UST is a global digital transformation solutions provider. For more than 20 years, UST has worked side by side with the world’s best companies to make a real impact through transformation. Powered by technology, inspired by people and led by purpose, UST partners with their clients from design to operation. With deep domain expertise and a future-proof philosophy, UST embeds innovation and agility into their clients’ organizations. With over 30,000 employees in 30 countries, UST builds for boundless impact—touching billions of lives in the process.
📌 Lead I - Data Engineering (Maharashtra)
🏢 UST
📍 Maharashtra