19 Nov
|
Calidad Infotech
|
Ahmedabad
19 Nov
Calidad Infotech
Ahmedabad
About the role
We need a hands-on Data Engineer with 5-7 years of experience to build and run batch and real-time data pipelines on Databricks. The work centres on PySpark and Structured Streaming, preferably on the Azure data stack. This is a build role, not a support or reporting role. Candidates must have written production PySpark code and owned streaming pipelines end to end.
Key responsibilities
Design, build and maintain batch and streaming pipelines on Databricks using PySpark.
Develop real-time ingestion and processing with Spark Structured Streaming, including checkpointing, watermarking and late-data handling.
Build and manage Delta Lake tables across bronze, silver and gold layers.
Tune Spark jobs for performance and cost: partitioning, joins, caching, cluster sizing.
Model data for analytics and downstream consumers.
Orchestrate and schedule workflows, and monitor pipelines for failures and data quality issues.
Put pipelines through version control, code review and CI/CD.
Work with data architects, analysts and business stakeholders to turn requirements into working pipelines.
Must-have skills
5-7 years of relevant, hands-on data engineering experience. Databricks: notebooks, jobs and workflows, clusters, Delta Lake. PySpark: DataFrame API, Spark SQL, UDFs, performance tuning. Structured Streaming: production streaming pipelines, with stateful processing, triggers, checkpoints and watermarks.
Core data engineering: ETL/ELT design, data modelling, data warehousing and lakehouse concepts, incremental loads and change data capture, data quality checks.
Solid SQL and Python.
Git-based development.
Preferred skills (Azure stack)
Azure Databricks and Azure Data Lake Storage Gen2.
Azure Data Factory or Databricks Workflows for orchestration.
Azure Event Hubs or Kafka as streaming sources.
Unity Catalog, Delta Live Tables and Auto Loader.
Azure Synapse Analytics or Microsoft Fabric.
Azure DevOps or GitHub Actions for CI/CD.
Databricks or Azure data engineering certification.
📌 Data Engineer (Ahmedabad)
🏢 Calidad Infotech
📍 Ahmedabad