Data Engineer (Ahmedabad)

Data Engineer (Ahmedabad)

19 Nov
|
Calidad Infotech
|
Ahmedabad

19 Nov

Calidad Infotech

Ahmedabad

About the role

We need a hands-on Data Engineer with 5-7 years of experience to build and run batch and real-time data pipelines on Databricks. The work centres on PySpark and Structured Streaming, preferably on the Azure data stack. This is a build role, not a support or reporting role. Candidates must have written production PySpark code and owned streaming pipelines end to end.

Key responsibilities

Design, build and maintain batch and streaming pipelines on Databricks using PySpark.

Develop real-time ingestion and processing with Spark Structured Streaming, including checkpointing, watermarking and late-data handling.

Build and manage Delta Lake tables across bronze, silver and gold layers.

Tune Spark jobs for performance and cost: partitioning, joins, caching, cluster sizing.

Model data for analytics and downstream consumers.

Orchestrate and schedule workflows, and monitor pipelines for failures and data quality issues.

Put pipelines through version control, code review and CI/CD.

Work with data architects, analysts and business stakeholders to turn requirements into working pipelines.





Must-have skills

5-7 years of relevant, hands-on data engineering experience. Databricks: notebooks, jobs and workflows, clusters, Delta Lake. PySpark: DataFrame API, Spark SQL, UDFs, performance tuning. Structured Streaming: production streaming pipelines, with stateful processing, triggers, checkpoints and watermarks.

Core data engineering: ETL/ELT design, data modelling, data warehousing and lakehouse concepts, incremental loads and change data capture, data quality checks.

Solid SQL and Python.

Git-based development.

Preferred skills (Azure stack)

Azure Databricks and Azure Data Lake Storage Gen2.

Azure Data Factory or Databricks Workflows for orchestration.

Azure Event Hubs or Kafka as streaming sources.

Unity Catalog, Delta Live Tables and Auto Loader.

Azure Synapse Analytics or Microsoft Fabric.

Azure DevOps or GitHub Actions for CI/CD.

Databricks or Azure data engineering certification.

📌 Data Engineer (Ahmedabad)
🏢 Calidad Infotech
📍 Ahmedabad

Reply to this offer

Impress this employer describing Your skills and abilities, fill out the form below and leave Your personal touch in the presentation letter.

Subscribe to this job alert:

Get the latest job offers by email for: data engineer (ahmedabad) / ahmedabad