We are hiring a Data Engineer with strong hands-on Azure experience to build and run the pipelines behind our analytics products. You will ingest large public and partner datasets, land them in Azure Blob Storage, transform them in ClickHouse and PostgreSQL, and orchestrate it all with Azure Data Factory and Azure Batch. You will own pipelines end to end: ingestion, modelling, data quality, monitoring and cost. We build with AI coding tools every day, and we expect you to as well.

Key responsibilities
• Build and maintain Python ETL pipelines (pandas, PyArrow, DuckDB) that ingest bulk files, APIs and scraped web sources, including legacy and fixed-width formats.
• Design and operate Azure Data Factory pipelines: scheduling, dependencies between monthly and daily loads, parameterisation, retries and failure alerts.
• Run containerised Python workloads on Azure Batch and Azure Container Registry, and manage pool sizing, quotas and node cost.
• Organise raw and curated data in Azure Blob Storage / ADLS Gen2 with explicit partitioning, versioned drops and retention rules.
• Model data in ClickHouse (MergeTree engines, partitioning, ordering keys, materialised views) and PostgreSQL for application-facing data.
• Build staging, intermediate and mart layers with dbt, including incremental and full-refresh strategies and audit tables that track rows added, updated and deleted per run.
• Write data quality checks and reconciliations against source systems, and investigate when numbers drift.
• Secure pipelines with Managed Identity, Key Vault and least-privilege RBAC; deploy through Azure DevOps CI/CD and Terraform.
• Monitor Azure spend and pipeline run times, and tune queries, storage and compute to keep both down.
• Work with analysts,



product and app teams to turn business questions into reliable tables and views. Required skills & qualifications
• 3–5 years building production data pipelines.
• Solid hands-on Azure experience (2+ years) with Data Factory, Blob Storage / ADLS Gen2, Azure Batch or Functions, Key Vault, Managed Identity, RBAC, networking basics and cost management.
• Strong Python for data work: pandas or PyArrow, packaging, logging, error handling and tests.
• Advanced SQL: window functions, CTEs, deduplication, slowly changing data and query tuning.
• Hands-on PostgreSQL: schema design, indexing, and bulk loading.
• Data modelling for analytics: star schemas, facts and dimensions, grain and keys.
• Git and CI/CD (Azure DevOps or GitHub Actions); Docker.
• Daily use of AI coding assistants (for example Claude Code, GitHub Copilot or Cursor) to write, test and refactor pipelines, with the judgement to review and verify what they produce.
• Clear written communication: you document pipelines and explain data issues to non-engineers.
• ClickHouse in production: table engines, partitions, mutations, and loading with clickhouse-connect.
• Snowflake, Databricks or another cloud warehouse.
• dbt (Core or Cloud), including macros, tests and incremental models. Nice to have
• Terraform or Bicep for Azure infrastructure as code.
• Web scraping and automation with Playwright for sources that have no API.
• SQL Server / Azure SQL and ODBC integrations.
• DuckDB for local and file-based processing; Parquet and columnar formats.
• Power BI or another BI tool on top of your models.
• Experience with oil and gas, energy, or government regulatory data (for example Texas RRC).
• Azure certifications such as DP-203 / DP-700 or AZ-104.

📌 Data Engineer (Coimbatore)
🏢 MangoBytes AI
📍 Coimbatore