Key Responsibilities

1. Data Architecture and Pipeline Design

- Architect end-to-end data pipelines supporting enterprise BI, reporting, analytics, and data warehousing requirements.

- Design scalable data ingestion, transformation, storage, and serving architectures across multiple data sources and destinations.

- Define appropriate approaches for batch processing, incremental ingestion, CDC, historical data loads, and data synchronization.

- Design data lake and lakehouse architectures, including raw, curated, and analytics-ready data layers.

- Make technical decisions on processing frameworks, storage formats, orchestration, data modelling, and pipeline reliability.

2.

Complex Data

Integrations

- Build integrations across heterogeneous enterprise systems, including relational databases, cloud platforms, APIs, SaaS applications, files, and legacy systems.

- Handle diverse data formats, inconsistent schemas, changing source structures, large data volumes, and complex transformation requirements.

- Implement incremental extraction, change detection, deduplication, reconciliation, and cross-source data consistency.

- Resolve integration challenges involving API pagination, rate limits, authentication, schema evolution, late-arriving data, and failed or partial loads.

- Develop reusable ingestion frameworks and standardized integration components to accelerate future implementations.

3.

Cloud Data

Engineering

- Design and implement cloud-based data platforms using GCP and/or Microsoft Azure.

- Develop data pipelines using technologies such as PySpark, Apache Spark, BigQuery, Cloud Storage, Dataproc, Cloud Data Fusion, Azure Data Factory, Azure Data Lake Storage, and Azure Databricks.

- Integrate cloud storage, processing engines, relational databases, and downstream analytics platforms.

- Evaluate technology choices based on scalability, maintainability, security, processing performance, and cloud costs.

4. BI and Analytics Data Platforms

- Build and maintain curated datasets and analytical data models for Power BI and other BI platforms.





- Design data structures that support reporting performance, historical analysis, dimensional modelling, and consistent business metrics.

- Understand the complete data lifecycle from source-system extraction to transformation, warehouse modelling, and BI consumption.

- Work with analytics teams to translate reporting requirements into robust, reusable data pipelines.

- Optimize data refresh strategies, incremental processing, and downstream query performance.

5. PySpark and Distributed Processing

- Develop production-grade data processing applications using Python, PySpark, and Apache Spark.

- Optimize large-scale transformations, joins, shuffles, partitioning, caching, and file layouts.

- Troubleshoot memory issues, skewed data, slow-running jobs, and distributed processing failures.

- Work with Parquet, Avro, JSON, CSV, and other structured or semi-structured formats.

- Apply sound engineering practices to build modular, testable, maintainable pipeline components.

6.

Production

Operations and Reliability

- Own pipelines throughout their lifecycle: design, development, deployment, monitoring, optimization, and support.

- Implement orchestration, dependency management, retries, checkpoints, idempotency, and recovery mechanisms.

- Set up logging, alerting, data quality validation, reconciliation, and failure notifications.

- Investigate production incidents and identify permanent corrective actions.

- Implement CI/CD and version control practices for controlled pipeline releases.

- Maintain technical documentation, architecture diagrams, and operational runbooks.

Mandatory Technical Experience





- Proven experience architecting and delivering complex, production-grade data pipelines.

- Strong hands-on expertise in Python, PySpark, Apache Spark, and SQL.

- Demonstrated experience integrating multiple enterprise data sources and systems.

- Experience building data platforms specifically for BI, analytics, or enterprise reporting.

- Practical experience with GCP or Microsoft Azure data engineering services.

- Strong understanding of data lake, lakehouse, data warehouse, and dimensional modelling concepts.

- Experience with orchestration, monitoring, production troubleshooting, and performance optimization.

- Ability to independently design solutions and implement them hands-on.

Preferred Experience

- Experience across both GCP and Azure.

- BigQuery, Cloud Data Fusion, Cloud Composer, Dataproc, Azure Data Factory, Azure Databricks, or Microsoft Fabric.

- Experience integrating APIs, transactional databases, SaaS platforms, and legacy enterprise systems.

- CDC, streaming data, schema evolution, and large-scale incremental processing.

- Power BI data preparation and refresh architecture.

What We Are Looking For

- An engineer who understands architecture but is equally comfortable writing and debugging production code.

- Someone who has solved real-world data integration challenges, not just developed isolated SQL transformations.

- Someone who can independently own complex client implementations from architecture through production support.

- Robust analytical thinking, troubleshooting ability, technical communication, and engineering discipline.

What This Role Is Not

- A SQL-only ETL developer role.

- A role limited to building reports or dashboards.

- A role focused exclusively on low-code ETL configuration.

- A role where pipeline development is handed off to someone else for deployment and maintenance.

Candidates will be evaluated on their ability to design, implement, optimize, and operate complete data integration solutions.

- Data Engineering

📌 Data Engineer- GCP/Azure (Chandigarh)
🏢 Innovantes
📍 Chandigarh