Job Description

- Design, develop, and maintain scalable ETL/ELT data pipelines using Azure Data Factory and Azure Databricks.

- Develop data ingestion and transformation workflows using Python, SQL, and PySpark.

- Work with Apache Spark and PySpark for large-scale distributed data processing.

- Implement batch processing, incremental loading, full loading, and Change Data Capture (CDC) mechanisms.

- Integrate data from multiple sources, including relational databases, APIs, and cloud storage.

- Work with Azure Data Lake Storage Gen2 and Azure Blob Storage.

- Develop and optimize data models, fact tables, dimension tables, and data warehouse structures.

- Perform data cleansing, validation, deduplication, and aggregation.

- Implement and manage workflows using Azure Data Factory and Apache Airflow.

- Optimize SQL queries, Spark jobs, and data pipelines for better performance.

- Work with Azure Synapse Analytics and relational databases.

- Implement error handling, logging, monitoring, and alerting for production pipelines.

- Manage version control and CI/CD pipelines using Git, Git Hub, and Azure Dev Ops.

- Troubleshoot production issues, perform root cause analysis, and provide production support.

- Maintain technical documentation and collaborate with cross-functional teams.

Requirements

Programming Languages

- Python, SQL, PySpark.

- Pandas, Num Py, OOP concepts, exception handling, and API integration.

Big Data Technologies

- Apache Spark and PySpark.

- Spark Data Frames, Spark SQL, transformations, actions, joins, aggregations, and window functions.

- Partitioning, caching, and performance optimization.

Azure Cloud Technologies

- Azure Data Factory (ADF).

- Azure Databricks.

- Azure Data Lake Storage Gen2 (ADLS Gen2).

- Azure Blob Storage.

- Azure Synapse Analytics.

- Azure Key Vault and Azure Monitor.

ETL & Data Engineering

- ETL/ELT pipeline development.

- Data ingestion, transformation, and migration.





- Change Data Capture (CDC).

- Incremental loading and full loading.

- Batch and distributed processing.

Database Technologies

- Microsoft SQL Server, PostgreSQL, MySQL, and Oracle.

- Complex SQL queries, indexing, stored procedures, and query optimization.

Data Modeling & Warehousing

- Dimensional and relational data modeling.

- Fact and dimension tables.

- Data normalization and denormalization.

- Schema design and data warehouse development.

Workflow Orchestration & Dev Ops

- Azure Data Factory and Apache Airflow.

- Git, Git Hub, Azure Dev Ops, and CI/CD pipelines.

File Formats

- CSV, JSON, XML, Parquet, and Delta Lake.

- 4–5 years of relevant experience in Data Engineering.

- Solid hands-on experience with Azure Data Factory, Azure Databricks, PySpark, Python, and SQL.

- Experience developing and optimizing scalable data pipelines.

- Good understanding of cloud-based data engineering and data warehousing concepts.

- Strong analytical, debugging, and problem-solving skills.

- Good communication and collaboration skills.

Requirements
Programming Languages • Python, SQL, PySpark. • Pandas, Num Py, OOP concepts, exception handling, and API integration. Big Data Technologies • Apache Spark and PySpark. • Spark Data Frames, Spark SQL, transformations, actions, joins, aggregations, and window functions. • Partitioning, caching, and performance optimization. Azure Cloud Technologies • Azure Data Factory (ADF). • Azure Databricks. • Azure Data Lake Storage Gen2 (ADLS Gen2). • Azure Blob Storage. • Azure Synapse Analytics. • Azure Key Vault and Azure Monitor. ETL & Data Engineering • ETL/ELT pipeline development. • Data ingestion, transformation, and migration. • Change Data Capture (CDC). • Incremental loading and full loading. • Batch and distributed processing. Database Technologies • Microsoft SQL Server, PostgreSQL, MySQL, and Oracle. • Complex SQL queries, indexing, stored procedures, and query optimization.

📌 Senior Data Engineer (India)
🏢 InFynd
📍 India