- Design, develop, and maintain scalable ETL/ELT data pipelines using Azure Data Factory and Azure Databricks.

- Develop data ingestion and transformation workflows using Python, SQL, and PySpark.

- Work with Apache Spark and PySpark for large-scale distributed data processing.

- Implement batch processing, incremental loading, full loading, and Change Data Capture (CDC) mechanisms.

- Integrate data from multiple sources, including relational databases, APIs, and cloud storage.

- Work with Azure Data Lake Storage Gen2 and Azure Blob Storage.

- Develop and optimize data models, fact tables, dimension tables, and data warehouse structures.

- Perform data cleansing, validation, deduplication, and aggregation.

- Implement and manage workflows using Azure Data Factory and Apache Airflow.

- Optimize SQL queries, Spark jobs, and data pipelines for better performance.

- Work with Azure Synapse Analytics and relational databases.

- Implement error handling, logging, monitoring, and alerting for production pipelines.

- Manage version control and CI/CD pipelines using Git, GitHub, and Azure DevOps.

- Troubleshoot production issues, perform root cause analysis, and provide production support.

- Maintain technical documentation and collaborate with cross-functional teams.

Requirements

Programming Languages

- Python, SQL, PySpark.
- Pandas, NumPy, OOP concepts, exception handling, and API integration.

Big Data Technologies

- Apache Spark and PySpark.
- Spark DataFrames, Spark SQL, transformations, actions, joins, aggregations,



and window functions.
- Partitioning, caching, and performance optimization.

Azure Cloud Technologies

- Azure Data Factory (ADF).
- Azure Databricks.
- Azure Data Lake Storage Gen2 (ADLS Gen2).
- Azure Blob Storage.
- Azure Synapse Analytics.
- Azure Key Vault and Azure Monitor.

ETL & Data Engineering

- ETL/ELT pipeline development.
- Data ingestion, transformation, and migration.
- Change Data Capture (CDC).
- Incremental loading and full loading.
- Batch and distributed processing.

Database Technologies

- Microsoft SQL Server, PostgreSQL, MySQL, and Oracle.
- Complex SQL queries, indexing, stored procedures, and query optimization.

Data Modeling & Warehousing

- Dimensional and relational data modeling.
- Fact and dimension tables.
- Data normalization and denormalization.
- Schema design and data warehouse development.

Workflow Orchestration & DevOps

- Azure Data Factory and Apache Airflow.
- Git, GitHub, Azure DevOps, and CI/CD pipelines.

File Formats

- CSV, JSON, XML, Parquet, and Delta Lake.

- 4–5 years of relevant experience in Data Engineering.
- Strong hands-on experience with Azure Data Factory, Azure Databricks, PySpark, Python, and SQL.
- Experience developing and optimizing scalable data pipelines.
- Good understanding of cloud-based data engineering and data warehousing concepts.
- Strong analytical, debugging, and problem-solving skills.
- Valuable communication and collaboration skills.

📌 Senior Data Engineer (Tamil Nadu)
🏢 InFynd
📍 Tamil Nadu