Responsibilities

- Design, develop, and maintain scalable data pipelines using PySpark, Spark, Databricks, Azure Data Factory (ADF), and AWS Glue.

- Build and optimize ETL/ELT workflows for processing large-scale structured and unstructured data.

- Develop and manage cloud-based data solutions on AWS, Azure, and GCP platforms.

- Work with Databricks, Delta Lake, ADLS, S3, BigQuery, Redshift, and Snowflake for data ingestion, transformation, and storage.

- Implement data quality, monitoring, and performance optimization best practices.

- Collaborate with data analysts, data scientists, and business stakeholders to understand requirements and deliver scalable solutions.

- Ensure data security, governance, and compliance standards are followed across cloud platforms.

- Optimize Spark jobs and SQL queries for improved performance and cost efficiency.

- Support real-time and batch data processing frameworks.

- Participate in code reviews, troubleshooting, and production support activities.

- Create technical documentation and maintain data engineering standards and best practices.

📌 Data Engineer | PySpark | Databricks | Azure Data Factory (Bengaluru)
🏢 Infosys
📍 Bengaluru