Responsibilities

- Design, develop, and maintain scalable data pipelines using PySpark, Spark, Databricks, Azure Data Factory (ADF), and AWS Glue.
- Build and optimize ETL/ELT workflows for processing large-scale structured and unstructured data.
- Develop and manage cloud-based data solutions on AWS, Azure, and GCP platforms.
- Work with Databricks, Delta Lake, ADLS, S3, BigQuery, Redshift, and Snowflake for data ingestion, transformation, and storage.
- Implement data quality, monitoring, and performance optimization best practices.
- Collaborate with data analysts, data scientists, and business stakeholders to understand requirements and deliver scalable solutions.
- Ensure data security, governance, and compliance standards are followed across cloud platforms.
- Optimize Spark jobs and SQL queries for improved performance and cost efficiency.
- Support real-time and batch data processing frameworks.
- Participate in code reviews, troubleshooting, and production support activities.
- Create technical documentation and maintain data engineering standards and best practices.

📌 Data Engineer | PySpark | Databricks | Azure Data Factory (Hyderabad)
🏢 Infosys
📍 Hyderabad