We are looking for a Data Engineer with robust SQL, Amazon Redshift, and Apache Airflow skills, along with knowledge of AWS Glue and Amazon EMR. The candidate should be able to understand business requirements, design efficient data solutions, and use AI tools to accelerate development and improve productivity. The ideal candidate should take end-to-end ownership of tasks, work independently, and deliver reliable solutions in phases within agreed timelines.

Key Responsibilities
• Develop, optimize, and maintain complex SQL queries for data processing, transformation, and reporting.
• Build, maintain, schedule, and troubleshoot data pipelines using Apache Airflow DAGs.
• Work extensively with Amazon Redshift, including query optimization, performance tuning, and data warehouse management.
• Understand business requirements and translate them into technical solutions.
• Work with AWS Glue for serverless ETL, data cataloging, and large-scale data transformation.
• Understand Amazon EMR and its use cases for distributed data processing using frameworks such as Apache Spark.
• Evaluate and use the appropriate AWS services based on data volume, performance, scalability, and cost requirements.
• Build and maintain ETL/ELT workflows, data models, and reporting datasets.
• Identify and resolve data quality issues, pipeline failures, and discrepancies in business reports.
• Use AI-powered development tools to accelerate coding, SQL development, debugging, testing, documentation, and troubleshooting.
• Validate AI-generated code and ensure correctness, security, performance, and maintainability before deployment.
• Collaborate with business, analytics, and engineering teams to deliver reliable data solutions.
• Take end-to-end ownership of assigned tasks,



from requirement gathering and development to testing, deployment, and monitoring. Required Skills
• Strong proficiency in SQL, including complex joins, CTEs, window functions, aggregations, and query optimization.
• Hands-on experience with Amazon Redshift and query performance optimization.
• Good experience developing and managing Apache Airflow DAGs.
• Strong understanding of data warehousing, ETL/ELT processes, and data pipeline design.
• Ability to understand business requirements and convert them into practical technical implementations.
• Familiarity with AWS Glue and Amazon EMR, including their advantages, limitations, and appropriate use cases.
• Experience using AI coding assistants and development tools to improve development speed and productivity.
• Strong analytical, debugging, and problem-solving skills.
• Ability to work independently, prioritize tasks, and deliver within agreed timelines. Preferred Skills
• Experience with Python for data processing, automation, and pipeline development.
• Practical experience with AWS Glue jobs, Glue Data Catalog, and serverless ETL workflows.
• Knowledge of Amazon EMR, Apache Spark, and distributed data processing.
• Familiarity with AWS S3 and other AWS data services.
• Experience with Git, version control, code reviews, and deployment practices.
• Experience with data quality checks, pipeline monitoring,



and production issue resolution.
• Familiarity with AI tools such as Claude, ChatGPT, GitHub Copilot, or similar tools for SQL generation, code development, debugging, and optimization. Experience
• 3–6 years of experience in Data Engineering or a related role.
• Strong hands-on experience with SQL, Amazon Redshift, and Apache Airflow is essential.
• Practical knowledge of AWS Glue, Amazon EMR, and AI-assisted development is an advantage. What We Are Looking For
• Business understanding: Can understand the requirement and clarify the expected outcome before starting development.
• End-to-end ownership: Can independently handle tasks from requirement gathering to implementation, testing, and deployment.
• Phased execution: Can break complex requirements into smaller, manageable phases and complete them systematically.
• Fast execution: Uses AI tools effectively to accelerate development without compromising code quality.
• Technical judgment: Knows when to use Redshift, Glue, EMR, or other appropriate services based on the problem.
• Problem-solving: Can identify root causes, resolve issues, and prevent recurring failures.
• Accountability: Delivers accurate, maintainable solutions within agreed timelines with minimal supervision. Key Success Criteria
• Efficient and optimized SQL queries and Redshift workloads.
• Reliable and maintainable Airflow DAGs and data pipelines.
• Appropriate use of AWS Glue and EMR for data processing requirements.
• Faster development cycles through effective use of AI tools.
• Accurate data outputs and timely resolution of production issues.
• Independent execution of business requirements from start to finish.

📌 Data Engineer (Hyderabad)
🏢 upGrad
📍 Hyderabad