About the Role
We are seeking a highly skilled AWS Data Engineer with robust expertise in PySpark, SQL, and AWS Redshift to design, build, and optimize scalable data pipelines and cloud-based data platforms. The ideal candidate should have hands-on experience with AWS data services, large-scale data processing, ETL development, and data warehousing solutions. The role involves working closely with data architects, data scientists, analysts, and business stakeholders to deliver reliable, high-performance data solutions supporting analytics, reporting, and advanced AI/ML initiatives.

Key Responsibilities

- Design, develop, and maintain scalable data pipelines using PySpark and AWS services.
- Build and optimize ETL/ELT workflows for structured and unstructured data.
- Develop and manage enterprise data warehouses using AWS Redshift.
- Write complex and optimized SQL queries for data extraction, transformation, and analysis.
- Ingest data from APIs, databases, flat files, and streaming platforms.
- Perform data cleansing, transformation, validation, and quality checks.
- Optimize Redshift schemas, tables, and queries for performance and scalability.
- Implement data governance, security, and compliance best practices.
- Monitor, troubleshoot, and resolve data pipeline and performance issues.
- Collaborate with cross-functional teams and maintain technical documentation.
- Support production deployments and ongoing data platform operations.



Required Technical Skills Core Data Engineering
- Strong hands-on experience with PySpark.
- Advanced SQL development and performance optimization.
- ETL/ELT pipeline development.
- Data modeling and data warehousing concepts.
- Working knowledge of distributed data processing frameworks. AWS Services
- AWS Redshift Mandatory
- AWS S3 and AWS Glue.
- AWS Lambda and AWS IAM.
- AWS CloudWatch and AWS Step Functions.
- AWS Secrets Manager. Big Data & Processing
- Apache Spark and PySpark.
- Data partitioning and performance tuning.
- Batch and near real-time data processing.
- Data quality and validation frameworks. Database Technologies
- Amazon Redshift, PostgreSQL, and MySQL.
- SQL Server is good to have. DevOps & Automation
- Git, GitHub, or Bitbucket.
- CI/CD pipelines.
- Infrastructure and deployment automation.
- Shell and Python scripting. Preferred Skills
- Experience with Data Lakes and Lakehouse architectures.
- Exposure to Databricks or Amazon EMR.
- Knowledge of Apache Airflow or AWS Managed Workflows for Apache Airflow (MWAA).
- Experience with Kafka or real-time streaming platforms.
- Familiarity with Terraform, CloudFormation, or Infrastructure as Code.
- Exposure to Snowflake or other cloud data warehouse platforms.
- Knowledge of data governance and metadata management tools.
- Experience supporting Analytics, BI, or Machine Learning workloads.

📌 Aws Data Engineer (Chennai)
🏢 Quess IT Solutions
📍 Chennai