Job Summary:

We are seeking an experienced AWS Data Engineer with strong expertise in Databricks, PySpark, and cloud-based data engineering solutions. The ideal candidate will be responsible for designing, developing, and maintaining scalable data pipelines, data lakes, and data warehouse solutions on AWS. The role requires hands-on experience in processing large-scale datasets, optimizing data workflows, and supporting analytics and business intelligence initiatives.

Key Responsibilities:

- Design, develop, and maintain scalable ETL/ELT data pipelines using PySpark and Databricks.
- Build and optimize data ingestion frameworks from various structured and unstructured data sources.
- Develop and manage data lakes and data warehouse solutions on AWS.
- Implement data transformation, cleansing, validation, and enrichment processes.
- Work with large datasets and optimize Spark jobs for performance and scalability.
- Integrate data from APIs, databases, streaming platforms, and third-party systems.
- Collaborate with Data Scientists, Analysts, and Business stakeholders to understand data requirements.
- Monitor, troubleshoot, and enhance data pipeline performance and reliability.
- Implement data quality, governance, security, and compliance best practices.
- Participate in code reviews, architecture discussions, and technical design sessions.

Required Skills:





- Strong experience in AWS Cloud Services (S3, EMR, Glue, Lambda, IAM, Redshift, Athena, CloudWatch).
- Hands-on experience with Databricks and Apache Spark.
- Strong programming skills in PySpark and Python.
- Experience in building ETL/ELT pipelines and data integration solutions.
- Strong knowledge of SQL and relational databases.
- Experience with Delta Lake, Spark SQL, and performance tuning.
- Understanding of data modeling concepts and data warehousing principles.
- Experience with version control systems such as Git.
- Knowledge of CI/CD practices and deployment automation.

Preferred Skills:

- Experience with Apache Kafka or other streaming technologies.
- Knowledge of Airflow, AWS Step Functions, or workflow orchestration tools.
- Experience with Snowflake or Redshift.
- Familiarity with Terraform or Infrastructure as Code (IaC).
- Exposure to machine learning data pipelines and MLOps.
- AWS Certification is a plus.

Educational Qualification:

- Bachelor's or Master's degree in Computer Science, Information Technology, Engineering, or a related field.

Key Competencies:

- Strong analytical and problem-solving skills.
- Excellent communication and stakeholder management abilities.
- Ability to work independently and in a team-oriented environment.
- Strong focus on performance optimization and data quality.

📌 Aws Data Engineer (Hyderabad)
🏢 Shrewd TechLink Services
📍 Hyderabad