Job Description: AWS Data Engineer (Python, PySpark, SQL, Databricks)

Role: AWS Data Engineer

Locations: Bangalore, Hyderabad, Chennai

F2F Interviews

Work Mode: Hybrid

Experience: 4 to 8 years (adjust as needed)

Employment Type: Full-time

About the Role

We are looking for a skilled AWS Data Engineer to design, build, and optimize scalable data pipelines and lakehouse solutions for global clients. You will work with Python, PySpark, SQL, Databricks, and core AWS services to turn raw data into reliable, analytics-ready datasets.

Key Responsibilities

Design, develop, and maintain scalable batch and streaming data pipelines on AWS and Databricks.

Write efficient, production-grade PySpark and Python code for large-scale data transformation.

Build complex SQL queries, views, and data models for analytics and reporting.

Implement lakehouse architecture using Delta Lake (Bronze/Silver/Gold layers).

Optimize Spark jobs for performance and cost (partitioning, caching, broadcast joins, skew handling).

Orchestrate workflows using Databricks Workflows, Airflow, or AWS Step Functions.

Ensure data quality, validation, lineage, and governance (e.g., Unity Catalog).

Work with CI/CD, Git, and infrastructure-as-code practices for deployments.

Collaborate with data scientists, analysts, and client stakeholders to understand requirements.

Troubleshoot production issues and perform root-cause analysis.

Mandatory Skills (all required)

Python: strong programming, OOP, data structures, error handling, unit testing

PySpark:



DataFrames, Spark SQL, window functions, joins, UDFs, performance tuning

SQL: advanced joins, CTEs, window functions, query optimization, data modeling

Databricks: notebooks, jobs and workflows, clusters, Delta Lake, Unity Catalog

AWS: S3, Glue, EMR, Lambda, Redshift, Athena, IAM, CloudWatch, Kinesis/MSK

Good to Have

Apache Airflow or AWS Step Functions

Terraform / CloudFormation

Kafka / Spark Structured Streaming

Data warehousing and dimensional modeling concepts

Docker, Jenkins/GitHub Actions

Databricks or AWS certification

Qualifications

B.E./B.Tech/M.Tech/MCA or equivalent in Computer Science, IT, or a related field.

Proven experience delivering end-to-end data engineering projects on AWS.

Strong analytical, problem-solving, and communication skills.

Interview Process

Round 1: Virtual (Coding and Concepts)

Live coding in Python, PySpark, and SQL

Concept questions on Spark internals, Delta Lake, Databricks, and AWS services

Round 2: Face-to-Face (Scenario and Project Based)

Scenario-based problem solving (pipeline design, optimization, failure handling)

Deep dive into your past projects, architecture decisions, and challenges

Why Join ?

Work on cutting-edge data and AI projects with global clients

Hybrid flexibility across Bangalore, Hyderabad, and Chennai

Strong learning culture and explicit growth path

Pay: ₹1,800,000.00 - ₹3,200,000.00 per year

Benefits:

- Food provided

Work Location: In person

📌 AWS Engineer (Chennai)
🏢 Spotshine Consulting
📍 Chennai