Job Description - Senior AWS Engineer
Design and implement distributed data processing on AWS EMR using PySpark for batch/stream workloads.
Build robust, scalable ETL/ELT pipelines integrating diverse data sources (files, RDBMS, APIs, streaming).
Optimize Spark jobs (partitioning, caching, broadcast joins), and tune EMR clusters (instance types, autoscaling, bootstrap actions).
Implement data lake patterns on S3 with appropriate file formats (Parquet/ORC), schema evolution, and table management.
Orchestrate workflows using AWS Step Functions, Airflow, or EMR managed workflow, and automate CI/CD.
Ensure cost governance (spot instances, right-sizing, scaling policies) and observability (CloudWatch, Spark metrics).
Enforce security & compliance (IAM, KMS, encryption at rest/in transit, VPC, SGs, Glue Catalog access controls).
Collaborate with analytics teams to expose curated datasets to Redshift, Athena, Glue, or downstream BI tools.
Troubleshoot performance bottlenecks and production issues; implement resiliency patterns and SLAs.
Document architecture, standards, runbooks; mentor junior engineers.

📌 Aws Emr,python With Pyspark Kochi (India)
🏢 Tata Consultancy Services
📍 India