- Senior AWS Engineer

- Design and implement distributed data processing on AWS EMR using PySpark for batch/stream workloads.

- Build robust, scalable ETL/ELT pipelines integrating diverse data sources (files, RDBMS, APIs, streaming).

- Optimize Spark jobs (partitioning, caching, broadcast joins), and tune EMR clusters (instance types, autoscaling, bootstrap actions).

- Implement data lake patterns on S3 with appropriate file formats (Parquet/ORC), schema evolution, and table management.

- Orchestrate workflows using AWS Step Functions, Airflow, or EMR managed workflow, and automate CI/CD.

- Ensure cost governance (spot instances, right-sizing, scaling policies) and observability (CloudWatch, Spark metrics).

- Enforce security & compliance (IAM, KMS, encryption at rest/in transit, VPC, SGs, Glue Catalog access controls).

- Collaborate with analytics teams to expose curated datasets to Redshift, Athena, Glue, or downstream BI tools.

- Troubleshoot performance bottlenecks and production issues; implement resiliency patterns and SLAs.

- Document architecture, standards, runbooks; mentor junior engineers.

📌 AWS EMR,Python with Pyspark (Chennai)
🏢 Tata Consultancy Services
📍 Chennai