Role- AWS Data Engineer (Kafka & PySpark)

Experience- 4-8 years

Location- Kolkata , Bangalore, Chennai

-

We are looking for an experienced AWS Data Engineer with strong hands-on expertise in PySpark and Apache Kafka to design, develop, and maintain scalable batch and real-time data pipelines on AWS.

Roles & Responsibilities

- Design and develop scalable ETL/ELT data pipelines using PySpark and AWS services.
- Build distributed data-processing solutions using PySpark/Spark on Amazon EMR. [#Personal#...Interview | Outlook]
- Develop real-time/streaming data pipelines using Kafka and Spark/PySpark.
- Implement data lake solutions using Amazon S3 and formats such as Parquet/ORC. [#Personal#...Interview | Outlook]
- Optimize Spark jobs through partitioning, caching, broadcast joins, and cluster tuning. [#Personal#...Interview | Outlook]
- Integrate data from files, databases, APIs, and streaming sources.
- Work with AWS services such as S3,



EMR, Glue, Redshift, IAM, CloudWatch, and Step Functions. [#Personal#...Interview | Outlook]
- Implement workflow orchestration using Airflow / AWS Step Functions.
- Monitor pipelines, troubleshoot production issues, and improve performance and reliability.
- Implement security controls using IAM, encryption/KMS, VPC, and appropriate access controls. [#Personal#...Interview | Outlook]

Mandatory Skills

- AWS Data Engineering
- Apache Kafka
- PySpark
- Apache Spark
- Python
- SQL
- Amazon S3
- AWS EMR
- AWS Glue
- Amazon Redshift
- AWS Step Functions / Airflow

Valuable to Have

- Spark Structured Streaming
- Data Lake / Lakehouse concepts
- Parquet / Avro / ORC
- CI/CD
- Databricks
- Data quality and monitoring

📌 AWS Data Engineer (Kafka & PySpark) (Bengaluru)
🏢 Tata Consultancy Services
📍 Bengaluru