AWS Senior Data Engineer (Kafka & PySpark)

Experience: 10-15 Years

Location: Chennai, Bengaluru, Hyderabad

We are seeking an experienced AWS Data Engineer with strong expertise in Kafka, PySpark, Spark SQL, Python, and AWS to build and optimize large-scale batch and real-time data platforms.

Key Skills

- PySpark, Spark SQL, Python, SQL
- Apache Kafka (Kafka Connect, Schema Registry, Streaming)
- AWS: S3, Glue, EMR, Lambda, IAM, Redshift, Step Functions, CloudWatch
- ETL/ELT & Real-Time Data Pipelines
- Spark Structured Streaming
- Data Modeling, Data Lakes & Lakehouse Architecture
- CI/CD, Git, Jenkins/GitHub Actions
- Data Quality, Governance & Security

Responsibilities

- Design and develop scalable batch and streaming data pipelines on AWS.
- Build high-performance PySpark and Spark SQL applications.




- Implement Kafka-based event-driven and real-time streaming solutions.
- Integrate and optimize AWS data services for reliability and cost efficiency.
- Implement data quality, governance, monitoring, and security best practices.
- Build CI/CD pipelines and support production deployments.
- Troubleshoot performance bottlenecks and production issues.

Valuable to Have

- Amazon MSK, Kinesis
- Databricks, Delta Lake, Unity Catalog
- Airflow / MWAA
- Snowflake, DBT
- Terraform

Primary Skills: AWS Data Engineering, Kafka, PySpark, Spark SQL, Python, Streaming Data Engineering.

📌 AWS Senior Data Engineer (Chennai)
🏢 Tata Consultancy Services
📍 Chennai