Roles and Responsibilities:
- API & Service Development: Design, build, and maintain high-performance, asynchronous microservices and RESTful APIs using FastAPI and Python.
- Data Pipeline Engineering: Architect, optimize, and scale distributed data pipelines using PySpark on AWS (EMR, Glue, or Databricks).
- AWS Cloud Integration: Deploy and manage cloud-native applications utilizing AWS services including Lambda, S3, ECS/EKS, API Gateway, DynamoDB, RDS, and SQS/SNS.
- Database Management: Write optimized queries, design schemas, and handle integrations across relational (PostgreSQL, MySQL) and NoSQL databases.
- Performance Tuning: Monitor and optimize Spark job performance, memory management, and API latency for high-throughput enterprise workloads.
- CI/CD & DevOps: Implement automated testing, continuous integration, and continuous deployment (CI/CD) pipelines using tools like GitHub Actions, GitLab CI, or Jenkins, backed by Infrastructure as Code (Terraform/CloudFormation).
- Code Quality & Testing: Write clean, modular,
testable code adhering to PEP 8 standards with high unit and integration test coverage (pytest).
Required Qualifications & Skills
- Python: 3+ years of skilled experience with advanced Python (asyncio, type hinting, object-oriented design).
- FastAPI: Hands-on experience developing, documenting (OpenAPI/Swagger), and securing RESTful APIs using FastAPI.
- PySpark: Proven experience developing distributed data processing applications and ETL jobs using PySpark/Apache Spark.
- AWS Cloud: Strong working experience with core AWS services (S3, EMR, Glue, Lambda, ECS/EKS, IAM, CloudWatch).
- Databases & Data Warehousing: Experience with SQL, relational databases (PostgreSQL/MySQL), and data storage formats (Parquet, ORC, Delta Lake).
- Containerization: Proficiency with Docker for containerizing microservices and local development.
- Agile & Version Control: Strong command of Git and experience working within Agile/Scrum delivery frameworks.
📌 Aws Data Engineer (Bengaluru)
🏢 CGI
📍 Bengaluru