- Experience in design, development and deployment of big data applications and ETL jobs using PySpark APIs/sparkSQL.
- Experience in design, build and deployment of Python based applications.
- Experience in writing complex SQL queries/procedures using Relational databases like SQL Server or Oracle
- Experience in version control system like Git and CI/CD pipeline is a must.
- Experience in Delta lake APIs is a plus
- Experience in Docker and Kubernetes is a plus
- Knowledge of AWS services like S3, Athena, Glue, Lambda, Redshift or Cloud platform is a plus
Responsibilities:
- Designing, implementing, and maintaining jobs written in PySpark for extracting raw data, applying transformation, and writing into file/tables on Apache Spark cluster.
- Writing complex queries as per the business rules and data extraction logic
- Query tuning and Performance optimization of various components as and when required.
- Work with development leads, System Architects and other teams (as needed) to manage dependencies, risks and issues.
- Contributing in all phases of the development lifecycle
- Support testers and defect resolution
- Update status/risks/issues/impediments in daily scrum calls to the customer and track it to closure.
- Writing testable, scalable and productive code
- Working on the necessary documentations as needed