Job Title: Senior Data Engineer
Location: Noida/Ahmedabad
Employment Type: Full-time
Experience Required: 8 - 12 Years
About the Role/Project
The role involves maintaining and developing data pipelines, serverless migration, and CI/CD process ownership. The solution product line encompasses several key areas, including marketing analytics, category optimisation, pricing optimisation, customer experience improvement, and sales performance enhancement.
Key Responsibilities
- Adapt existing pipelines and code for serverless environments, ensuring stable execution
- Maintain and enhance CI/CD processes, including proposing architectural improvements
- Monitor pipelines and repositories, review merge requests, and oversee code changes
- Develop and maintain data pipelines while understanding underlying business logic
Required Skills & Qualifications
- 8 years of experience in Data Engineering
- Strong experience with Databricks and Delta Lake
- Strong experience designing Lakehouse Architecture
- Strong experience with PySpark and Spark SQL
- Experience building configuration-driven data pipelines.
- Strong Python programming skills with experience developing reusable Python libraries/packages.
- Experience with Delta table optimisation and maintenance.
- Recent project experience on the Databricks ecosystem is mandatory.
- Experience with serverless architectures
- Understanding of pricing or financial data domains
Preferred / Positive to Have
- Experience with Azure Data Platform.
- Knowledge of CI/CD for data pipelines.
- Performance tuning and optimisation of large-scale data pipelines.