Key expectations

Optimize Spark PySpark and SQL workloads to improve performance reliability cost efficiency and scalability

Support production data pipelines through monitoring troubleshooting incident resolution and continuous improvement

Implement data engineering standards CICD practices automated deployment processes unit testing and code quality expectations

Partner with BI Product Engineering and clientfacing teams to translate business and reporting requirements into scalable data solutions

Document technical solutions data flows pipeline logic and operational processes to support knowledge sharing and longterm maintainability

Responsibilities

Design build maintain and optimize data pipelines using Python SQL PySpark Databricks and AWSbased data services

Develop ETLELT processes that support data warehousing analytics reporting and business intelligence use cases

Build and optimize Spark jobs with a focus on performance scalability reliability and productive resource utilization

Design and implement data models for structured semistructured and NoSQL data where applicable

Implement CICD practices automated deployments unit tests and code quality standards for data engineering workflows





Monitor troubleshoot and support production data pipelines resolving issues and recommending improvements

Collaborate with BI Analysts Product Engineering Data and clientfacing teams to understand requirements and support reporting needs

Work Experience Requirements

Minimum of 35 years of experience in Data Engineering preferably working with AWSbased cloud data platforms

Handson experience building maintaining and supporting cloudbased data pipelines

Strong knowledge of PySpark preferably on the Databricks platform

Handson experience with Databricks

Strong proficiency in SQL including query optimization

Strong proficiency in Python

Strong knowledge of data modeling data warehousing ETLELT and analytics concepts

Experience with CICD practices automated deployment processes unit testing and code quality standards

Experience troubleshooting monitoring and supporting production data pipelines

Experience documenting technical solutions data flows and pipeline logic

Mandatory Skills : AWS ML, Data engineering, ETL & integration, Databricks, Prompt Engineering & RAG, Python

Good to Have Skills : AWS AI Services, AWS Bedrock, AWS S3, AWS SageMaker, Data Architecture, Databricks Python

📌 Senior Specialist - Data Sciences (Kolkata)
🏢 LTM
📍 Kolkata