Leverage AWS services to ingest data from source systems within their business domain
Leverage databricks platform to process, clean and transform data
Ensure and measure data quality according to business requirements and tracked KPIs
Monitor and maintain existing data product such as data ingestion pipeline and ETLs
Alert and act on data quality issues and system incidents within their business domain
Ensure that data is available, accurate and fit for purpose following governance guidelines and compliance policies
Ensure that knowledge of data mesh framework is shared
Write and publish clear documentation of approaches and functionalities
Take part in all agile ceremonies (planning, backlog refinement, retrospective, stand ups…) and necessary team meetings
Make all artifacts available through version control tool such as git
Be part of the review process both as a reviewer as well as a submitter (PRs, architecture decisions, approaches….)
Skill Set
5 to 9 years of Solid experience developing software in an Agile environment following best practices
Experience with AWS cloud platform
Ability to write production grade python and Scala code
Ability to leverage Spark through Databricks platform (PySpark or Scala)
Ability to create and automate data ingestion pipelines for both batch and streaming data
Experience with monitoring and alerting tooling such as AWS Cloudwatch, Datadogs…
Experience deploying software using orchestration tools such as Jenkins
CoffeeBeans Consulting is an award-winning Tech company, founded by IITM and BITS Pilani alumnus, that offers services that are powered by AI and Big Data. We are proud owners of a recommendation engine called WRU, in addition, we are also getting our hands dirty into creating a shared marketplace named Shoptype.Apart from our products we also help eminent consulting firms with their Big Data Pricing strategies. From building core web/mobile apps to helping startups rapidly build their dream products, we have done it all!
📌 Data Engineer - Data Science Engineering (Bengaluru)
🏢 CoffeeBeans Consulting
📍 Bengaluru
Reply to this offer
Impress this employer describing Your skills and abilities, fill out the form below and leave Your personal touch in the presentation letter.