01 Aug
|
Nameless
|
Bengaluru
01 Aug
Nameless
Bengaluru
We
are looking for a Data Engineer with a
strong interest in capital markets and financial data to build scalable data platforms that power
trading, analytics, and decision -making. In this role, you will design and
manage robust data pipelines that ingest and process large volumes of market, trading, and user data. You will ensure high data quality, reliability, and
performance while collaborating with product, analytics, and engineering teams
to deliver accurate datasets that support trading insights, reporting, and
business intelligence.
Key
Responsibilities:
- Design,
build, and maintain scalable ETL/ELT pipelines to ingest and transform data from multiple sources,
supporting both batch and near real -time processing.
- Develop
and manage data
workflows using Apache Airflow,
including scheduling, dependency management, retries, monitoring, and
troubleshooting production issues.
- Ensure data reliability and quality by implementing validation
checks, schema enforcement, and monitoring systems to maintain consistent and
accurate datasets.
- Optimize pipeline performance and scalability to support increasing data
volumes and complex processing requirements.
- Maintain clear documentation, metadata, and data lineage to enhance data governance
and transparency across teams.
- Collaborate
with analytics, product, and business stakeholders to understand data
requirements, translate them into structured datasets and data models, and
communicate pipeline status or potential data issues effectively.
Must
Have
and Python.
pipelines.
with workflow orchestration tools such as Apache Airflow.
platforms (AWS, Google Cloud, or Microsoft Azure) and related storage/compute services
for data pipelines.
warehousing and data modelling concepts.