Job Description

------------------------------------------------

About the role

The Data Engineer in our AI & Data team will be responsible for designing and building the data structures and pipelines our AI Engineers rely on across Azure, Snowflake, Databricks, and Lakebase (our managed Postgres / OLTP layer). The primary mission of this role is to enable the AI Engineering team- translating the needs of machine-learning and computer-vision workflows into reliable, well-modelled, and cost-effective data foundations.

Tasks include setting up new data pipelines and transformations, ingesting structured and

unstructured data into the data lake and warehouse, monitoring the performance and cost

effectiveness of existing data jobs, and docking machine-learning processes into the existing data landscape. The Data Engineer works hand in hand with AI Engineers and is the go-to person for making trusted data available for models, products, and analytics.

Main Responsibilities

● As part of the AI & Data team, design and build the data structures, schemas, and models

that AI Engineers depend on for training, feature engineering, and inference.

● Develop and orchestrate scalable data pipelines on Databricks (Spark, Delta Lake) and load curated, analytics-ready data into Snowflake.

● Own data ingestion, transformation (ELT/ETL), and storage across the Azure cloud (e.g.

ADLS, Data Factory, Event Hubs/Synapse), including structured, semi-structured, and

unstructured data such as text, images, and video.

● Dock machine-learning and computer-vision models into the data pipelines, and design the data flow that feeds and consumes those AI services.





● Sync curated lakehouse data into Lakebase (managed Postgres) for low-latency serving,

manage change-data-capture back into Delta tables, and support online feature stores and

agent state for AI Engineers.

● Build and maintain API integrations and automated data ingestion from internal systems

and external third-party sources.

● Monitor pipeline performance, reliability, and cost; troubleshoot failed jobs and optimize

Snowflake and Databricks workloads.

● Implement data quality, validation, and lineage, and document the data dictionary and ETL processes.

● Partner with AI Engineers and stakeholders to translate model and business requirements

into extensions of the data platform.

Skills, Qualifications & Education

● Bachelor’s degree in Computer Science, Data Engineering, or a related field.

● At least 4 years of work experience in data engineering or a similar data-focused role.

● Hands-on production experience with Databricks (Apache Spark, Delta Lake, notebooks,

workflows).

● Hands-on production experience with Snowflake (data modeling, performance tuning,

access control, cost management).

● Solid experience with Microsoft Azure data services (e.g. ADLS, Data Factory, Event Hubs /

Synapse).





● Experience with PostgreSQL and OLTP databases; familiarity with Lakebase (Databricks

managed Postgres) is a strong plus.

● Working knowledge of JavaScript / TypeScript, used for data APIs, microservices, or app

facing integrations.

● Strong expertise in SQL (will be tested during the recruitment process).

● Robust Python literacy, especially for data handling and pipeline development.

● Comfortable working with both structured and unstructured data; does not shy away from troubleshooting failed ETL processes or API integrations.

● Outstanding data-structure and data-modeling design skills.

● Working knowledge of machine-learning, NLP, or computer-vision workflows is a plus.

● Experience with dbt, Airflow, or Databricks Workflows, and with CI/CD and infrastructure

as-code, is a plus.

● Solid ability to translate ideas between technical and non-technical audiences.

● Curious, collaborative, self-motivated, and organized; able to run multiple projects against tight deadlines.

Pay: ₹509,523.85 - ₹1,868,850.93 per year

Application Question(s):

- What is your notice period in days? (15)
- What is your expected CTC in LPA? (18)

Experience:

- overall: 4 years (Required)
- Databricks: 4 years (Required)
- Snowflake: 2 years (Required)
- Azure/AWS: 3 years (Required)
- Data Engineer: 4 years (Required)
- PostgreSQL: 2 years (Required)
- JavaScript: 1 year (Required)
- Typescript: 1 year (Required)
- Python: 1 year (Required)
- SQL: 2 years (Required)
- ETL(Extract, Transform, Load): 4 years (Required)
- CI/CD: 3 years (Required)

Work Location: Remote

📌 Data Engineer (India)
🏢 OptimHire
📍 India