Platform Data Engineering LeadCompany Overview

Boston Insights is an innovative startup creating competitive advantage for pharmaceutical companies by unlocking their clinical supply chain data and enabling end-to-end visibility. We augment risk resiliency and agility to ensure uninterrupted supply of investigational drugs to patients on time. Our mission is to transform how pharmaceutical companies manage their clinical supply chains through cutting-edge data solutions.

Position

Overview

We are seeking a Data Engineer with 5+ years of experience, deep expertise in the Microsoft Azure technology stack, and proven ability in integrating data from external data lakes, AWS data warehouses, and enterprise supply chain solutions like SAP. The ideal candidate will also have strong experience in data governance, building data automation tools using Python and related languages, and developing AI agents using Claude to support governed data discovery, lineage, quality checks, and compliance workflows. Key ResponsibilitiesData Integration & Pipeline Development

Lead the integration strategy for ingesting and harmonizing data from external sources, including AWS-based data lakes/warehouses (such as S3, Redshift) and SAP systems.

Design, build, and maintain scalable data pipelines using Azure Data Factory, Azure Synapse Analytics, and Azure Data Lake / MS Fabric OneLake.

Design reusable metadata-driven ingestion framework for rapid on-boarding of new datasets.

Hands-on experience in metadata driven pipelines with event-based ingestion.

Automate ETL processes for data extraction, transformation, and loading across hybrid and multi-cloud environments.

Build and maintain real-time and batch data integration workflows between Azure, AWS, and on-premises sources. Data Architecture & Infrastructure

Design and implement data lake and data warehouse solutions on Azure platform

Establish data governance frameworks and ensure data quality across all pipelines

Implement security best practices for handling sensitive pharmaceutical data

Create and maintain data documentation and lineage tracking Data Governance & Quality

Define and enforce data governance frameworks: data cataloging, lineage, quality, privacy, and compliance

Implement robust data validation, cleansing, and monitoring systems to ensure accuracy and reliability





Support security standards through effective data management practices.

Design and develop Claude-based AI agents to assist with data governance use cases, including metadata discovery, lineage interpretation, policy checks, anomaly triage, and stewardship workflows. Automation & Tooling

Develop data automation tools and reusable components using Python, PySpark (and other relevant frameworks/languages)

Enable end-to-end process automation for data ingestion, processing, and reporting.

Implement CI/CD processes for data solutions, including testing, monitoring, and alerting. Analytics & Reporting Support

Collaborate with data scientists and analysts to support advanced analytics

Build data models that enable risk assessment and supply chain optimization

Develop APIs and data services to support front-end applications

Create monitoring and alerting systems for data pipeline health Collaboration & Support

Partner with supply chain, analytics, and business stakeholders to understand business requirements and translate them into scalable technical solutions.

Collaborate with ERR, IRT/RTSM functional and technical teams to optimize data extraction and synchronization.

Ideal Candidate

Profile The strongest candidate combines hands-on Azure data engineering expertise with the judgment to build governed, automated, and scalable data solutions for complex pharmaceutical supply chain environments.

Required Technical

Experience

5+ years of professional data engineering experience.

Data Integration: Proven track record integrating data from AWS services (S3, Redshift, Glue, etc.) into Azure or other cloud environments

Azure Data Services: Expert-level knowledge of Azure Data Factory, Azure Synapse Analytics, Data Bricks, Azure Data Lake Storage, and Azure SQL Database, Apache Spark

Database Technologies: Strong knowledge of both relational (SQL Server, PostgreSQL) and NoSQL (Cosmos DB) databases

Programming Languages: Proficiency in Python, SQL, PySpark and PowerShell for data automation and wrangling

Claude AI Agent Development:



Hands-on experience developing Claude-based agents for enterprise data governance use cases, including:

Building agents that support data cataloging, metadata enrichment, lineage interpretation, and data stewardship workflows.

Designing prompt workflows, tool-use patterns, and guardrails for governed interaction with sensitive pharmaceutical and supply chain data.

Integrating Claude agents with data platforms, APIs, and automation scripts to streamline data quality review, compliance checks, and anomaly triage.

Evaluating agent outputs for accuracy, traceability, security, and alignment with governance policies.

Version Control: Proficiency with Git and Azure DevOps

Hands-on experience with SAP data models and integrating SAP data with Azure data lake Preferred Technical Skills

Experience with API-based data integration for cloud and enterprise applications.

Experience with Infrastructure as Code (ARM templates, Terraform)

Familiarity with data quality tools, metadata management, and automated data lineage tracking.

Knowledge of containerization (Docker, Kubernetes) for data automation workflows.

Knowledge of machine learning pipelines and MLOps practices

Experience with data visualization tool, Power BI Qualified Skills

Strong problem-solving and analytical thinking abilities

Excellent communication skills with ability to explain technical concepts to non-technical stakeholders

Experience with Agile development methodologies

Attention to detail and commitment to data quality The Opportunity We Offer

Competitive salary commensurate with experience.

Professional development training and certifications.

Remote, hybrid work environment

Work on cutting edge technologies Application Instructions

Please submit your resume and a cover letter at https://docs.google.com/forms/d/e/1FAIpQLSden_nhpLt0rQI9wQv_kvp_T46EsUC1j9EAIFPBpF4yeXco-Q/viewform?usp=preview

Please highlight your experience in the following areas

Experience in architecting meta data-based data extraction, transformation and processing within MS Fabric, Databricks and Azure, environments.

Examples of data automation tools or frameworks you have developed.

Experience developing Claude-based agents or AI-assisted workflows that improve data governance, lineage, quality, or compliance processes.

📌 Lead Data Engineer/ Data Architect (India)
🏢 Boston Insights
📍 India