Databricks Engineer – PostgreSQL-to-Databricks Pipeline & Delta Sharing (AWS)

3-Month Fixed Term Employment| Remote (India)

About the Role

Clinpex is looking for a hands-on Databricks Engineer for a 3-month contract to design and build a data pipeline that ingests data from PostgreSQL database into Databricks on AWS, and shares it with an external pharma client via Delta Sharing. This is an end-to-end build: you'll gather requirements directly with the client, architect the solution, implement it, and validate it with them before handoff.

Key Responsibilities

- Meet with the client to understand their data requirements, schema expectations, refresh cadence, and consumption patterns
- Stand up and configure a Databricks workspace on AWS, including Unity Catalog and metastore configuration
- Design and build a batch ingestion pipeline to extract data from PostgreSQL into Databricks
- Model and structure ingested data into Delta Lake tables (bronze/silver/gold or similar layering), ensuring data quality and consistency
- Configure Unity Catalog catalogs, schemas, and access controls for the shared datasets
- Set up Delta Sharing to securely share specified datasets with the client's Databricks workplace (open or Databricks-to-Databricks sharing)
- Manage AWS infrastructure components supporting the pipeline (S3 storage, IAM roles, networking/connectivity to PostgreSQL)
- Implement job scheduling, monitoring, and alerting (e.g., Databricks Jobs, retries, failure notifications)
- Validate data with the client post-delivery and iterate based on feedback
- Document architecture, pipeline logic, sharing configuration,



and operational runbooks

Required Qualifications

- Hands-on experience building data pipelines into Databricks from relational databases (PostgreSQL experience strongly preferred)
- Strong working knowledge of Unity Catalog and metastore administration on AWS
- Practical experience setting up and managing Delta Sharing (provider-side), including recipient/share configuration
- Experience with Delta Lake table design and best practices (partitioning, schema evolution, layered architecture)
- Familiarity with AWS services relevant to Databricks deployments (S3, IAM, VPC/networking, Secrets Manager)
- Experience with Databricks Jobs/Workflows for scheduling and orchestration
- Comfortable working directly with external clients to gather requirements and communicate technical decisions
- Strong independent problem-solving skills — this is a build-from-scratch engagement with light oversight

Nice to Have

- Experience connecting Databricks to PostgreSQL via JDBC, Lakehouse Federation, or CDC tools (e.g., Debezium, AWS DMS)
- Databricks certifications (Data Engineer Associate/Professional, Platform Administrator)
- Experience in pharma/life sciences or other regulated data environments
- Terraform or infrastructure-as-code experience for Databricks/AWS provisioning

Engagement Details

- Duration: 3 months (potential extension based on need)
- Location: Remote, India-based candidates only
- Cloud Platform: AWS
- Extension: The fixed term employment may be extended beyond the first 3 months

To Apply, complete the form below https://airtable.com/app3eTmH0cjP2IAZL/shrnyDe6erKArbeOz

📌 Databricks Engineer – PostgreSQL-to-Databricks Pipeline & Delta Sharing (AWS) (India)
🏢 Clinpex
📍 India