Job Description

Core Technical Skills

•
Python
- Advanced level (pandas, requests, boto3 libraries)

•
API Integration
- REST APIs, authentication (OAuth, API keys), error handling

•
Cloud Platforms
- AWS (S3, EC2, IAM) proficiency

•
Databricks/Spark
- PySpark, Delta Lake, SQL optimization

•
Data Pipeline Tools
- Airflow, Databricks Workflows, or equivalent orchestration

•
SQL
- Complex queries, window functions, performance tuning

Security & Enterprise Knowledge

• Understanding of security telemetry (logs, alerts, events, metrics)

• Familiarity with at least 2-3 security platforms (JAMF, Intune, Qualys, CrowdStrike, Okta)

• Experience with AWS CloudTrail or similar cloud audit logs

• Knowledge of data security and compliance (encryption, access controls)

Additional Competencies

• Version control (Git)

• CI/CD pipeline familiarity

• Data quality and validation frameworks

• Performance tuning and optimization mindset

Education Qualificaiton

BE, BTech, MCA

Job Title:
Data Integration Engineer

Roles & Responsibilities





Data Integration & Connectivity

• Design and build connectors between enterprise security platforms (JAMF, Intune, Qualys, CrowdStrike, Okta, AWS CloudTrail) and
Databricks

• Establish secure API connections and authentication mechanisms with each source system

• Implement data extraction logic using REST APIs, SDKs, and native connectors

• Handle varying data formats and transformation requirements across heterogeneous sources

Data Pipeline Development

• Develop automated Python scripts for continuous data ingestion

• Build orchestration workflows using Apache Airflow, Databricks Workflows, or similar tools

• Implement error handling, retry mechanisms, and monitoring for pipeline reliability

• Ensure near real-time or scheduled data synchronization based on requirements

AWS S3 & Data Staging

• Configure S3 bucket architecture and data partitioning strategy

• Land raw security telemetry data in S3 with proper folder struct

📌 Data Integration Engineer (Bengaluru)
🏢 Metasys Technologies
📍 Bengaluru