Key Responsibilities

- Analyze existing data pipelines, ETL jobs, and data warehouse objects in Amazon Redshift and AWS Glue.
- Design and implement migration strategies to move data and workloads to Databricks.
- Develop, optimize, and maintain ETL/ELT pipelines using Databricks, Apache Spark, and Python.
- Migrate SQL scripts, stored procedures, and data transformation logic to Databricks-compatible solutions.
- Validate data accuracy, completeness, and consistency before and after migration.
- Optimize data models, partitioning, and performance in Databricks.
- Implement Delta Lake best practices for data storage, versioning, and optimization.
- Collaborate with architects and stakeholders to ensure business requirements are met.
- Troubleshoot migration issues and optimize processing performance.
- Create technical documentation, migration runbooks, and knowledge transfer materials.
- Ensure adherence to security, governance, and compliance standards throughout the migration.

Required Skills & Qualifications

- Bachelor's degree in Computer Science, Information Technology, Engineering, or a related field.
- 4+ years of experience in data engineering or data platform migrations.
- Strong experience with:
- Amazon Redshift
- AWS Glue
- Databricks
- Apache Spark
- Python
- SQL

- Experience with cloud platforms, preferably AWS.
- Robust understanding of data warehousing concepts and dimensional modeling.
- Experience with Delta Lake and Lakehouse architecture.
- Knowledge of ETL/ELT frameworks and workflow orchestration.
- Experience with Git and CI/CD pipelines.




- Strong analytical and problem-solving skills.
- Excellent communication and documentation skills.

Preferred Qualifications

- Experience with Databricks Workflows, Unity Catalog, and Delta Live Tables.
- Familiarity with Apache Airflow or similar orchestration tools.
- Experience with Infrastructure as Code (Terraform or CloudFormation).
- AWS and/or Databricks certifications are a plus.
- Experience with large-scale enterprise data migrations.

Technical Skills

- Cloud: AWS
- Data Warehouse: Amazon Redshift
- ETL: AWS Glue
- Target Platform: Databricks
- Languages: Python, SQL
- Frameworks: Apache Spark, PySpark
- Storage: Delta Lake
- Version Control: Git
- CI/CD: Azure DevOps, GitHub Actions, or Jenkins (preferred)

Success Criteria

- Successfully migrate Redshift databases and AWS Glue ETL jobs to Databricks.
- Achieve data accuracy and reconciliation with minimal discrepancies.
- Improve pipeline performance and scalability on the Databricks Lakehouse platform.
- Deliver comprehensive migration documentation and operational handover.
- Complete migration within agreed timelines while minimizing business disruption.

Employment Type

- Full-time / Contract (based on project requirements)

Experience

- Required: 48 years in Data Engineering
- Relevant Migration Experience: At least 2 years migrating cloud-based data platforms to Databricks

This role is ideal for a data engineer with hands-on experience in cloud data migrations and modernizing enterprise data platforms using Databricks.

📌 Data Migration Developer (Telangana)
🏢 Techtrinite Consulting
📍 Telangana