Responsible for designing and implementing scalable data storage, migration, performance optimization, and data management solutions across Cloudera, AWS, and Databricks platforms.
Key Responsibilities
- Define migration approach for Hive tables, HDFS datasets, and relational databases.
- Design Delta Lake storage architecture and partitioning strategy.
- Develop archival and retention policies using AWS storage services.
- Ensure data integrity and reconciliation during migration.
- Perform performance tuning and optimization.
- Define backup, recovery, and disaster recovery strategies.
- Support database security and access control requirements.
- Required Skills
- Hive, HDFS, Impala
- Delta Lake Architecture
- SQL and Data Warehousing
- AWS S3, Glacier, RDS
- Data Migration Tools (AWS DMS, DistCp)
- Database Performance Tuning
- Data Replication and Recovery
- Data Security and Encryption
- ETL/ELT Architecture