Responsible for designing and implementing scalable data storage, migration, performance optimization, and data management solutions across Cloudera, AWS, and Databricks platforms.
Key Responsibilities
Define migration approach for Hive tables, HDFS datasets, and relational databases.
Design Delta Lake storage architecture and partitioning strategy.
Develop archival and retention policies using AWS storage services.
Ensure data integrity and reconciliation during migration.
Perform performance tuning and optimization.
Define backup, recovery, and disaster recovery strategies.
Support database security and access control requirements.
Required Skills
Hive, HDFS, Impala
Delta Lake Architecture
SQL and Data Warehousing
AWS S3, Glacier, RDS
Data Migration Tools (AWS DMS, DistCp)
Database Performance Tuning
Data Replication and Recovery
Data Security and Encryption
ETL/ELT Architecture