While attempting to patch HAA Servers, it has been observed that after enabling maintenance mode, they are not being assigned any shards.
Issue:
Recover the shards after maintenance mode
Description:
At this point, rladmin status displays the node’s shards field in yellow, which indicates that shards cannot migrate to the node.
After turning on maintenance mode for the node, Redis Enterprise saves a snapshot of its configuration and then moves its shards and endpoints to the next node. Now node 2 has 0/0 shards because shards cannot migrate to it while it is in maintenance mode.
Solution:
Redis Enterprise saves a snapshot of the node configuration each time the maintenance mode is turned on. In the presence of multiple snapshots, a particular snapshot can be restored upon turning off the maintenance mode.
- To get a list of available snapshots, execute the following command:
rladmin node snapshot list
- To specify a snapshot when turning off the maintenance mode, execute the following command:
rladmin node maintenance_mode off snapshot_name
Note:
If an error occurs during the activation of maintenance mode, the snapshot is not deleted. Upon rerunning the command, utilize the snapshot from the initial attempt, as it preserves the original state of the node.
If the issue persists, please provide the following details and raise a support case:
Diagnosing the issue
- rladmin status extra all
- rlcheck --continue-on-error
- Check the Redis Logs. On the master node check these two locations
- /var/log/redis/redis-server.log
- /var/log/redis_*.log