Redis CRDB Flush Operation Fails Due to Redirect Handling

News Knowledge Base
1

How to overcome the flush task operation on CRDB, as it may remain in a pending state and fail to complete?

Issue description:

During Redis CRDB maintenance activities, such as a FLUSHDB operation, the flush task may remain in a pending state and fail to complete. This issue is typically related to redirect handling being disabled in the cluster, which prevents API requests from reaching the master node.

Observed Error Logs:

The following log entry may appear:

WARNING urllib3.connectionpool MainThread: Retrying (Retry(total=9, connect=None, read=None, redirect=None, status=None)) after connection broken by 'NewConnectionError( '<urllib3.connection.HTTPConnection object at 0x7f5d22c97400>: Failed to establish a new connection: [Errno 110] Connection timed out')': /coordination/v1/info

API connectivity tests may return HTTP 307 Temporary Redirect responses from CRDB node URLs, but those redirects are not followed if redirect handling is disabled.

Cause:

By default, Redis CRDB clusters do not handle redirects. If the cluster DNS resolves to a non-master node, and redirect handling is not enabled, the node will not forward the request to the master. As a result, operations like FLUSHDB may fail or remain pending.

Resolution:

Enable redirect handling in the Redis cluster configuration to allow non-master nodes to forward incoming requests to the master node.

Steps:

  1. SSH into the master node of the Redis CRDB cluster.
  2. Run the following command: - rladmin cluster config handle_redirects enabled
  3. Retry the flush operation.

Troubleshooting Steps

Verify API Behavior:

  1. curl -v -k -u <user>:<password> -H "Content-Type: application/json" -X GET http://demo-crdb-node1:8080/v1/crdbs
  2. curl -v -k -u <user>:<password> -H "Content-Type: application/json" -X GET http://demo-crdb-node2:8080/v1/crdbs
    • Check for HTTP/1.1 307 Temporary Redirect responses.

Check Cluster Configuration:

    • Use rladmin status or similar tools to verify if handle_redirects is currently disabled.

Enable Redirect Handling:

  • If not enabled, run:
    • rladmin cluster config handle_redirects enabled

Retry the Operation:

    • Once the setting is enabled, reattempt the flush operation.