Volume Mount Failures in Longhorn Docker Registry Pods

News Knowledge Base
1

How to Resolve Volume Mount Failures in Longhorn Docker Registry Pods?

Issue Description

There may be instances where the Docker registry pod in Longhorn experiences issues, particularly with volume mounting failures. For example, you might see logs indicating that all replicas have failed and the engine salvage request is triggered. A typical log entry would look like:

"e957bdab28be logs/longhorn-manager-t2l8n/longhorn-manager.log:32962:2025-01-28T09:13:22.321039146Z time="2025-01-28T09:13:22Z" level=info msg="All replicas are failed, set engine salvageRequested to true" accessMode=rwo controller=longhorn-volume frontend=blockdev migratable=false node=10-80-21-123.ap-seast-1.compute.internal owner=ip-10-80-21-123.ap-seast-1.compute.internal state=detached volume=pvc-7de37359-99c7-4bf7-b0ab-e957bdab28beadfw123fx"

This indicates that the Longhorn volume is in a detached state and could potentially require manual intervention to resolve.

Resolution

  1. Check Volume Mounting Status: The pod logs suggest the volume fails to mount. To gain insights into why the volume is in a failed state, run the following command to check the logs of the longhorn-manager container in the longhorn-system namespace: 
    kubectl logs -l app=longhorn-manager -n longhorn-system -c longhorn-manager --prefix=true | grep registry_pv

Note: This command filters logs related to the longhorn-manager container in the longhorn-system namespace.

  1. Check for Faulted Longhorn Replicas: If the Longhorn replicas are in a faulted state, the volume may not be able to attach and could remain in the detached state. To identify if the volume requires manual salvage, execute the following command: 
    kubectl logs -l app=longhorn-manager -n longhorn-system -c longhorn-manager --prefix=true --tail=-1 | grep "set engine salvageRequested to true" | grep <PV_NAME>
    • Note: Replace <PV_NAME> with the name of your Persistent Volume (PV).
    • Sample output of the above command:

2023-11-20T18:22:16.667609096+11:00 time="2023-11-20T07:22:16Z" level=info msg="All replicas are failed, set engine salvageRequested to true" accessMode=rwo controller=longhorn-volume frontend=blockdev migratable=false node=rpa-suite-dev-01.it.csiro.au owner=rpa-suite-dev-01.it.csiro.au state=detaching volume=pvc-031fd6bc-9cfe-420a-9213-da38509d733a

To resolve the issue, follow these steps:

  1. Scale down the workload pods that are using the affected volume to avoid conflicts during the salvage process. Note: We have to look at the Docker registry pods.

  2. Find the replicas of the relevant volume using the following command: 
    kubectl get replicas.longhorn.io -n longhorn-system | grep <PV_NAME>
    Note: Replace <PV_NAME> with the name of your Persistent Volume (PV).

  3. Edit the replica configuration for the relevant volume by running:

    kubectl edit replicas.longhorn.io -n longhorn-system <REPLICA_NAME>

Note: In the edit screen, locate the spec.failedat field and set it to an empty string ("").

  1. Scale up the workload pods after fixing the replica configuration to restore the functionality of the volume.