[AutomationSuite] Exploring ceph objectstore Using s3cmd

News Knowledge Base
1

How can I explore Automation Suite in-cluster objectstore using s3cmd?

1. What tooling is already in the cluster

s3cmd ships inside the sf-k8-utils-rhel image, which is already present in every cluster including air-gapped ones. That image also has rclone, kubectl and jq. There is no separate debug image, and mc (the MinIO client) is not shipped.

Because the image is already there, you do not need internet access or a registry change to get a shell with s3cmd in it.

2. Start a temporary pod

Find the image the cluster already uses, so this works offline and behind a private registry:

sf_utils=$(kubectl get pods -n uipath-infra \
  -o jsonpath="{.items[*].spec.containers[*].image}" \
  | tr -s '[[:space:]]' '\n' | sort -u | grep sf-k8-utils | head -1)
echo "$sf_utils"

If that is empty, run the same pipeline with kubectl get pods -A, or set it by hand.

cat <<EOF | sed "s;__IMAGE__;${sf_utils};g" | kubectl apply -f -
apiVersion: v1
kind: Pod
metadata:
  name: s3cmd
  namespace: uipath-infra
spec:
  containers:
  - name: s3cmd
    image: __IMAGE__
    imagePullPolicy: IfNotPresent
    command: ["/bin/bash"]
    args: ["-c", "tail -f /dev/null"]
    securityContext:
      allowPrivilegeEscalation: false
      capabilities:
        drop: [ALL, NET_RAW]
      readOnlyRootFilesystem: true
      runAsGroup: 1
      runAsNonRoot: true
      runAsUser: 1
    volumeMounts:
    - mountPath: /.kube
      name: kubedir
  volumes:
  - emptyDir: {}
    name: kubedir
  restartPolicy: Always
EOF

Get a shell with kubectl -n uipath-infra exec -it s3cmd -- bash, and delete the pod when you are done:

kubectl -n uipath-infra delete pod s3cmd

3. Point s3cmd at in-cluster Ceph

s3cmd here is driven entirely by environment variables — no .s3cfg file is involved. Build the exports, then use them inside the pod shell:

HOST=$(kubectl -n rook-ceph get svc rook-ceph-rgw-rook-ceph -o jsonpath="{.spec.clusterIP}")
PORT=$(kubectl -n rook-ceph get svc rook-ceph-rgw-rook-ceph -o jsonpath="{.spec.ports[0].port}")
export AWS_HOST=$HOST
export AWS_ENDPOINT=$HOST:$PORT
export AWS_ACCESS_KEY_ID=$(kubectl -n rook-ceph get secret ceph-object-store-secret \
  -o jsonpath='{.data.OBJECT_STORAGE_ACCESSKEY}' | base64 -d)
export AWS_SECRET_ACCESS_KEY=$(kubectl -n rook-ceph get secret ceph-object-store-secret \
  -o jsonpath='{.data.OBJECT_STORAGE_SECRETKEY}' | base64 -d)

These are the Ceph admin credentials, so they can see every bucket. To act as a specific bucket's owner instead, see section 6.

4. Command cookbook

The empty --host-bucket= is required. Ceph uses path-style addressing; without it s3cmd invents a hostname per bucket and fails.

s3cmd ls --host=${AWS_HOST} --host-bucket= --no-ssl                    # list buckets
s3cmd ls s3://uipath/ --host=${AWS_HOST} --host-bucket= --no-ssl       # list one bucket
s3cmd du -H s3://uipath --host=${AWS_HOST} --host-bucket= --no-ssl     # bucket size
s3cmd info s3://uipath/<key> --host=${AWS_HOST} --host-bucket= --no-ssl
s3cmd get s3://uipath/<key> /tmp/ --host=${AWS_HOST} --host-bucket= --no-ssl

s3cmd du -H per bucket is the quickest way to find what is filling a full Ceph cluster. The usual culprits are sf-logs and the metrics buckets.

5. Getting the right key for a specific bucket

To act as the bucket's actual owner rather than as admin:

bucket="uipath"
user=$(kubectl -n rook-ceph exec deploy/rook-ceph-tools -- \
  radosgw-admin bucket stats --bucket=$bucket | jq -r .owner)
kubectl -n rook-ceph exec deploy/rook-ceph-tools -- radosgw-admin user info --uid=$user

This is also how you confirm whether a product's secret holds the key Ceph actually expects.

6. Ceph health and bucket inventory

kubectl get cephobjectstore rook-ceph -n rook-ceph -o wide
kubectl -n rook-ceph get pods -l app=rook-ceph-rgw
kubectl -n rook-ceph exec deploy/rook-ceph-tools -- ceph status
kubectl -n rook-ceph exec deploy/rook-ceph-tools -- ceph health detail
kubectl -n rook-ceph exec deploy/rook-ceph-tools -- radosgw-admin bucket list
kubectl -n rook-ceph exec deploy/rook-ceph-tools -- radosgw-admin bucket stats --bucket=<name>

Trust the CephObjectStore resource, not the ArgoCD app health. The ArgoCD application can show unhealthy while objectstore is perfectly usable, and chasing that status wastes time. Check .status.phase — you want Ready.

To reclaim space that is pending deletion:

kubectl -n rook-ceph exec deploy/rook-ceph-tools -- radosgw-admin gc process --include-all

You can explore external objectstore using this image by point the FQDN to the external objectstore endpoint and updating the credentials. But most of external objectstores can be viewed with their respective tools more conveniently.