How to debug objectstore issues in Automation Suite?
** When using docs links referenced in this article, ensure that right version and flavour is selected according to environment before proceeding further
1. Start here
Identify the flavour first.
Only RKE2 flavour supports in-cluster ceph objectstore.
# in-cluster Ceph present?
kubectl get cephobjectstore rook-ceph -n rook-ceph
If you get an output for above, the cluster is using in-cluster objectstore. Else, check input.json and check if external object store is provided.
In-cluster objectstore is only available for RKE2, so if the cluster is AKS or EKS or Openshift, you can skip steps intended for rook-ceph / in-cluster objectstore.
In shared clusters, the namespace used by uipath products might be different. For the sake of simplicity, this article uses "uipath" namespace. Replace it by the actual namespace wherever applicable.
Then pick your route:
- Authentication or access is failing → section 2, then the matching issue in section 3.
- You recognise the symptom → go straight to section 3.
- Nothing matches → section 4, and escalate with the evidence listed there.
Objectstore faults usually present as product faults — "I can't upload a package", "my dataset is empty", "the pipeline failed". Several unrelated products failing at once is the signal.
2. Authentication failures: start at the failing pod
The path is: read the logs and events → find the secret that pod uses → does it exist? → does it match the source of truth? → re-run the job, or fix the secret → restart and confirm.
2.1 Read the failing pod's logs and events
Check the failing pod logs/events to understand the specific issue with objectstore
kubectl -n uipath logs $POD
kubectl -n uipath describe pod $POD | sed -n '/Events/,$p'
What you are looking for, and what it means:
| In the logs | What it points at |
|---|---|
403, AccessDenied, InvalidAccessKeyId, SignatureDoesNotMatch | Credentials or policy — continue with 2.2 |
x509: certificate signed by unknown authority, cert-init or cert-trustor init container failing | A certificate problem, not credentials — go to 3.1 and 3.2 |
NoCredentialProviders, IMDS or metadata timeouts | Instance profile or Managed Identity — go to 3.4 |
| A host name you did not expect (for example the internal Ceph service name) | Wrong endpoint — go to 3.8 |
NoSuchBucket, 404 on a bucket | Bucket missing or misnamed — check the bucket value in 2.2 |
| Connection refused, timeouts, proxy errors | Network or proxy — go to 3.3 |
Note the endpoint host name the pod tried to reach, not just the error code. A correct key against the wrong endpoint fails in exactly the same way as a wrong key.
If the failure is a certificate or endpoint problem, stop here and go to section 3. The rest of section 2 is only for credential faults.
2.2 Find which secret that pod uses
kubectl -n uipath get pod $POD -o json \
| jq -r '[.spec.containers[],.spec.initContainers[]?]
| .[].env[]?.valueFrom?.secretKeyRef?.name,
.[].envFrom[]?.secretRef?.name' | sort -u
kubectl -n uipath get pod $POD -o jsonpath='{range .spec.volumes[*]}{.secret.secretName}{"\n"}{end}' | sort -u
For reference, the expected names are:
- External objectstore —
<product>-external-storage-secret, in the product namespace (orchestrator-,aicenter-,documentunderstanding-,platform-,taskmining-,testmanager-,dataservice-,processmining-,ba-,maestro-). - In-cluster Ceph — one secret per product group:
uipath-service-rook-ceph-secret(Orchestrator, DU, Task Mining, Apps, platform),aicenter-service-rook-ceph-secret(AI Center, AI Storage),processmining-service-rook-ceph-secret,airflow-rook-ceph-secret,dataservice-rook-ceph-secret,maestro-service-rook-ceph-secret,mongodb-service-rook-ceph-secret. The Ceph administrative credentials live separately inceph-object-store-secretin therook-cephnamespace.
2.3 Does the secret exist?
SECRET=<secret-name> kubectl -n uipath get secret $SECRET
If it is missing, check whether it exists elsewhere before assuming storage is broken:
kubectl get secret -A | grep "$SECRET"
A secret that exists in rook-ceph or uipath-infra but not where the pod needs it, or that is missing entirely while storage itself is healthy, means credential-manager-job/uipathctl did not finish for that product. Go to 2.5. This is not a Ceph fault.
If it exists, list its key names — an empty or partly written secret looks present but fails like a missing one:
kubectl -n uipath get secret $SECRET -o jsonpath='{range $k,$v := .data}{$k}{"\n"}{end}'
External storage secrets should carry STORAGE_TYPE, BUCKET, FQDN, PORT, REGION and the credential fields; in-cluster secrets should carry OBJECT_STORAGE_ACCESSKEY, OBJECT_STORAGE_SECRETKEY and the host and port fields.
2.4 Does the secret match the source of truth?
In-cluster Ceph — Ceph itself is authoritative. Compare the values of each:
kubectl -n uipath get secret $SECRET \
-o jsonpath='{.data.OBJECT_STORAGE_ACCESSKEY}' | base64 -d
kubectl -n rook-ceph exec deploy/rook-ceph-tools -- \
radosgw-admin user info --uid=<ceph-user> | jq -r '.keys[0].access_key'
If you do not know which Ceph user owns the bucket, ask Ceph:
kubectl -n rook-ceph exec deploy/rook-ceph-tools -- \ radosgw-admin bucket stats --bucket=<bucket> | jq -r .owner
External objectstore — the configuration is authoritative. Compare the secret against the external_object_storage block in input.json
kubectl -n uipath get secret $SECRET \
-o jsonpath='{.data.BUCKET}{"\n"}{.data.FQDN}{"\n"}{.data.REGION}{"\n"}' | base64 -d
If the secret matches and the endpoint is right, this is not a credential problem. Go to section 3.
If it does not match, go to 2.5 — and prefer that over editing the secret by hand.
2.5 Rewrite the secret from the source of truth
This is the preferred fix for both a missing and a mismatched secret. Which command you use depends on the whether the objectstore is internal or external
2.5a In-cluster objectstore -> credential-manager exists: re-run the job
Get the namespace where credential-manager job exists.
kubectl get job -A | grep credential-manager
The job rewrites the secret from Ceph, and it is safe to run more than once. Store the namespace from above in $CM_NS
First read its logs, so you know why it failed the first time ($CM_NS from 2.4):
kubectl -n $CM_NS logs job/credential-manager-job --tail=200 kubectl -n $CM_NS logs -l app.kubernetes.io/component=credential-manager-clients --tail=200
A completed Job cannot be restarted, so recreate it. The jq filter strips the immutable selector and pod labels:
kubectl -n $CM_NS get job credential-manager-job -o json \ | jq 'del(.spec.selector, .spec.template.metadata.labels)' \ | kubectl replace --force -f -
Use the same pattern for rook-ceph-configure-script-job (uipath-infra) or rook-ceph-configure-job (rook-ceph) when the Ceph administrative credentials are the problem.
Then re-check with 2.3 and 2.4, and continue to 2.6.
2.5b External objectstore: re-run uipathctl
The secrets were written directly by uipathctl, so that is what has to write them again. Put the correct credentials in input.json — the updated password if it has been rotated, or the existing one if you are simply repairing a missing or corrupted secret — then re-apply:
uipathctl manifest apply input.json --versions versions.json
To limit the run to the affected product rather than the whole cluster, add --override <product>.
Then continue to 2.6 and restart the consumers.
2.6 Restart the consumers, then confirm
kubectl -n uipath rollout restart deploy
This step is not optional. Products build their S3 client once at pod startup, so a running pod keeps using the old credentials even after the secret changes. Some charts carry reload annotations, but do not rely on them.
Then go back to 2.1 and read the pod's logs again to confirm the original error is gone.
One residual failure mode: an individual product's secret can drift while the main objectstore secret looks correct. If s3cmd works with the administrative key but one product still fails, repeat 2.2 to 2.4 for that product specifically.
3. Common issues
Authentication and access
3.1 — x509: certificate signed by unknown authority on every product
Symptom: prereq or validation fails for every service at once (aicenter, ba, dataservice, documentunderstanding, insights, orchestrator, platform, testmanager, infra). curl to the endpoint works from the node, and the CA is already in the node trust store.
Cause: the node trust store is irrelevant — Automation Suite carries its own trust bundle.
Fix: add the full root chain, not just the leaf certificate, to additional_ca_certs in input.json, then apply.
3.2 — Certificate errors continue after the CA was updated
Symptom: a product feature such as an AI Center pipeline still fails reading data after the certificate was fixed.
Cause: the additional-ca-certificates secret is cumulative. One invalid entry stops cert-init building the bundle, so no pod trusts objectstore, and updating the secret does not remove the bad entry.
Fix: recreate the secret from a validated root CA bundle, then restart all deployments in uipath. Confirmed.
3.3 — S3 403 with a null Request ID, on a cluster behind a proxy
Symptom: S3Exception: (Service: S3, Status Code: 403, Request ID: null). IAM and bucket policies look correct.
Cause: not authentication — the proxy is blocking it. With no_proxy set to .domain, AWS SDK v2 treats the leading dot as "match exactly one character", so the objectstore hostname never matches the exclusion and the request goes to the proxy.
Fix: change no_proxy in input.json from .domain to *.domain. Confirmed. On any proxied cluster, check this before investigating IAM.
3.4 — NoCredentialProviders on EC2 with an instance profile
Symptom: uipathctl prereq create fails with NoCredentialProviders, but the AWS CLI and the readiness script both succeed from the same host.
Cause: pods cannot reach the EC2 instance metadata service because the IMDSv2 hop limit is too low. A hop limit of 3 failed; 10 resolved it.
Fix: raise the IMDSv2 hop limit. Useful for reading the status code: a low hop count gives 401; firewall or security group problems give 403; a missing or not-yet-applied role gives 404 / 400. Prereq messages were improved in 2025.10.2 and 2024.10.8.
3.5 — 403 AccessDenied or 405 MethodNotAllowed from the prereq CHECK_API
Symptom: Object storage test failed for <product>: ... test GetObjectAPI failed.
Cause: usually the IAM or bucket policy is narrower than what the product needs.
Fix: widen the policy to cover every operation the product needs. The prerequisite checker names the exact operation that failed, so start there. The ones most often missing are the multipart set (CREATE_MULTIPART_UPLOAD, UPLOAD_PART, UPLOAD_PART_COPY, COMPLETE_MULTIPART_UPLOAD, LIST_MULTIPART_UPLOADS) required by Orchestrator and Process Mining, and PUT_BUCKET / HEAD_BUCKET / LIST_BUCKETS required by AI Center and Task Mining.
3.6 — Credentials rotated but products still fail
Symptom: the secret holds the new key, but uploads keep failing.
Cause: S3 clients are created at pod startup, so running pods keep the old key. Separately, a per-product secret can be left behind while the main secret is correct.
Fix: follow 2.5 to 2.6, including the restart, then verify that product's secret with 2.4.
3.7 — Azure user-assigned Managed Identity fails prereq validation
A known product defect, version dependent. Fixed in 2024.10.8 and 2.2510.2
Wrong endpoint
3.8 — Uploads go to the internal Ceph service name
Symptom: the browser network tab shows requests to rook-ceph-rgw-rook-ceph.rook-ceph.svc.cluster.local instead of objectstore.<fqdn>. Uploads fail or are blocked as mixed content. Often appears after an upgrade. Customers sometimes ask whether to allow-list that hostname — they should not; it is an internal Kubernetes name that is not reachable from outside the cluster.
Cause: the product is resolving the wrong endpoint.
Fix: Check below items and ensure the endpoint is set to objectstore.fqdn
-
global.fqdn.ceph parameter in sfcore app in argocd
-
secret service-cluster-configurations → OBJECTSTORE_FQDN
-
VirtualService in rook-ceph
3.9 — Uppercase FQDN breaks presigned URLs
Symptom: 403 on presigned URLs. RGW logs show err_no=-2027 new_err_no=-2027 with http_status=403. AI Center reports a generic "slow or lost internet connection"; Orchestrator reports "Could not connect to the server (#101)".
Cause: the FQDN was entered in uppercase in cluster_config.json.
Fix: set it to lowercase and reapply infra, fabric and services. This fixed AI Center.
3.10 — Objectstore hostname missing from the Istio gateway after an FQDN change
Symptom: after an import/export or FQDN change, objectstore calls fail with TLS or 404 errors.
Cause: the shared uipath/main-gateway Gateway still has TLS server blocks and certificates only for the old objectstore hostname — the ingress path was only half-repointed.
Fix: confirm a server block and certificate exist for the new hostname:
kubectl -n uipath get gateway main-gateway -o yaml | grep -A5 objectstore
Changing the FQDN also requires the CORS AllowedOrigin on every bucket to be rewritten.
Migrating in-cluster to external
3.11 — Migration fails with md5 hash differ on AWS S3 with SSE-KMS
Symptom: every migration job fails with corrupted on transfer: md5 hash differ, with Requests specifying Server Side Encryption with AWS KMS managed keys require AWS Signature Version 4 in the logs.
Cause: SSE-KMS requires pure SigV4. Older AWS regions still accept SigV2 for backwards compatibility and Automation Suite uses whatever the region allows, so it fails in us-east-1.
Fix: the buckets must sit in a region created after January 2014 where only SigV4 is enabled — us-east-2 works, us-east-1 does not.
3.12 — AI Center data does not migrate
Symptom: every product migrates except AI Center; its bucket is near-empty and ML packages cannot be downloaded.
Fix: on some versions executeMigration is not set — set executeMigration: true manually on the aicenter app in ArgoCD and resync with prune. Known fix in 2023.10.7 and 2024.10.1.
3.13 — "Failed to remove in-cluster object storage" after a successful migration to external objectstore
Symptom: Failed to remove in-cluster object storage because you have not configured any external object storage, even though uploads are demonstrably landing on external storage.
Cause: a known defect in the removal step — the migration itself succeeded.
Fix: fixed in 2023.10.3; on 2024.10.x confirm the fix status before acting. Do not re-run the migration.
Before removing in-cluster objectstore, know that it is irreversible — it cannot be reinstalled afterwards. The uninstall also deletes the rook-ceph namespace and requires rm -rf /var/lib/rook/ on every node. Installer error codes you may see: 15021 (no external objectstore configured), 15023 (products still using in-cluster objectstore), 15024 (failed to delete rook-ceph object store), 55003 (run rm -rf /var/lib/rook/ on all nodes), 11052 (bucket name empty), 14022 (error retrieving rook-ceph-tools pod).
One that looks like objectstore but is not
3.14 — Some failures only look like objectstore. On OpenShift, Context Grounding index creation failing has been traced to a CephFS PVC permissions problem — SELinux MCS labels not matching and a missing fsGroup — not to S3 at all. If your objectstore checks all pass, confirm the failing path actually touches objectstore before triaging it as a storage issue.