Automation Suite Service Fabric logs cleanup script updated

News Knowledge Base
1

How to clean up Old Logs from S3 in Offline/Online Environments?

Issue description:

How to clean up Old Logs from S3 in Offline/Online Environments?

Resolution:

Cleaning Up Old Logs from S3 in Offline/Online Environments

Purpose

This article provides the steps to deploy a temporary pod that cleans up old logs stored in the sf-logs S3

Step 1: Identify Available Image Version

For offline environments:

podman search localhost:30071/uipath/sf-k8-utils-rhel --tls-verify=false --list-tags

For online environments:

podman search registry.uipath.com/uipath/sf-k8-utils-rhel --list-tags

Use the latest available tag in the YAML manifest.

Step 2: Create the Cleanup YAML File

Copy the following content into a file named cleanup.yaml.

apiVersion: v1
kind: ConfigMap
metadata:
name: cleanup-script
namespace: istio-system
data:
cleanup_old_logs.sh: |
#!/bin/bash

function parse_args() {
CUTOFFDAY=7
SKIPDRYRUN=0
while getopts 'c:sh' flag "$@"; do
case "${flag}" in
c) CUTOFFDAY=${OPTARG} ;;
s) SKIPDRYRUN=1 ;;
h) display_usage; exit 0 ;;
*) echo "Unexpected option ${flag}"; display_usage; exit 1 ;;
esac
done
shift $((OPTIND - 1))
}

function display_usage() {
echo "usage: $(basename "$0") -c <number> [-s]"
echo " -s skip dry run, really deletes the log dirs"
echo " -c logs older than how many days to be deleted (default: 7 days)"
echo " -h help"
echo "NOTE: Default is dry run, to really delete logs set -s"
}

function setS3CMDContext() {
OBJECT_GATEWAY_INTERNAL_HOST=$(kubectl -n rook-ceph get services/rook-ceph-rgw-rook-ceph -o jsonpath="{.spec.clusterIP}")
OBJECT_GATEWAY_INTERNAL_PORT=$(kubectl -n rook-ceph get services/rook-ceph-rgw-rook-ceph -o jsonpath="{.spec.ports[0].port}")
AWS_ACCESS_KEY=$1
AWS_SECRET_KEY=$2

export AWS_HOST=$OBJECT_GATEWAY_INTERNAL_HOST
export AWS_ENDPOINT=$OBJECT_GATEWAY_INTERNAL_HOST:$OBJECT_GATEWAY_INTERNAL_PORT
export AWS_ACCESS_KEY_ID=$AWS_ACCESS_KEY
export AWS_SECRET_ACCESS_KEY=$AWS_SECRET_KEY
}

function setS3CMDContextForLogs() {
BUCKET_NAME='sf-logs'
AWS_ACCESS_KEY=$(kubectl -n logging get secret s3-store-secret -o json | jq '.data.S3_ACCESSKEY' | sed -e 's/^"//' -e 's/"$//' | base64 -d)
AWS_SECRET_KEY=$(kubectl -n logging get secret s3-store-secret -o json | jq '.data.S3_SECRETKEY' | sed -e 's/^"//' -e 's/"$//' | base64 -d)
setS3CMDContext "$AWS_ACCESS_KEY" "$AWS_SECRET_KEY"
}

function delete_old_logs() {
local cutoffdate=$1
days=$(s3cmd ls s3://sf-logs/ --host="${AWS_HOST}" --host-bucket= s3://sf-logs --no-check-certificate --no-ssl)
days=${days//DIR}
if [[ $SKIPDRYRUN -eq 0 ]]; then
echo "DRY RUN. Following log dirs are selected for deletion"
fi
for day in $days; do
day=${day#*sf-logs/}
day=${day::-1}
if [[ ${day} < ${cutoffdate} ]]; then
if [[ $SKIPDRYRUN -eq 0 ]]; then
echo "s3://$BUCKET_NAME/$day"
else
echo "###############################################################"
echo "Deleting Logs for day: {$day}"
echo "###############################################################"
s3cmd del "s3://$BUCKET_NAME/$day/" --host="${AWS_HOST}" --host-bucket= --no-ssl --recursive || true
fi
fi
done
}

function main() {
setS3CMDContextForLogs
echo "Bucket name is $BUCKET_NAME"

CUTOFFDATE=$(date --date="${CUTOFFDAY} day ago" +%Y_%m_%d)
echo "logs older than ${CUTOFFDATE} will be deleted"

delete_old_logs "${CUTOFFDATE}"
if [[ $SKIPDRYRUN -eq 0 ]]; then
echo "NOTE: For really deleting the old log directories run with -s option"
fi
}

parse_args "$@"
main
exit 0
---
apiVersion: v1
kind: Pod
metadata:
name: cleanup-old-logs
namespace: istio-system
spec:
serviceAccountName: fluentd-logs-cleanup-sa
containers:
- name: cleanup
image: localhost:30071/uipath/sf-k8-utils-rhel:0.8
command: ["/bin/bash"]
args: ["/scripts-dir/cleanup_old_logs.sh", "-s"]
volumeMounts:
- name: scripts-vol
mountPath: /scripts-dir
securityContext:
privileged: false
allowPrivilegeEscalation: false
readOnlyRootFilesystem: true
runAsUser: 9999
runAsGroup: 9999
runAsNonRoot: true
capabilities:
drop: ["NET_RAW"]
volumes:
- name: scripts-vol
configMap:
name: cleanup-script

Step 3: Deploy the Cleanup Pod

kubectl apply -f cleanup.yaml

Step 4: Monitor Progress

Check the logs of the cleanup pod:

kubectl -n istio-system logs cleanup-old-logs -f

Step 5: Clean Up Resources

After logs are deleted, remove the pod and configmap:

kubectl delete -f cleanup.yam