Disabling Prometheus Port 9100 On Kubernetes Replicated System

News Knowledge Base
,
1

A security scan found that the (AIFabric on Linux) Document Understanding server was using a self-signed cert on port 9100, which is a security violation by policy. If the security team is concerned about potential mismatch scenarios if they export the certification from server 1 and install it on server 2. This would create a mismatch situation.

Root Cause:

Per MCS, Prometheus, using port 9100, is installed as a node exporter on replicated systems as part of the Kubernetes clusters. This is being exposed by Kubernetes via by kube-rbac-proxy

  • tcp 0 0 172.22.1.5:9100 0.0.0.0:* LISTEN 20363/./kube-rbac-p
  • tcp 0 0 127.0.0.1:9100 0.0.0.0:* LISTEN 18361/node_exporter

which is part of the node-exporter pod.

It is opening the port to the world as shown in the Netstat trace. It is doing so through a little known Kubernetes resource called hostPort, think of this as the predecessor to nodePort, this is where the container can directly open a port on the hostVM. Below is the snippet from the Prometheus version, 0.33.0. By default, this uses the same certs as use for k8s internal cluster communication. Refer the post node-exporter-daemonset.yaml .

To address it currently is via Kustomize or manually updating it. It would probably have to be a feature request to modify the default certs.

The requirement is that port 9100, which is being opened by the kube-rbac-proxy container on the host VM, and that is accessible by other machines that have a network route to the host VM, be bound with a certificate that is not self-signed.



Resolution: It is advised to run the below step to disable exposing port 9100 from Kubernetes cluster,

kubectl patch daemonsets -n monitoring node-exporter --patch '{"spec": {"template": {"spec": {"hostNetwork": false}}}}'

The above command will disable the hostNetwork and port 9100 will not be exposed anymore.

  • If newer Prometheus addon (> 0.33.0) is run, check the Daemonset name and adjust the command accordingly.

To check the Daemonset running the current cluster, run

$ kubectl get daemonsets -n monitoring

NAME DESIRED CURRENT READY UP-TO-DATE AVAILABLE NODE SELECTOR AGE
node-exporter 1 1 1 1 1 kubernetes.io/os=linux 14h

and use the right NAME in kubectl patch command accordingly.

Post the kubectl patch command is run,

  1. Node-exporter pods are restarted automatically
  2. Port 9100 exposure will be disabled
  3. Check netstat or ping the 9100 port to verify.