AKS Proxy Monitoring Missing Parameter

[system]

Data missing in Grafana dashboards

Issue Description

The dashboard data in Grafana is missing and error 'Failed to fetch' is present in the UI.

Also, there are no namespaces to choose from.

When checking the monitoring-grafana pod logs, there is an error like the one below:

msg="Internal server error" error="[plugin.downstreamError] client: failed to call resources: error querying resource: Post \"http://monitoring-kube-prometheus-prometheus.monitoring:9090/api/v1/series\": net/http: request canceled (Client. Timeout exceeded'

Resolution

monitoring-kube-prometheus-prometheus.monitoring was missing in the no_proxy values in the proxy config.

Besides the ones mentioned in the docs here, monitoring-kube-prometheus-prometheus.monitoring needs to be added also.

AKS Cluster Proxy settings need to be updated with the below command:

az aks update --name name_of_the_cluster --resource-group resource_group_name --http-proxy-config proxy_config.json

Rollout restart the monitoring deployment: kubectl rollout restart deployment -n monitoring