Federating System and User metrics to Azure Blob storage in Azure Red Hat OpenShift

By default Azure Red Hat OpenShift (ARO) stores metrics in Ephemeral volumes, and its advised that users do not change this setting. However its not unreasonable to expect that metrics should be persisted for a set amount of time.

This guide shows how to set up Thanos to federate both System and User Workload Metrics to a Thanos gateway that stores the metrics in Azure Blob Container and makes them available via a Grafana instance (managed by the Grafana Operator).

ToDo - Add Authorization in front of Thanos APIs

Pre-Prequsites

1. An ARO cluster

2. Set some environment variables to use throughout to suit your environment

   Note: AZR_STORAGE_ACCOUNT_NAME must be unique

Azure Preperation

1. Create an Azure storage account

   modify the arguments to suit your environment

2. Get the account key and update the secret in thanos-store-credentials.yaml

3. Create a namespace to use

4. Add the MOBB chart repository to your Helm

5. Update your repositories

6. Use the mobb/operatorhub chart to deploy the grafana operator

7. Wait for the Operator to be ready

8. Use Helm deploy the OpenShift Patch Operator

9. Wait for the Operator to be ready

10. Deploy ARO Thanos Azure Blob container Helm Chart (mobb/aro-thanos-af)

    **> Note: enableUserWorkloadMetrics=true will overwrite configs for cluster and userworkload metrics. If you have customized them already, you may need to modify patch-monitoring-configs.yaml in the Helm chart to include your changes.

Validate Grafana is installed and seeing metrics from Azure Blob storage

1. get the Route URL for Grafana (remember its https) and login using username admin and the password password.

2. Once logged in go to Dashboards->Manage and expand the aro-thanos-af group and you should see the cluster metrics dashboards. Click on the Use Method / Cluster Dashboard and you should see metrics. \o/.

   Note: If it complains about a missing datasource run the following: oc annotate -n $NAMESPACE grafanadatasource aro-thanos-af-prometheus "retry=1"

   

Cleanup

1. Uninstall the aro-thanos-af chart

2. Uninstall the federated-metrics-operators chart

3. Delete the aro-thanos-af namespace

4. Delete the storage account