Federating System and User metrics to S3 in Red Hat OpenShift for AWS

This guide walks through setting up federating Prometheus metrics to S3 storage.

ToDo - Add Authorization in front of Thanos APIs

Prerequisites

• A ROSA cluster deployed with STS
• aws CLI

Set up environment

1. Create environment variables

2. Create namespace

AWS Preperation

1. Create an S3 bucket

2. Create a Policy for access to S3

3. Apply the Policy

4. Create a Trust Policy

5. Create Role for AWS Prometheus and CloudWatch

6. Attach the Policies to the Role

Deploy Operators

1. Add the MOBB chart repository to your Helm

2. Update your repositories

3. Use the mobb/operatorhub chart to deploy the needed operators

Deploy Thanos Store Gateway

1. We use Grafana Alloy to scrape the prometheus metrics and ship them to Thanos, which will then store them in S3. Currently Grafana Alloy requires running as a specific user so we must set a SecurityContextConstraint to allow it.

2. Deploy ROSA Thanos S3 Helm Chart

3. Append remoteWrite settings to the user-workload-monitoring config to forward user workload metrics to Thanos.

   Check if the User Workload Config Map exists:

   If the config doesn’t exist run:

   Otherwise update it with the following:

Check metrics are flowing by logging into Grafana

1. Get the Route URL for Grafana (remember its https) and login using username root and the password you updated to (or the default of secret).

2. Once logged in go to Dashboards->Manage and expand the federated-metrics group and you should see the cluster metrics dashboards. Click on the Use Method / Cluster Dashboard and you should see metrics. \o/.

   

Cleanup

1. Delete the Helm Charts

2. Delete the namespace

3. Delete the S3 bucket

4. Delete the AWS IAM Role and Policy