Prometheus Remote Write

The Prometheus Remote Write integration allows you to export Prometheus metrics from your CloudAMQP cluster to any Prometheus remote write endpoint, such as self-hosted Mimir, Thanos or VictoriaMetrics. Available on RabbitMQ and LavinMQ dedicated instances.

If you send metrics to Grafana Cloud, use the dedicated Grafana Cloud integration instead. It is preconfigured for Grafana Cloud authentication.

Endpoint

The endpoint is the full URL of your remote write receiver, including the path. It must use HTTPS. The path differs between receivers:

https://mimir.example.com/api/v1/push
https://metrics.example.com/api/prom/push
https://thanos.example.com/api/v1/receive
https://vm.example.com/api/v1/write

Authentication

Four authentication modes are available:

None
No credentials are sent as such. Use this when the receiver carries its token in the endpoint URL as a query parameter, for example ?token=your-token
Basic Auth
A username and a password or token are sent as an HTTP Basic Authorization header. This is what most managed and proxied remote write endpoints expect.
Headers
The credentials live in the headers below, for example Authorization: Bearer your-token Use this for bearer tokens and for receivers that expect an API key header.
OAuth2
We fetch an access token with the OAuth2 client credentials flow and refresh it as needed. Enter the client ID, the client secret and the token URL of your identity provider, plus any scopes it requires.
Authentication runs through the OpenTelemetry Collector oauth2client extension. Of its options we expose four, with the same names it uses: client_id , client_secret , token_url and scopes . Enter scopes space or comma separated, and we pass them on as a list.

Azure managed Prometheus

Azure Monitor accepts remote write with an OAuth2 access token from Microsoft Entra. Copy Metrics ingestion endpoint from the Overview page of your Azure Monitor workspace and use it as the endpoint, then set the authentication mode to OAuth2 with the client ID and client secret of an Entra app registration and these values:

token_url: https://login.microsoftonline.com/<tenant-id>/oauth2/v2.0/token
scopes:    https://monitor.azure.com//.default

The double slash in the scope is intentional: the resource is https://monitor.azure.com/ and .default is appended to it.

Assign the app the Monitoring Metrics Publisher role on the data collection rule linked from the workspace Overview page, not on the workspace itself. Role assignments on the workspace do not reach the data collection rule. The assignment takes up to 30 minutes to take effect, and until it does Azure answers every write with 403 and the message The authentication token provided does not have access to ingest data for the data collection rule . That is expected during those minutes, and the integration retries on its own.

Headers

Headers are sent with every request, whichever authentication mode you pick, so you can combine them with basic auth. Enter one header per line in the format key: value

Multi-tenant receivers need a tenant header to know which tenant the metrics belong to, and reject writes without it. Mimir and Cortex read X-Scope-OrgID and Thanos reads THANOS-TENANT. Leave the header out if your receiver runs in single-tenant mode, or if an authenticating proxy in front of it sets the header for you.

Setup

  1. Enter the remote write endpoint of your receiver.
  2. Select an authentication mode and fill in the credentials it needs.
  3. Add any headers your receiver expects, including a tenant header if it is multi-tenant.
  4. Click Save.

All metrics are exported with the labels/tags instance and cluster, which are populated with the name of the server and cluster. You can also set custom tags when configuring the integration.

Exported server metrics

Metric Metric labels CloudAMQP labels
system_cpu_utilization_ratio state instance, cluster
system_disk_io_bytes_total direction instance, cluster
system_disk_operation_time_seconds_total direction instance, cluster
system_disk_operations_total direction instance, cluster
system_filesystem_usage_bytes state instance, cluster
system_filesystem_utilization_ratio instance, cluster
system_memory_usage_bytes state instance, cluster
system_memory_limit_bytes instance, cluster
system_network_io_bytes_total direction instance, cluster
system_paging_usage_bytes state instance, cluster

Exported LavinMQ metrics

Metric Metric labels CloudAMQP labels
lavinmq_global_messages_acknowledged_total protocol, queue_type instance, cluster
lavinmq_global_messages_delivered_total protocol, queue_type instance, cluster
lavinmq_global_messages_redelivered_total protocol, queue_type instance, cluster
lavinmq_global_messages_confirmed_total protocol instance, cluster
lavinmq_connections instance, cluster
lavinmq_channels instance, cluster
lavinmq_consumers instance, cluster
lavinmq_disk_space_available_bytes instance, cluster
lavinmq_process_open_fds instance, cluster
lavinmq_process_resident_memory_bytes instance, cluster
lavinmq_queues instance, cluster
lavinmq_queue_messages instance, cluster
lavinmq_queue_messages_ready instance, cluster
lavinmq_queue_messages_unacked instance, cluster

Exported RabbitMQ metrics

Metric Metric labels CloudAMQP labels
rabbitmq_global_messages_acknowledged_total protocol, queue_type instance, cluster
rabbitmq_global_messages_delivered_total protocol, queue_type instance, cluster
rabbitmq_global_messages_delivered_get_manual_ack_total protocol, queue_type instance, cluster
rabbitmq_global_messages_delivered_get_auto_ack_total protocol, queue_type instance, cluster
rabbitmq_global_messages_redelivered_total protocol, queue_type instance, cluster
rabbitmq_global_messages_confirmed_total protocol instance, cluster
rabbitmq_global_messages_unroutable_dropped_total protocol instance, cluster
rabbitmq_global_messages_unroutable_returned_total protocol instance, cluster
rabbitmq_alarms_memory_used_watermark instance, cluster
rabbitmq_alarms_free_disk_space_watermark instance, cluster
rabbitmq_connections instance, cluster
rabbitmq_channels instance, cluster
rabbitmq_consumers instance, cluster
rabbitmq_disk_space_available_bytes instance, cluster
rabbitmq_exchange_messages_published_total instance, cluster
rabbitmq_process_open_fds instance, cluster
rabbitmq_process_resident_memory_bytes instance, cluster
rabbitmq_queues instance, cluster
rabbitmq_queue_get_total instance, cluster
rabbitmq_queue_get_ack_total instance, cluster
rabbitmq_queue_messages instance, cluster
rabbitmq_queue_messages_acked_total instance, cluster
rabbitmq_queue_messages_delivered_ack_total instance, cluster
rabbitmq_queue_messages_persistent instance, cluster
rabbitmq_queue_messages_published_total instance, cluster
rabbitmq_queue_messages_ready instance, cluster
rabbitmq_queue_messages_unacked instance, cluster
rabbitmq_queue_head_message_timestamp instance, cluster

Metrics Filtering

The metrics listed above are the metrics exported by default. However, you can use the metrics_filter endpoint to specify exactly which metrics you want to export. See the API documentation for more details.

The complete list of available metrics that can be included in the metrics filter includes:

  • All metrics available from the server's /metrics endpoint (broker metrics)
  • All metrics available from the server's /metrics/detailed endpoint (detailed broker metrics). Note: Detailed metrics can be resource-intensive on servers with many objects such as connections and queues.
  • Server metrics:
    system_cpu_utilization_ratio
    system_disk_io_bytes_total
    system_disk_operation_time_seconds_total
    system_disk_operations_total
    system_filesystem_usage_bytes
    system_filesystem_utilization_ratio
    system_memory_limit_bytes
    system_memory_usage_bytes
    system_linux_memory_available_bytes
    system_network_io_bytes_total
    system_paging_usage_bytes

Histograms and summaries are selected by their family name, for example rabbitmq_message_size_bytes for the RabbitMQ message size histogram. The collector merges the bucket, sum and count series into that single metric, so listing the family name exports all of them. The full list of names accepted by the filter for your instance is available from the valid_metrics endpoint.

← Back to Metrics Integrations

↑ Back to top