Cluster migration with RabbitMQ Queue Federation

RabbitMQ Federation plugin can be used when migrating to another cluster without stopping all producers and consumers while doing so. This article explains how to migrate between two clusters with help of queue federation.

This article is a living document that is continually updated. Last updated January 2022.

Queue federation connects an upstream queue to transfer messages to the downstream queue when there are consumers that have capacity on the downstream queue. It's perfect to use when migrating between two clusters. Consumers and publishers can be moved in any order and the messages won't be duplicated (which is the case if you do exchange federation). Instead, messages are transferred to the new cluster when your consumers are there. The federated queue will only retrieve messages when it has run out of messages locally, it has consumers that need messages, and the upstream queue has "spare" messages that are not being consumed.

Let us start by defining the concept of upstream and downstream servers. Upstream servers are the servers towards messages are published initially. Downstream servers are where the messages get forwarded to. The downstream cluster can be thought of as subscribing to messages from the upstream cluster.

Queue Federation
Instance Map

Queue Federation example setup

In this article we have our cluster set up in Amazon, in US-East-1 (in the image marked with a star). We need to migrate to a cluster in EU-West-1 (in the image marked with a moon). The servers in US-East-1 will be defined as the upstream servers of the servers in EU-West-1.

The old cluster in US-East-1 has some queues that should be federated, the metric-queues (metric.cpu and metric.memory).

Upstream Server Queues
  1. Set up the new cluster

    • Start by setting up the new cluster. (In this case a new cluster in EU-West-1).

      It is possible to copy all configuration settings (queues, exhcanges, enabled plugins etc.) from an old CloudAMQP cluster to a new one by selecting "copy settings (optional)" in step 3 of 4, as in the image below.
  2. Create a policy that matches the queues that you would like to federate.

    A policy is a pattern that queue names are matched against. The "pattern" argument is a regular expression used to match queue (or exchange) names. In this case, we tell the policy to federate all queues whose name begin with "metric.".

    • Navigate to 'Admin' -> 'Policies' and press 'Add / update a policy' to create the policy.
      A policy can apply to an upstream set or a single upstream of exchanges and/or queues. In this example we apply to all upstreams, federation-upstream-set is set to all. set up policy NOTE: Policies are matched every time an exchange or queue is created.
  3. Set up the federation to the upstream server.

    • Open the management interface for the downstream server (EU-West-1) and go to the 'Admin'->'Federation Upstreams' screen and press 'Add a new upstream'. Fill in all information needed, the URI should be the URI of the upstream server.
    • Leave expiry time and TTL blank. Leave this blank means that it will stay "forever". set up federation
  4. Set up the queues that should be federated.

    • The queues can be created as you always create your queues (i.e manually from the management interface or from the application code). The queues should be marked as federated if the policy is applied, like in the image below. If the queues are not marked as federated, check that the regexp is working. federation queues
  5. Everything is now set up.

    It is time to move the publisher OR the consumer to the new cluster. You can move the publisher and/or the consumer in any order. The federated queue will only retrieve messages when it has run out of messages locally when it has consumers that need messages, or when the upstream queue has "spare" messages that are not being consumed.

  6. Verify that messages are consumed

    Verify that the messages published to the upstream server are consumed by the downstream server when the consumers are moved to the downstream server.

Note: when using queue federation, the original routing key of the messages will be removed as they are used to publish towards the downstream cluster. If your consumers rely on this information we recommend using shovels instead.

Please email us at contact@cloudamqp.com if you have any suggestions, questions or feedback.

CloudAMQP - industry leading RabbitMQ as a service

Start your managed cluster today. CloudAMQP is 100% free to try.

13,000+ users including these smart companies