What is new in RabbitMQ 3.8?

The past two releases of RabbitMQ featured increased security and liability. The newest RabbitMQ version, 3.8, brings authorization and failover features along with increased improvements to components such as logging and visualization. CloudAMQP wants to welcome RabbitMQ 3.8 by highlighting some of the new features.

Feature Flags and Rolling Upgrades

The feature flag mechanism allows RabbitMQ nodes of different versions to determine if they are compatible with each other, regardless of their version. The feature flags mechanism controls features that are enabled for the cluster and indicate a RabbitMQ node's capabilities to other nodes in the cluster. Essentially, feature flags are a mechanism that controls enabled or available features on all cluster nodes. Coupling between one node and a cluster is only allowed when critical features match. A node can join or re-join a cluster only if it supports all feature flags enabled in the cluster and if all the feature flags are enabled on that node.

The feature flag mechanism simplifies upgrades, since it makes it possible for two nodes with different versions to live in the same cluster. You might be able to do a rolling upgrade of cluster members without needing to shut down the cluster entirely. Cluster-wide shutdowns might be necessary for versions depending on the changes. Read more at rolling upgrades here in new versions: https://www.rabbitmq.com/upgrade.html#rolling-upgrades

Upgrades in CloudAMQP

CloudAMQP continues to perform rolling upgrades (taking down only one node at the time) for patch versions of RabbitMQ. Upgrading between minor or major versions still result in some downtime, as with Erlang upgrades. We are working on a new automated processes for rolling upgrades for later versions.

How can I secure RabbitMQ?

Certain laws in Europe and the United States are in place to guarantee privacy. Eliminating backdoors in a RabbitMQ deployment is not always enough to avoid unwanted access. A hierarchy needs to be created in order to assign roles and privileges only when absolutely necessary.

RabbitMQ OAuth2

The experimental RabbitMQ OAuth2 token authorization backend utilizing JWT tokens allows users to grant specific privileges to different users more thoroughly than through a standard authorization backend. The OAUth2.0 flow grants a secure token verified with each request. While any flow is supported, using the implicit grant is not recommended.

Scope and JWT Tokens

Token authentication is not a complete substitute for full authorization but allows developers to control resource access. Scopes allow administrators to limit access to functionality. In RabbitMQ, scopes must follow a specific style.

rabbitmq.read:*/*

A feature followed by a colon with a forward slash and a host grants access. Every node must be configured for the Oauth2 backend for the process to work.

What are quorum queues?

Quorum queues are a new queue type for RabbitMQ that is used as an alternative to durable mirrored queues. This queue type relies on the agreement between replicas in RabbitMQ clusters on its state and in the data. While this type of queue ensures that data is more than likely to be accurate, there are restrictions. Unlike other types, quorums are always durable; they are not exclusive or lazy, and at the time of this writing, quorums do not allow TTL nor do they react to memory alarms. Messages sent to quorums are always persistent, and quorum queues are always durable. Quorums can handle poison messages, which are messages that are deemed "problematic" and cannot be processed for some reason.

Fault tolerance works on consensus. The tolerated number of failures increases with the number of nodes in the cluster. A quorum queue is only available when a majority (quorum) of replicas are available. A quorum queue with a replication factor of three, on a three broker cluster, can only tolerate the loss of a single broker. If two brokers go down then the queue becomes unavailable to clients. Likewise, if a network partition splits the replicas into one group of two and another of one, then only the majority side can continue to function.

Likewise, message durability (consistency) is only guaranteed when a quorum remains available. A quorum queue will be unrecoverable if the data of a quorum of replicas is permanently lost.

When should I use quorum queues?

Quorum queues work well under specific circumstances. Nodes must agree while activity is monitored through logs. Avoid using quorum queues in situations where available memory is limited or latency is intolerable. Due to these limitations, use in supporting MQTT is not always advisable. MQTT is a high-speed machine-to-machine protocol that works well in IoT environments. Quorum queues are intended for use cases where queues exist for a long time and are critical to certain aspects of system operation; e.g. when tolerance and data safety is more important than low latency and advanced queue features, or where the potential of losing messages would have a significant impact on the system.

For a deep dive into Quorum Queues you better check out: RabbitMQ 3.8 Feature Focus - Quorum Queues

Does RabbitMQ support failover in individual consumers?

Building on the ability to easily upgrade a broker and roll out updates, RabbitMQ 3.8 supports the use of a single active consumer. This capability allows for the definition of one consumer among many as active, promoting a high level of availability.

To use this feature, create a RabbitMQ queue and register consumers with the following header:

x-single-active-consumer=true;

One consumer is randomly chosen as currently active. The remaining nodes remain on standby until the chosen node is unreachable.

The management UI and CLI, through the list_consumers command, provides information on the currently active consumer. Each maintains an active state flag.

Data durability in RabbitMQ

Changes to the latest version of RabbitMQ promote data durability and availability. Thanks to these changes, nearly continuous uptime is achievable with support for upgrades. Quorum queues and single active consumers provide data safety.

For a deep dive into single active consumers you better check out: RabbitMQ 3.8 Feature Focus - Single Active Consumer

Monitoring in RabbitMQ version 3.8

Administrators need tools that track the use of clusters. Auditing is often a requirement and even legally required in the case of a growing number of privacy laws.

Prometheus and Grafana support

As of version 3.8.0, RabbitMQ has built-in Prometheus and Grafana support. Grafana laboratories provide dashboards that integrate with clusters and new monitoring options. RabbitMQ also works with Prometheus to report on failures in real-time. CloudAMQP are working on a solution to enable Prometheus and Grafana support.

Feature Changes in RabbitMQ 3.8

Administrators now have more control over production deployment than before thanks to the continuous-evolving RabbitMQ. The latest iteration brings support for constantly available clusters, rolling upgrades, data safety, authorization, and logging. The RabbitMQ API documentation hints at more enhancements in the near future.