We're excited to announce that RabbitMQ 4.2 is now available on CloudAMQP! This version release features Khepri as the default metadata store for improved performance and scalability, along with local shovel enhancements, new message interceptors, CLI improvements, and other noteworthy updates.

Khepri feature flag marked as stable

By now, Khepri has become a familiar part of RabbitMQ. Built to replace Mnesia as more maintainable and scalable, Khepri stores RabbitMQ metadata, such as queue definitions, exchanges, bindings, users, and virtual hosts. Just as the new Raft-based metadata store in RabbitMQ, Khepri brings increased performance and more well-defined behavior in failure scenarios. This release is another major step towards Khepri.

After many releases in experimental state (since 3.13), the Khepri feature flag is now marked as stable. The 4.2 release marks Khepri as the default metadata store for new clusters. This means that if you just enable all stable feature flags after an upgrade (something we do for all CloudAMQP RabbitMQ clusters and highly recommend if you run your own RabbitMQ), Khepri will also be enabled in existing clusters.

As a user, you might not notice any difference, but as an operator, you need to get used to the differences. For example, in file layout, or how to check the health of the metadata store itself.

An observation made, and something that can be surprising with Khepri, is that you cannot define an exchange or queue (or make any metadata change) if only the smaller half of the cluster nodes are available.

Local shovel improvements

The RabbitMQ Shovel plugin is a familiar tool for our users; it behaves like a dedicated application that reliably moves or copies messages from a source (usually a queue) to a destination (usually an exchange or another queue). Historically, shovels used the AMQP 0.9.1 or AMQP 1.0 protocols to communicate with a remote or local broker, typically over TCP.

The new "local" shovel option, introduced in RabbitMQ 4.2, provides a more efficient way to move data within a single cluster. The 4.2 release also includes many general shovel-related fixes and performance improvements that benefit all shovel types, something we want to give a big shoutout to the RabbitMQ core team for!

Difference between a traditional AMQP Shovel and the new Local Shovel

The fundamental difference between a traditional AMQP Shovel and the new Local Shovel lies in how they communicate with the broker.

A standard AMQP Shovel operates exactly like any external client application: it must initiate a separate TCP connection to both the source and the destination and uses the standard AMQP protocol. The benefit is that the source and destination brokers don’t have to be on the same version, and don’t even have to be the same product as long as they support the appropriate AMQP protocol.

However, in the vast majority of cases, the shovel worker runs within either the source or the destination broker, often the same one. In this case, there is no need to maintain this interoperability. For a long time, RabbitMQ has supported a so-called “Direct” AMQP 0.9.1 connection that avoided an external TCP connection while still retaining all the other “ingredients” of a fully fledged network protocol.

Local Shovels go a long way further and call RabbitMQ's internal APIs directly. They don’t have to wrap each message in AMQP commands. This also means that while AMQP shovels start an additional pair of AMQP 0.9.1 channel or 1.0 session processes, Local Shovels only use a single shovel process. This architecture gives the Local Shovel an advantage by enabling a lower resource footprint and the highest possible throughput for internal operations such as message migration.

Message interceptors

An interesting internal feature in 4.2 is the addition of a new RabbitMQ extension point. This is similar to the existing channel interceptor mechanism, but differs in that it's not only AMQP 0.9.1-specific. The new message interceptors work with almost all protocols, such as AMQP 0.9.1 and 1.0, and MQTT 3 and 5. It also addresses former limitations. For example, here, messages are intercepted on both the incoming (publishing) and outgoing (delivery to consumers) sides.

Two new message interceptors ship with 4.2:

Outgoing timestamp: if you enable both the existing incoming timestamp and new outgoing timestamp annotations, you'll get an idea of how much time each message spent in the broker using your consuming clients. Neat, isn't it?
MQTT client ID annotation is an often-requested feature. It is just the first step, but message interceptors open the way to implement more elaborate message enrichment for IoT using 3rd-party plugins.

Minor CLI addition

Message size histogram metrics were added in 4.1. They are super helpful for gaining insight into your payload size distribution and fine-tuning your broker accordingly. But they were only exposed via the Prometheus endpoint.

In 4.2, you can also check the values via CLI using `rabbitmq-diagnostics message_size_stats`, which is very handy.

More…

We're happy that CQ per vhost message store crash with eof fixed (again)

Read complete 4.2 release notes here!

Get Started with RabbitMQ 4.2 on CloudAMQP

RabbitMQ 4.2 brings Khepri as the stable default, along with improvements to reliability, observability, and protocol support. To get started with a new instance, explore the CloudAMQP plans, and sign up for an account.

To check which version your RabbitMQ cluster is running, go to the "Versions" tab in the CloudAMQP Console. From there, you'll also find the option to upgrade to a newer version of RabbitMQ and Erlang. Read all about upgrades in this blog post.

Have questions about this release or need help upgrading? We're ready to help. Contact CloudAMQP support team!