How LavinMQ implements Streams with AMQP 0.9.1 and how it differs from RabbitMQ

High-performance messaging with Streams

Streaming offers a set of capabilities that differ from queueing. For example, unlike traditional queues, streams do not remove messages after consumption; instead, they keep them in a log that allows one or multiple consumers to read from it at any time, and/or replay messages as needed.

Both LavinMQ and RabbitMQ implement the AMQP 0.9.1 protocol, and both offer streaming. However, when RabbitMQ released stream queues in June 2021, it came with an additional stream protocol. LavinMQ on the other hand choose to implement all streaming features within the AMQP protocol.

From queueing to streaming - the adoption

Before diving into this topic, it's worth mentioning that RabbitMQ and LavinMQ are two very different brokers that both share support for AMQP 0.9.1. When RabbitMQ was released in 2007, streaming was hardly even a term used in IT architectures. Adopting streaming, therefore, required adjustments to an already existing software. For implementing streams to RabbitMQ, three key streaming capabilities needed to be adopted:

• First: It needed persistent message storage where messages remain available for replay instead of being removed after acknowledgment.
• Second: It required increased throughput by reducing round-trip and message copying.
• Third: It needed efficient multi-consumer access without message duplication and the ability to store a huge amount of messages long-term.

And, even though the core benefit of a stream: the persistent, append-only log, is achievable with AMQP, RabbitMQ 3.9 introduced a stream plugin with a new binary protocol to reach the performance goals.

LavinMQ: Built for high I/O performance

Circling back to LavinMQ, it was first released in 2020. By this time - streaming was a well-known term in the technical world. The architects behind LavinMQ knew that they needed a strategy for the easy implementation of streams, and as these were introduced in LavinMQ version 1.2.0, performance showed great results. The LavinMQ broker is optimized to run streams, allowing users to continue using the same AMQP tools and libraries that they already use.

Streaming with CloudAMQP

CloudAMQP offers hosting for both LavinMQ and RabbitMQ. The implementation of stream queues in a hosted environment depends on your specific requirements and goals. Here's a summary to help you navigate your options.

Running RabbitMQ and want to start streaming?

RabbitMQ stream queues using AMQP

The essence of a stream queue is achievable over AMQP 0.9.1. However, running RabbitMQ stream queues on AMQP requires consideration of your throughput requirements and offset management needs. It's noteworthy that if a client disconnects or restarts, applications need to implement their own mechanism for tracking and restoring the consumer's position (offset) in the stream.

RabbitMQ stream queues using the streams protocol

Using a client that supports the stream binary protocol gives you a lot of additional benefits with using stream queues. One is performance, it’s much faster, you also get the possibility of sharding, meaning that you can consume from streams that are on different nodes in the cluster and without the proxying that happens using the AMQP protocol. Additionally you also get support for broker side storage of offset, so when you restart your consumer it will pick up where you left off automatically. The downside of this is that it’s not the normal AMQP you are used to, you need a new client and has different semantics and this might force you to change more that you wish for in your application.

Running LavinMQ and want to start streaming?

This option has very low thresholds. Just declare streams just like any other queues using an AMQP client. When declaring a new queue; set x-queue-type to stream and ensure durable=true as LavinMQ requires all streams to be durable.

New to streaming?

If you are entirely new to streaming and are interested in trying it out, the first thing you need to do is determine the requirements of your setup. You can create a free instance on CloudAMQP and, from that moment, set up a streaming environment.

For a deeper understanding of streaming concepts and how they can benefit you, we recommend the following articles:

• A deepdive into RabbitMQ streams (three part blog series)
• LavinMQ streams vs. RabbitMQ stream vs. Kafka
• LavinMQ stream Queues - feature walkthrough

Conclusion

Both RabbitMQ and LavinMQ support streams using the AMQP 0.9.1 protocol, but they're fundamentally different brokers.

For RabbitMQ, using stream-queues over the AMQP protocol is possible, although RabbitMQ provides significantly higher throughput with its dedicated streams protocol. The main limitation when using AMQP for streaming is offset management. You must manually track consumer positions, and if a client disconnects, you risk losing track of which offset was last processed. If you wish to run RabbitMQ with the streams protocol on CloudAMQP, the protocol must be treated as a separate client.

LavinMQ doesn't need a special protocol for streaming as it achieves impressive benchmark numbers over AMQP 0.9.1. The broker was optimized to be fast and efficient, and its stream features are compatible with most AMQP clients. This means developers can implement streaming functionality without needing to adopt a new protocol or rewrite existing code.

If LavinMQ caught your interest and you want to learn more, the LavinMQ ebook is free for download and will give you an introduction to message queueing and streaming.

If you have any questions or thoughts about this blog, reach out to us at contact@cloudamqp.com