This blog series is a living document that is continually updated. Last updated February 2018.
RabbitMQ is a message-queueing software called a message broker or queue manager. Simply said; It is a software where queues can be defined, applications may connect to the queue and transfer a message onto it.
A message can include any kind of information. It could, for example, have information about a process/task that should start on another application (that could be on another server), or it could be just a simple text message. The queue-manager software stores the messages until a receiving application connects and takes a message off the queue. The receiving application then processes the message in an appropriate manner.
All articles from Getting Started with RabbitMQ can be downloaded as a free ebook here.
Table of Contents
RabbitMQ for beginners - What is RabbitMQ?
Gives a brief understanding of messaging and important RabbitMQ concepts are defined
RabbitMQ step-by-step coding instructions
Step-by-step instructions which show how to set up a connection, how to publish to a queue, and how to subscribe from the queue
The management interface
Describes how to monitor and handle your RabbitMQ server from a web browser
Exchanges, routing keys and bindings
Explains the different types of exchanges in RabbitMQ and how exchanges and queues are associated with each other
A message broker can act as a middleman for various services (e.g. a web application, as in this example). They can be used to reduce loads and delivery times by web application servers since tasks, which would normally take quite a bit of time to process, can be delegated to a third party whose only job is to perform them.
In this guide, we follow a scenario where a web application allows users to upload information to a website. The site will handle this information and generate a PDF and email it back to the user. Handling the information, generating the PDF and sending the email will in this example case take several seconds and that is one of the reasons of why a message queue will be used.
When the user has entered user information into the web interface, the web application will put a "PDF processing" - task and all information into a message and the message will be placed onto a queue defined in RabbitMQ.
The basic architecture of a message queue is simple, there are client applications called producers that create messages and deliver them to the broker (the message queue). Other applications, called consumers, connects to the queue and subscribes to the messages to be processed. A software can be a producer, or consumer, or both a consumer and a producer of messages. Messages placed onto the queue are stored until the consumer retrieves them.
When and why should you use RabbitMQ?
Message queueing allows web servers to respond to requests quickly instead of being forced to perform resource-heavy procedures on the spot. Message queueing is also good when you want to distribute a message to multiple recipients for consumption or for balancing loads between workers.
The consumer can take a message of the queue and start the processing of the PDF at the same time as the producer is queueing up new messages on the queue. The consumer can be on a totally different server than the publisher, or they can be located on the same server. The request can be created in one programming language and handled in another programming language - the two applications will only communicate through the messages they are sending to each other. Due to that, the two applications will have a low coupling between the sender and the receiver.
- The user sends a PDF creation request to the web application.
- The web application (the producer) sends a message to RabbitMQ, including data from the request, like name and email.
- An exchange accepts the messages from a producer application and routes them to correct message queues for PDF creation.
- The PDF processing worker (the consumer) receives the task and starts the processing of the PDF.
Messages are not published directly to a queue, instead, the producer sends messages to an exchange. An exchange is responsible for the routing of the messages to the different queues. An exchange accepts messages from the producer application and routes them to message queues with the help of bindings and routing keys. A binding is a link between a queue and an exchange.
Message flow in RabbitMQ
- The producer publishes a message to an exchange. When you create the exchange, you have to specify the type of it. The different types of exchanges are explained in detail later on.
- The exchange receives the message and is now responsible for the routing of the message. The exchange takes different message attributes into account, such as routing key, depending on the exchange type.
- Bindings have to be created from the exchange to queues. In this case, we see two bindings to two different queues from the exchange. The Exchange routes the message into the queues depending on message attributes.
- The messages stay in the queue until they are handled by a consumer
- The consumer handles the message.
Types of exchanges
In Part 2 of the tutorial are only direct exchanges used. Deeper understanding about the different exchange types, binding keys, routing keys and how/when you should use them can be found in part 4 about exchanges: Part 4: RabbitMQ for beginners - Exchanges, routing keys and bindings.
- Direct: A direct exchange delivers messages to queues based on a message routing key. In a direct exchange, the message is routed to the queues whose binding key exactly matches the routing key of the message. If the queue is bound to the exchange with the binding key pdfprocess, a message published to the exchange with a routing key pdfprocess will be routed to that queue.
- Fanout: A fanout exchange routes messages to all of the queues that are bound to it.
- Topic: The topic exchange does a wildcard match between the routing key and the routing pattern specified in the binding.
- Headers: Headers exchanges use the message header attributes for routing.
RabbitMQ and server concepts
Here are some important concepts that need to be described before we dig deeper into RabbitMQ. The default virtual host, the default user, and the default permissions are used in the examples that follow, but it is still good to have a feeling of what it is.
- Producer: Application that sends the messages.
- Consumer: Application that receives the messages.
- Queue: Buffer that stores messages.
- Message: Information that is sent from the producer to a consumer through RabbitMQ.
- Connection: A connection is a TCP connection between your application and the RabbitMQ broker.
- Channel: A channel is a virtual connection inside a connection. When you are publishing or consuming messages from a queue - it's all done over a channel.
- Exchange: Receives messages from producers and pushes them to queues depending on rules defined by the exchange type. In order to receive messages, a queue needs to be bound to at least one exchange.
- Binding: A binding is a link between a queue and an exchange.
- Routing key: The routing key is a key that the exchange looks at to decide how to route the message to queues. The routing key is like an address for the message.
- AMQP: AMQP (Advanced Message Queuing Protocol) is the protocol used by RabbitMQ for messaging.
- Users: It is possible to connect to RabbitMQ with a given username and password. Every user can be assigned permissions such as rights to read, write and configure privileges within the instance. Users can also be assigned permissions to specific virtual hosts.
- Vhost, virtual host: A Virtual host provides a way to segregate applications using the same RabbitMQ instance. Different users can have different access privileges to different vhost and queues and exchanges can be created so they only exist in one vhost.
At the beginning of this article series, we had one producer (the website application) and one consumer (the PDF processing application). If the PDF processing application crashes, or if there is a lot of PDF requests coming in the same time, messages would continue to stack up in the queue until the consumer starts again. It would then process all the messages, one by one.
Set up a RabbitMQ instance
To be able to follow this guide you need to set up a CloudAMQP instance or you need to download and install RabbitMQ. CloudAMQP is a hosted RabbitMQ solution, meaning that all you need to do is sign up for an account and create an instance. You do not need to set up and install RabbitMQ or care about cluster handling, CloudAMQP will do that for you. CloudAMQP can be used for free with the plan little lemur. Go to the plan page and sign up for any plan and create an instance.
When your instance is created, press on details for your instance to find your username, password and connection URL for your cloud hosted RabbitMQ instance.
Getting started with RabbitMQ
Immediately after a RabbitMQ instance has been created it is possible to send a message cross languages, platforms, and OS. This way of handling messages decouple your processes and creates a highly scalable system. You can now start by opening the management interface to get an overview of your RabbitMQ server.
The Management Interface - Management and Monitoring
RabbitMQ provides a web UI for management and monitoring of your RabbitMQ server. The RabbitMQ management interface is enabled by default in CloudAMQP and a link can be found on the details page for your CloudAMQP instance.
From the management interface, it is possible to handle, create, delete and list queues. It is possible to monitor queue length, check message rate, change and add users permissions much more.
More information about the management interface can be found in Part 3 - The management interface.
Publish and subscribe messages
RabbitMQ speaks a protocol called AMQP by default. To be able to communicate with RabbitMQ you need a library that understands the same protocol as RabbitMQ. You need to download the client-library for the programming language that you intend to use for your applications. A client-library is an applications programming interface (API) for use in writing client applications. A client library has several methods that can be used, in this case, to communicate with RabbitMQ. The methods should be used when you, for example, connect to the RabbitMQ broker (using the given parameters, hostname, port number, etc) or when you declare a queue or an exchange. There is a choice of libraries for almost every programming language.
Steps to follow when setting up a connection and publishing a message/consuming a message:
- First of all, we need to set up/create a connection object. Here, the username, password, connection URL, port etc, will be specified. A TCP connection will be set up between the application and RabbitMQ when the start method is called.
- Secondly, a channel needs to be opened. A channel needs to be created in the TCP connection. The connection interface can be used to open a channel and when the channel is opened it can be used to send and receive messages.
- Declare/create a queue. Declaring a queue will cause it to be created if it does not already exist. All queues need to be declared before they can be used.
- In subscriber/consumer: Set up exchanges and bind a queue to an exchange. All exchanges need to be declared before they can be used. An exchange accepts messages from a producer application and routes them to message queues. For messages to be routed to queues, queues need to be bound to an exchange.
Publish a message to an exchange
In subscriber/consumer: Consume a message from a queue.
- Close the channel and the connection.
Sample code will be given in part 2, starting with Part 2.1 - Ruby, followed by Part 2.2 - Node.js, and Part 2.3 Python, It is possible to have different programming languages on different parts of the system. The publisher could, for example, be written in node.js and the subscriber in Python.
Hope this article helped you get some understanding about RabbitMQ!
Please email us at firstname.lastname@example.org if you have any suggestions or feedback.