Batching small messages, using long-lived connections, and tuning your prefetch count are standard recommendations for anyone sending a high volume
of messages. But when it comes to cutting down on network costs, there's more you can do.
Optimize your messages and traffic patterns
The most effective way to cut down on network costs is to reduce the amount of data being transferred. This can be done at the application level (in the consumer).
- Send only the essential data: Audit your message payloads and send only the essential data. For example, instead of a complete user profile, send just the user ID if that's all you need.
- Compress your payload: For messages that are still large, such as serialized data structures or images, compress your payload before publishing the message. A simple compression algorithm can drastically reduce the data size, leading to significant savings on bandwidth and improving message processing efficiency.
- Batch small messages: For high-volume, low-latency use cases, consider batching small messages into a single, larger message. This reduces the overhead of TCP packets and RabbitMQ protocol headers per message.
- Use long-lived connections: Establishing a new TCP connection is an expensive operation in terms of both latency and network packets. Implement long-lived connections in your application and multiplex them with channels. A single connection can serve multiple threads, reducing the number of costly connection handshakes.
- Tune your prefetch count: A properly tuned prefetch count is recommended for both performance and cost. It controls how many messages a consumer receives at once. If it’s set too high or low, it can lead to wasted network bandwidth. We recommend checking your prefetch settings. Here are some tips on how to optimize prefetch count.
The cost of redelivery
An often-overlooked source of network costs is message redelivery. In a system where consumers fail to process a message and reject it (or the connection is lost), RabbitMQ will redeliver that message. This can be a source of unexpected network costs.
- Transient failures: If your consumers are experiencing transient failures (e.g., database connection timeouts), a message can be redelivered repeatedly, each time adding to your network bill. A robust consumer should be designed to handle these failures gracefully. It should either let the message go to a dead-letter queue or wait for a progressively longer period before each new attempt, rather than immediately rejecting the message.
- Network instability: Unstable network connections between the broker and the consumer can cause messages to be redelivered. Each redelivery is a new network transfer, which adds to your cost.
By implementing these strategies, you can take control of your cloud spending and ensure that RabbitMQ remains a cost-effective and powerful component of your microservices architecture.
If you want help to reduce your costs, don’t hesitate to contact us at support@cloudamqp.com.
