Guides

Why a database is not always the right tool for a queue based system

Why a database is not always the right tool for a queue based system
A database can handle a job queue, and for many applications, it might be all you need. This post explores when that works well and when a message broker is the better choice.

Originally published in November 2015. Updated in November 2026.

As your application grows, with more workers, more background jobs, and more services depending on each other, managing a queue in a database can become increasingly complicated.

This is where a message broker such as RabbitMQ or LavinMQ can help.

But how do you know when it’s time to introduce one? Let’s walk through a practical example.

A simple PDF conversion service

Imagine you’re building a web application that allows customers to upload text documents, convert them into PDFs, and receive the finished files by email.

The process is straightforward:

  1. A customer uploads a document.
  2. Your application converts the document into a PDF.
  3. The completed PDF is emailed to the customer.

This is a simple linear flow — until many customers start uploading at once.

Database for a queue based system

Now imagine hundreds of customers uploading documents at the same time. Each conversion takes a few seconds, and you don’t want customers waiting for their PDFs while your web server handles every request.

You need a way to accept requests quickly and process them in the background.

A queue seems like a natural solution. But where should that queue live?

Using a database as a job queue

If your application already uses PostgreSQL or MySQL, creating a database table for background jobs might seem like the simplest option.

For example, you could create a table called pdf_jobs containing the document ID, customer email, and processing status.

When a customer uploads a document, your application inserts a new record:

INSERT INTO pdf_jobs (document_id, email, status)
VALUES (123, 'customer@example.com', 'NEW');

A background worker periodically checks the table for new jobs:

SELECT *
FROM pdf_jobs
WHERE status = 'NEW'
ORDER BY created_at
LIMIT 1;

Once it finds a job, the worker updates its status to PROCESSING, converts the document, sends the email, and marks the job as FINISHED.

The diagram below shows how a database-backed queue fits into the application flow — multiple workers polling the same table, each checking for available rows.

Database vs message queue

For a small application, this approach can work perfectly well. You already have a database, and there’s no additional infrastructure to manage.

But what happens when the number of requests increases?

What changes when you add more workers?

Let’s say your application receives 1,000 PDF conversion requests, and a single worker cannot process them quickly enough.

You add five more workers.

Now all six workers are checking the same database table for new jobs.

Without proper coordination, two workers might pick up the same job and process it twice. That could mean converting the same document twice or sending duplicate emails.

To prevent this, you need a mechanism that allows workers to claim jobs safely. Each worker must acquire an exclusive lock on a row, process it, and release the lock — a cycle that has to be managed carefully under concurrent load.

transactions

Modern databases have ways to handle this. PostgreSQL, for example, supports FOR UPDATE SKIP LOCKED, which allows workers to skip rows already locked by other transactions.

This is useful for building database-backed queues, but it doesn’t solve every problem.

You still need to decide what happens when a worker crashes, how long a job can remain in progress, when a failed job should be retried, and how completed jobs should be cleaned up.

None of these problems is impossible to solve with a database. The question is whether you want to build and maintain that functionality yourself.

Polling and database load

Another consideration is how workers discover new jobs.

In a typical database-backed queue, workers repeatedly query the database looking for available work.

If they check too frequently, they generate unnecessary database traffic. If they check less frequently, jobs may wait longer before being processed. This tradeoff is a fundamental constraint of polling.

Frequent DB queries

PostgreSQL also supports LISTEN/NOTIFY, which can help applications react to new work without relying entirely on frequent polling. However, notifications aren’t a durable replacement for the job records themselves.

As your workload grows, queue-related queries, updates, indexes, and cleanup operations compete with the rest of your application’s database activity.

This is one reason teams eventually separate background job processing from their primary database.

Using a message broker instead

Now let’s look at the same PDF conversion application using a message broker.

Instead of inserting every new job into a database queue table, the application publishes a message to LavinMQ or RabbitMQ.

The message might look like this:

{
  "job_id": "pdf-123",
  "document_id": 123,
  "email": "customer@example.com"
}

The broker routes the message to a queue, where it waits for an available worker.

Workers subscribe to the queue and receive jobs as they become available. They don’t need to continuously query a database to discover new work.

When a worker receives a message, it converts the document and sends the email. Once processing succeeds, it acknowledges the message.

The broker can then remove the acknowledged message from the queue.

If the worker crashes before acknowledging the message, the broker can make it available for redelivery.

This is one of the main advantages of using a message broker: message delivery and worker coordination are handled by infrastructure designed for that purpose.

What about duplicate processing?

There is an important detail here.

Imagine a worker successfully converts a document and sends the email, but crashes before acknowledging the message.

The broker doesn’t know that the work was completed. It may deliver the same message again.

This is known as at-least-once delivery. It helps prevent work from being silently lost, but it also means your application must be prepared to handle duplicate messages.

For our PDF service, we could use the job_id to check whether the job has already been completed before performing an action that shouldn’t happen twice.

Message acknowledgements, durable queues, persistent messages, and publisher confirms all play different roles in reliable messaging. They work together, but no single feature guarantees that every business operation happens exactly once.

Scaling background processing

Let’s return to our PDF application.

Initially, you might have two workers processing documents. As more customers arrive, the queue starts growing.

With a message broker, you can add more consumers to the same queue to process jobs concurrently.

The broker distributes messages among available consumers, taking factors such as consumer availability and prefetch settings into account.

This makes it easier to scale background processing independently of the web application.

Of course, adding workers doesn’t automatically solve every performance problem. The PDF conversion service still needs enough CPU and memory, and the broker and downstream services must be able to handle the additional load.

But the architecture gives you a useful separation: the web application accepts requests, the broker buffers and distributes work, and the workers process it.

The same pattern applies to AI pipelines

When an application dispatches tasks to a language model, runs inference in the background, or coordinates steps across multiple AI agents, the tradeoffs are identical to our PDF example. A message broker distributes inference work across many workers, handles retries when a model call fails or times out, and decouples the parts of the system that submit work from the parts that execute it. As AI workloads grow heavier and more concurrent, message brokers fit for exactly the same reasons they fit for PDF conversion.

When a database is still the right choice

Not every application needs a message broker.

If you’re building a small application with a handful of background jobs, a database-backed queue may be simpler to deploy, maintain, and debug.

For example, if your application sends a few hundred emails per day, introducing a separate broker might add more operational complexity than it removes.

It is also worth noting that mature libraries handle much of the database queue complexity for you. pg-boss (Node.js) and Procrastinate (Python) are purpose-built for PostgreSQL job queues, implementing locking, retries, and cleanup without you writing any of it. Sidekiq (Ruby) does the same on top of Redis. Using one of these is a reasonable middle ground before introducing a dedicated broker.

A message broker becomes more attractive when you need multiple independent consumers, flexible message routing, high volumes of asynchronous work, or more control over message delivery and retries.

Here’s a practical way to think about the decision:

Your application needs Consider
A small number of background jobs Database-backed queue
Simple jobs within an existing application Database-backed queue
Multiple workers processing jobs concurrently Either, depending on workload
Events delivered to several independent services Message broker
Flexible routing, acknowledgements, and dead-letter queues Message broker
Long-term records and complex queries Database
Business data plus asynchronous background processing Database and message broker

The important thing is to choose based on your application’s requirements, not assume one technology is always better.

Using a database and message broker together

In many production applications, the best solution is to use both.

For our PDF service, the database could store customer information, document metadata, and the final processing status.

LavinMQ could handle the conversion requests and distribute them among workers.

The database remains the source of truth for application data, while the broker manages asynchronous communication.

There is one additional challenge worth considering: what happens if the application successfully saves a job in the database but fails to publish its message?

One common solution is the transactional outbox pattern. The application saves both the business record and an event record in the same database transaction. A separate process then publishes the event to the broker, retrying when necessary.

This avoids relying on two independent operations succeeding at exactly the same time.

Getting started with LavinMQ

If your application has outgrown a simple database-backed queue, LavinMQ is one option for handling background jobs.

LavinMQ supports AMQP 0-9-1, allowing applications to use familiar messaging concepts such as exchanges, queues, routing keys, acknowledgements, and publisher confirms.

For our PDF application, the publisher side might look like this in Python:

import pika, json

connection = pika.BlockingConnection(pika.URLParameters("amqps://your-cloudamqp-url"))
channel = connection.channel()

channel.queue_declare(queue="pdf_jobs", durable=True)

channel.basic_publish(
    exchange="",
    routing_key="pdf_jobs",
    body=json.dumps({"job_id": "pdf-123", "document_id": 123, "email": "customer@example.com"}),
    properties=pika.BasicProperties(delivery_mode=2),  # persist across broker restart
)

connection.close()

And the worker that consumes and acknowledges:

def on_message(channel, method, properties, body):
    job = json.loads(body)
    convert_and_email(job)
    channel.basic_ack(delivery_tag=method.delivery_tag)

channel.basic_qos(prefetch_count=1)  # one job at a time per worker
channel.basic_consume(queue="pdf_jobs", on_message_callback=on_message)
channel.start_consuming()

delivery_mode=2 makes messages durable so they survive a broker restart. basic_ack tells the broker the job completed — if the worker crashes before acking, the message is redelivered automatically.

You can run LavinMQ yourself or use a managed instance through CloudAMQP — the free plan is enough to get started.

For a first implementation, start with one queue and one worker. Test what happens when the worker crashes, when processing fails, and when the application publishes more jobs than the worker can handle. Those edge cases will teach you more about reliable messaging than any amount of reading.

Final thoughts

Databases are excellent at storing and querying information. They can also support job queues, especially when the workload is relatively simple.

Message brokers are built to handle communication between applications, distribute work among consumers, and manage message delivery.

The distinction isn’t that databases cannot handle queues. It’s that a dedicated message broker can remove much of the delivery and coordination work that applications would otherwise need to implement themselves.

Start with the simplest architecture that meets your needs. When background processing, concurrency, and service communication become harder to manage, that’s a good time to consider a message broker.

And in many cases, the right answer isn’t choosing between a database and a broker. It’s letting each do what it does best.

If you’re ready to get started, check out our article: RabbitMQ for beginners, what is RabbitMQ?