Serverless queue consumers

Function as a Service (FaaS) platforms, like AWS Lambda, Cloud Run Functions, and Azure Functions, are growing in popularity because they're easy to use and manage. However, integrating traditional message queues with FaaS can be a bit tricky.

Wait... don’t queue consumers need servers?

Imagine you've built your app completely serverless but need reliable messaging. Traditionally, you'd use a broker (LavinMQ, or RabbitMQ) and publish messages to a queue.

Usually, these message brokers need consumers; programs that continuously listen for messages, consume them from the queue, and process them. But if you think about it, running these long-lived consumers means managing servers or infrastructure, which doesn't fit well with a serverless approach.

I’ve been there, wondering how to integrate these traditional tools into a fully serverless setup.

Enter serverless queue consumers

The good news is that some brokers like LavinMQ can directly trigger your serverless functions by making HTTP calls. Here's how it works: when a message arrives in the queue, the broker sends an HTTP request to your function’s URL, automatically activating it to handle the message.

Consider this common scenario:

• A user uploads an image.
• Your application places a message on the broker’s queue, saying: "This image needs resizing."
• Instead of running a dedicated server waiting for messages, your broker instantly calls an HTTP endpoint.
• This HTTP call triggers your serverless function.
• The function resizes the image and saves it back to storage.

Instead of managing your own consumer apps, the broker itself becomes the consumer, triggering your code automatically. This way, your message broker fits smoothly into your serverless workflow without additional infrastructure. I find this approach not only minimal but also a novel take on integrating traditional queues into modern architectures.

Let someone else host the broker: Queueing as a service

One key consideration is deciding where your message broker will run. Since going serverless is about minimizing the work you have to manage, it makes sense to offload the broker as well.

The good news? Many cloud providers offer managed queue services, so you don’t have to run your own. For the demo that follows, we’ll use LavinMQ on CloudAMQP as the message broker.

That said, there are other managed options like Amazon MQ, which offers hosted RabbitMQ and ActiveMQ. However, it's worth noting that not all brokers support HTTP callbacks out of the box, and I can't confirm whether they do.

Now, let’s walk through a demo using LavinMQ on CloudAMQP with AWS Lambda.

Let’s build it: LavinMQ + AWS Lambda in action

LavinMQ has native support for HTTP callbacks. You simply provide it with an HTTP endpoint, and it automatically makes a request every time a message appears in your queue. This feature makes LavinMQ especially easy for serverless integration.

In the subsequent steps, we will see how to set up a managed LavinMQ broker on CloudAMQP and trigger Lambda actions for every message published to a queue.

Step 1: Create a managed LavinMQ instance

1. Log into or create a CloudAMQP account  and hit the  Create New Instance button
2. Give your instance a name and go with the LavinMQ Lemming plan – it’s free
3. Create a queue named webhook-queue from the management interface - we will be needing it later.

Step 2: Create a lambda function

1. Sign in to your AWS account and search for Lambda. Select it from the list of services.
2. Click Create function.
3. Make sure Author from scratch is selected.
4. Enter resize_image as the function name.
5. Choose Python 3.13 as the runtime.
6. Leave all other settings as they are and click Create function.
7. In the Function code section, replace the existing code in the lambda_function.py editor with the code below.

     import time
     import json
     import logging
   
     logger = logging.getLogger()
     logger.setLevel(logging.INFO)
   
     def lambda_handler(event, context):
         event_body = event.get('body', 'No body provided')
         logger.info("Received event with body: %s", event_body)
         
         logger.info("Starting simulated image resize task...")
         time.sleep(5)
         logger.info("Simulated image resize task complete.")
         
         response = {
             'statusCode': 200,
             'body': json.dumps('Simulated image resize complete!')
         }
         logger.info("Returning response: %s", response)
         
         return response

8. Click the Deploy button to save code changes.

Step 3: Testing the Lambda function

1. Switch to the Test tab
2. Keep  Create new event  selected.
3. Enter  ImageResizeTest  in the  Event name  field
4. Paste the json snippet below in the Event JSON field

   {
     "body": "Test payload for image resize simulation"
   }
   

   
5. Click Test to try out the function. If everything works, you should get the response below:

   {
     "statusCode": 200,
     "body": "\"Simulated image resize complete!\""
   }
   

Step 4: Configure an HTTP endpoint

Lambda provides two primary ways to call a function via an HTTP request: function URLs  and  API Gateway. We'll use function URLs, because they are simpler and work well for our needs in this blog.

A function URL provides your function's unique HTTP(S) address. Once set up, you can call it with any HTTP client (like a browser, curl, or Postman). In our case, LavinMQ uses this URL to trigger our Lambda function.

1. Open the resize_image function page, go to the Configuration tab and select Function URL.
2. Click Create function URL.
3. Choose NONE for Auth type - for brevity’s sake.
4. In Additional settings, select Configure cross-origin resource sharing (CORS) and keep the default wildcard (*) for Allow origin.
5. Click Save.
6. You can test your function URL with curl like so:

   curl -v '[your-url]/?message=HelloWorld' \ 
   -H 'content-type: application/json' \
   -d '{ "example": "test" }'
   

Step 5: Configure a LavinMQ webhook

To set up a webhook in LavinMQ, just create a shovel. You’ll need to specify the LavinMQ server, the source queue or exchange from which to pull messages, and the destination—an HTTP endpoint that can receive POST requests.

In this example, we’re creating a shovel named AWSLambdaShovel.

In the source section, we point it to the LavinMQ server and the webhook-queue we created earlier.

In the destination section, we enter the HTTP endpoint where the messages should be sent. Once LavinMQ detects a valid HTTP URL, the Type and Endpoint fields are automatically locked in.

Step 6: Testing

You can now test the entire setup by publishing a message to the webhook-queue. Every message published will trigger your Lambda function.

Conclusion

This setup really simplifies things. Instead of running extra code just to consume messages, you let the broker do the work and call your function when it’s needed. It keeps your architecture clean and lightweight. Personally, I find it a really neat way to bring message queues into a serverless setup without overcomplicating things.