Microservices and message queues, Part 4: Introducing the demo project

The previous article reinforced how RabbitMQ can allow services to communicate asynchronously. This post will stretch our knowledge of the application of RabbitMQ in a microservice architecture beyond just “knowing”. To do that, it will walk us through building a demo project that uses RabbitMQ in a microservice architecture. But first, let’s introduce the demo project.

It is common for job hunters to regularly visit different job portals to find relevant jobs to apply for. As you’d imagine, doing this every day could be very time-consuming, but what if this could be automated?

Yes, we will be building a solution that solves this problem. We will create a web application that accepts a target keyword (e.g., Python Engineer), a location (e.g., Nigeria or Remote), and an email address from a user. The software will then grab relevant job postings from the web matching the target keyword and email them to the user.

Let’s call our piece of software GetMeHired. Essentially, GetMeHired is supposed to eliminate the need to visit different job portals and search for relevant jobs manually. This will make it painless for job hunters to discover new and relevant job postings.

Before we proceed, keep in mind that “supposed” has been deliberately emphasized in the previous paragraph to underscore an important point – our implementation of GetMeHired will be so simplified that it is not going to be anywhere near a good solution.

Let’s talk about the implementation details a little bit.

GetMeHired: High-level architecture

GetMeHired will adopt the microservice architecture, where it will be broken down into two smaller services. We will call the first service GetMeHired-producer and the second service GetMeHired-consumer.

We will then have RabbitMQ sit between these two services to allow them to communicate with each other. Let’s explore the function of each service.

GetMeHired-producer

As the name suggests, this service will serve as the producer of messages in our architecture. Essentially, it will expose just one REST API endpoint that accepts a POST request from an end user. Each request will carry the search keyword, location, and email address as its payload. For example, see the snippet below:

{
  “search term” : “Python Engineer”,
  “location” : “Nigeria”,
  “email”: “random@gmail.com”
}

This service will then publish each request it receives from an end user to RabbitMQ. But before publishing a request to RabbitMQ, the service will perform some validations – for example, verify that the submitted email address is valid.

Remember that this service will publish each POST request it receives as a message to RabbitMQ. Think of each message published in this case as an instruction for the consumer to grab job postings matching the keyword in the payload from the web and send the job links to the email in the payload.

The low-level details of how the GetMeHired-consumer would fetch the relevant job postings and email them to the user is totally abstracted from the first service. We will implement the GetMeHired-producer service with FastAPI, a Python web framework.

GetMeHired-consumer

As its name implies, this service will function as the consumer in our architecture. This service will be responsible for all the heavy lifting – it will receive messages from RabbitMQ. For each message, it will search Indeed and Linkedin for relevant jobs matching the parameters in the payload.

This service will use the Linkedin Jobs Search and Indeed Jobs Search APIs for its searches. It will then merge the results of the two searches and email them to the user whose email address was submitted in the payload.

Furthermore, this service will leverage Twilio Sendgrid to send emails to users. We will also implement this service with FastAPI.

Should you decide to extend this project beyond this tutorial, scraping, let's say, the top five job portals directly instead of connecting to an API is a good idea. That way, you can be in control of a lot of things. For example, you get to decide what portals to scrap.

You can also build a standalone user interface with which users can interact . Right now, the only way to interact with our software is through the first service's API endpoint .

Overall, this is what our architecture will look like in picture.

Figure 1 - GetMeHired high-level architecture

The genius of this architecture lies in the following:

  • Even if GetMeHired-consumer accidentally crashes, GetMeHired-producer will keep producing messages and adding them to RabbitMQ and the end users wouldn’t even notice
  • If we want to process more messages quickly we can create more copies of GetMeHired-consumer
  • Lastly, we can have separate teams working on the two services in parallel. They can also decide to adopt whatever technology they think is best for the job and not necessarily Python/FastAPI for both services

Project requirements

To successfully follow the tutorials in the subsequent articles that would help you build this demo project, you should meet the following requirements.

Conclusion

This article gave us an overview of the demo project we will be building – GetMeHired. This project will adopt the microservices architecture, breaking k it down into two smaller services. To begin, in the next article we will build the first service – GetMeHired-producer and have it connect to RabbitMQ.

For any suggestions, questions, or feedback, get in touch with us at contact@cloudamqp.com

CloudAMQP - industry leading RabbitMQ as a service

Start your managed cluster today. CloudAMQP is 100% free to try.

13,000+ users including these smart companies