Celery is a great task queue library for Python.

There are some important settings for celery users on, especially for users on shared instances with limited conenctions and number of messages per month.

Recommended settings

BROKER_URL = os.environ.get('CLOUDAMQP_URL')
BROKER_POOL_LIMIT = 1 # Will decrease connection usage
BROKER_HEARTBEAT = None # We're using TCP keep-alive instead
BROKER_CONNECTION_TIMEOUT = 30 # May require a long timeout due to Linux DNS timeouts etc
CELERY_RESULT_BACKEND = None # AMQP is not recommended as result backend as it creates thousands of queues
CELERY_SEND_EVENTS = False # Will not create celeryev.* queues
CELERY_EVENT_QUEUE_EXPIRES = 60 # Will delete all celeryev. queues without consumers after 1 minute.
Command-line arguments
--without-gossip --without-mingle --without-heartbeat

This will decrease the message rates substantially. Without these flags Celery will send hundreds of messages per second with different diagnostic and redundant heartbeat messages. Read more about --without-gossip , --without-mingle and --without-heartbeat in the Celery Worker documentation.

BROKER_POOL_LIMIT

Set the BROKER_POOL_LIMIT to 1 and make sure that your webserver spawns less processes/workers than your current CloudAMQP plan allows for concurrent connections.

Each process creates its own connection pool, so the formula for calculating your concurrent connection need is:

BROKER_POOL_LIMIT * (web dynos * web workers + worker dynos * worker concurrency)

So make sure that you limit the number of gunicorn web workers with the -w flag and worker concurrency with -c .

BROKER_HEARTBEAT

We recommend to disable heartbeats . We've enabled low TCP keep-alive intervals on all our RabbitMQ servers so that stale connections will be detected on the TCP level instead of in the application layer.

BROKER_HEARTBEAT = None
BROKER_CONNECTION_TIMEOUT

We recommend to set BROKER_CONNECTION_TIMEOUT to 30. BROKER_CONNECTION_TIMEOUT is the default timeout in seconds before it give up establishing a connection to the CloudAMQP server.

BROKER_CONNECTION_TIMEOUT = 30
CELERY_RESULT_BACKEND

If you don't consume task results, set CELERY_RESULT_BACKEND = None.
If you do want the task results and you want to use RabbitMQ to store them, then use CELERY_RESULT_BACKEND = 'rpc'.

NOTE: We highly advise against using the deprecated CELERY_RESULT_BACKEND = 'amqp'

CELERY EVENTS and EVENT QUEUES

If no celery event monitoring tools are used, we do recommend to set CELERY_SEND_EVENTS = False , see CELERY_SEND_EVENTS for more information.

If you don't want to disable events you can instead expire temporary event queues faster. In that case we recommend to set CELERY_EVENT_QUEUE_EXPIRES = 60 . CELERY_EVENT_QUEUE_EXPIRES set to 60 seconds will delete all celeryev. queues after 1 minute - if they do not have any consumers.

Migrate between plans in Celery

When the setting CELERY_ALWAYS_EAGER is set to True, will all taskes be executed locally by blocking until the task returns. That is, tasks will be executed locally instead of being sent to the queue. This forces all calls to .delay()/.apply_async() that would normally get delegated to the worker to instead execute synchronously. You can use this setting when migrating between dedicated plans:

  1. Add CELERY_ALWAYS_EAGER=True to settings.py, deploy
  2. Verified that the worker queue is empty
  3. Remove the CloudAMQP addon from your Heroku app
  4. Add it again with heroku addons:add cloudamqp:bunny
  5. Immediately redeployed without CELERY_ALWAYS_EAGER

Celery Flower

Flower is a web based tool for real-time monitoring and administrating Celery clusters (it is still under development). From Flower it is possible to overview task progress and history, show task details and graphs and statistics about the tasks. You can overview scheduled tasks, revoke or terminate tasks and much more.

celery cluster

Flower uses the RabbitMQ Managment Plugin to get info about queues. To get the broker tab working with CloudAMQP you need to set the --broker_api flag to the URL of the RabbitMQ HTTP API including user credentials, see example below. Broker API port should be 443.

--broker=amqp://username:password@hostname:5671/vhost?ssl=true

--broker_api=https://username:password@hostname:443/api/