Celery is a great task queue library for Python.
There are some important settings for celery users on, especially for users on shared instances with limited conenctions and number of messages per month.
BROKER_URL = os.environ.get('CLOUDAMQP_URL') BROKER_POOL_LIMIT = 1 # Will decrease connection usage BROKER_HEARTBEAT = None # We're using TCP keep-alive instead BROKER_CONNECTION_TIMEOUT = 30 # May require a long timeout due to Linux DNS timeouts etc CELERY_RESULT_BACKEND = None # AMQP is not recommended as result backend as it creates thousands of queues CELERY_SEND_EVENTS = False # Will not create celeryev.* queues CELERY_EVENT_QUEUE_EXPIRES = 60 # Will delete all celeryev. queues without consumers after 1 minute.
--without-gossip --without-mingle --without-heartbeat
This will decrease the message rates substantially. Without these flags Celery will send hundreds of messages per second with different diagnostic and redundant heartbeat messages. Read more about --without-gossip , --without-mingle and --without-heartbeat in the Celery Worker documentation.
Set the BROKER_POOL_LIMIT to 1 and make sure that your webserver spawns less processes/workers than your current CloudAMQP plan allows for concurrent connections.
Each process creates its own connection pool, so the formula for calculating your concurrent connection need is:
BROKER_POOL_LIMIT * (web dynos * web workers + worker dynos * worker concurrency)
So make sure that you limit the number of
gunicorn web workers
flag and worker concurrency with
We recommend to disable heartbeats . We've enabled low TCP keep-alive intervals on all our RabbitMQ servers so that stale connections will be detected on the TCP level instead of in the application layer.
BROKER_HEARTBEAT = None
We recommend to set BROKER_CONNECTION_TIMEOUT to 30. BROKER_CONNECTION_TIMEOUT is the default timeout in seconds before it give up establishing a connection to the CloudAMQP server.
BROKER_CONNECTION_TIMEOUT = 30
If you don't consume task results, set
CELERY_RESULT_BACKEND = None.
If you do want the task results and you want to use RabbitMQ to store them, then use
CELERY_RESULT_BACKEND = 'rpc'.
NOTE: We highly advise against using the deprecated
CELERY_RESULT_BACKEND = 'amqp'
If no celery event monitoring tools are used, we do recommend to set
CELERY_SEND_EVENTS = False
for more information.
If you don't want to disable events you can instead expire temporary event queues faster.
In that case we recommend to set
CELERY_EVENT_QUEUE_EXPIRES = 60
set to 60 seconds will delete all celeryev. queues after 1 minute - if they do not have any consumers.
When the setting CELERY_ALWAYS_EAGER is set to True, will all taskes be executed locally by blocking until the task returns. That is, tasks will be executed locally instead of being sent to the queue. This forces all calls to .delay()/.apply_async() that would normally get delegated to the worker to instead execute synchronously. You can use this setting when migrating between dedicated plans:
heroku addons:add cloudamqp:bunny
Flower is a web based tool for real-time monitoring and administrating Celery clusters (it is still under development). From Flower it is possible to overview task progress and history, show task details and graphs and statistics about the tasks. You can overview scheduled tasks, revoke or terminate tasks and much more.
Flower uses the RabbitMQ Managment Plugin to get info about queues. To get the broker tab working with CloudAMQP you need to set the --broker_api flag to the URL of the RabbitMQ HTTP API including user credentials, see example below. Broker API port should be 443.