Celery is an open source asynchronous task queue or job queue which is based on distributed message passing. While it supports scheduling, its focus is on operations in real time. It is entirely written in python. It has a best support for django and very easy to install and begin using.
One of the monitoring tool we are going to see how to install today is flower one of best and recommended realtime mornitoring for celery and it offers a dashboard and authentication from GoogleAuth to GithubAuth for securing access to it's dashboard.
What frames intergrate with celery?
- Pyramid - pyramid_celery
- Pylons - celery-pylons
- Flask - (not neede)
- web2py - web2py-celery
- Tornado - tornado-celery
- Tryton - celery_tryton
Celery best parts
- Simple. Celery is easy to use and maintain, and it doesn’t need configuration file. It has an active, friendly community you can talk to for support. Here’s one of the simplest applications you can make:
from celery import Celery app = Celery('hello', broker='amqp://[email protected]//') @app.task def hello(): return 'hello world'
Highly Available: Workers and clients will automatically retry in the event of connection loss or failure, and some brokers support HA in way of Primary/Primary or Primary/Replica replication
- Fast. A single Celery process can process millions of tasks a minute, with sub-millisecond round-trip latency (using RabbitMQ, librabbitmq, and optimized settings).
- Flexible. Almost every part of Celery can be extended or used on its own, Custom pool implementations, serializers, compression schemes, logging, schedulers, consumers, producers, broker transports, and much more.
What brokers are supported by celery?
- Amazon SQS
- and more brokers.
What are celery's result stores?
- SQLAlchemy, Django ORM
- Apache Cassandra, Elasticsearch
Install Celery using python pip
$ pip3 install -U Celery
We just mentioned about brokers and results brokers. Celery also defines a group of bundles that can be used to install Celery and the dependencies for a given feature. You can specify these in your requirements or on the pip command-line by using brackets. Multiple bundles can be specified by separating them by commas.
$ pip3 install "celery[librabbitmq]" $ pip3 install "celery[librabbitmq,redis,auth,msgpack]"
Installing and configuring Flower Celery monitoring tool
Now that we have installed celery very nice and it's running smoothly, how can we configure it's web based mornitoring tool?
$ pip install flower
$ flower -A proj --port=5555
Or if you want to launch it from celery itself
$ celery flower -A proj --address=127.0.0.1 --port=5555
Configure reverse proxy Nginx to access and secure it