“Celery is an asynchronous task queue based on distributed message passing.”
Let’s break that down:
asynchronous => events happen outside of the main program flow.
task queue based=> queues up the tasks to be performed. (The tasks themselves are queued up in a tool like Redis or RabbitMQ.)
distributed message passing => multiple processes can read and write to the message (task) queue without being connected to one another.
Why do we need Celery?
Say you want to connect to a third-party service, Slack/Email/External Photo Storage, anything — these APIs takes time to respond; as would any web/HTTP request and your Python Web Server would get stuck waiting for the response from the API. The user in the meantime only sees your slow website which is a very bad user experience.
Using Celery you can call the third party API in a separate task (worker), so the main thread is free to return and accept other incoming HTTP requests.
- Whenever we make a call that depends on external factors – for instance calling a third-party API.
- If a task takes too much time – for example I card generation of 2000 students (delegate it to Celery – store the Icard.zip in a MEDIA_ROOT folder – send the user an email that their download is ready.)
Since the whole idea is that the task is done in a separate thread, you need to:
Run a separate “worker” process that handles the functions that uses Celery.
Use the message queue (like Redis or RabbitMQ) where the tasks are queued.
The code itself is rather simple:
Regular function call:
def regular_function(): print("Regular function call") regular_function()
Vs. Same fuction call with Celery:
from celery.decorators import task // Use the 'task' decorator def function_called_asynchronously(): print("Function decorated with Celery task()") # Call function using .delay() function_called_asynchronously.delay()
The function can be called without celery too, simply call function_called_asynchronously() from anywhere.
P.S.: Celery vs Python’s async
There’s another idea around Asynchronous handling that is important,
async (Asynchronous processing) helps in handling multiple HTTP connections on a single server instance. That’s a topic on its own I’ll cover some other time.
Python has support for
async in version 3.5. Django is on it’s way to add async support by Dec 2019.