Why and When to use Celery with Python Web Servers

“Celery is an asynchronous task queue based on distributed message passing.”

Let’s break that down:

asynchronous => events happen outside of the main program flow.

task queue based=> queues up the tasks to be performed. (The tasks themselves are queued up in a tool like Redis or RabbitMQ.)

distributed message passing => multiple processes can read and write to the message (task) queue without being connected to one another.

Why do we need Celery?

Say you want to connect to a third-party service, Slack/Email/External Photo Storage, anything — these APIs takes time to respond; as would any web/HTTP request and your Python Web Server would get stuck waiting for the response from the API. The user in the meantime only sees your slow website which is a very bad user experience.

Using Celery you can call the third party API in a separate task (worker), so the main thread is free to return and accept other incoming HTTP requests.

When?

  1. Whenever we make a call that depends on external factors – for instance calling a third-party API.
  2. If a task takes too much time – for example I card generation of 2000 students (delegate it to Celery – store the Icard.zip in a MEDIA_ROOT folder – send the user an email that their download is ready.)

How?

Since the whole idea is that the task is done in a separate thread, you need to:

  1. Run a separate “worker” process that handles the functions that uses Celery.

  2. Use the message queue (like Redis or RabbitMQ) where the tasks are queued.

The code itself is rather simple:

Regular function call:

def regular_function():
  print("Regular function call")
  
regular_function()

Vs. Same fuction call with Celery:

from celery.decorators import task

// Use the 'task' decorator
@task
def function_called_asynchronously():
  print("Function decorated with Celery task()")
  
# Call function using .delay()
function_called_asynchronously.delay()

The function can be called without celery too, simply call function_called_asynchronously() from anywhere.

P.S.: Celery vs Python’s async

There’s another idea around Asynchronous handling that is important, async (Asynchronous processing) helps in handling multiple HTTP connections on a single server instance. That’s a topic on its own I’ll cover some other time.

Python has support for async in version 3.5. Django is on it’s way to add async support by Dec 2019.

 

Leave a Reply

Your email address will not be published. Required fields are marked *