Python,Flask和Celery的并发异步流程

目前我计划在Apache2 WSGI上使用带有Celery的Flask应用程序作为任务队列.

在下面,如果有3个或更多工作者可用,a_long_process(),another_long_process()和yet_another_long_process()会同时执行吗？在进程执行时是否会阻止Flask应用程序？

来自Flask app：

@myapp.route('/foo')
def bar():
    task_1 = a_long_process.delay(x, y)
    task_1_result = task_1.get(timeout=1)
    task_2 = another_long_process.delay(x, y)
    task_2_result = task_2.get(timeout=1)
    task_3 = yet_another_long_process.delay(x, y)
    task_3_result = task_3.get(timeout=1)
    return task_1 + task_2 + task_3

tasks.py：

from celery import Celery
celery = Celery('tasks', broker="amqp://guest@localhost//", backend="amqp://")
@celery.task
def a_long_process(x, y):
    return something
@celery.task
def another_long_process(x, y):
    return something_else
@celery.task
def yet_another_long_process(x, y):
    return a_third_thing

@myapp.route('/foo')
def bar():
    # start tasks
    task_1 = a_long_process.delay(x, y)
    task_2 = another_long_process.delay(x, y)
    task_3 = yet_another_long_process.delay(x, y)
    # fetch results
    try:
        task_1_result = task_1.get(timeout=1)
        task_2_result = task_2.get(timeout=1)
        task_3_result = task_3.get(timeout=1)
    except TimeoutError:
        # Handle this or don't specify a timeout.
        raise
    # combine results
    return task_1 + task_2 + task_3

此代码将阻止,直到所有结果都可用(或达到超时).

Will the Flask app be blocked while the processes are executing?

此代码仅阻止WSGI容器的一个worker.整个站点没有响应,取决于您使用的WSGI容器. (例如Apache mod_wsgi,uWSGI,gunicorn等)大多数WSGI容器会产生多个worker,因此在代码等待任务结果时只会阻塞一个worker.

对于这种应用程序,我建议使用gevent,它为每个请求生成一个单独的greenlet,并且非常轻量级.