mirror of
https://gitlab.com/allianceauth/allianceauth.git
synced 2025-07-09 12:30:15 +02:00
Add docs for smoothing out task execution
This commit is contained in:
parent
c6890dd2c6
commit
292fb7b29d
@ -168,6 +168,49 @@ example.apply_async(priority=3)
|
|||||||
For defining a priority to tasks, you cannot use the convenient shortcut ``delay()``, but instead need to start a task with ``apply_async()``, which also requires you to pass parameters to your task function differently. Please check out the `official docs <https://docs.celeryproject.org/en/stable/reference/celery.app.task.html#celery.app.task.Task.apply_async>`_ for details.
|
For defining a priority to tasks, you cannot use the convenient shortcut ``delay()``, but instead need to start a task with ``apply_async()``, which also requires you to pass parameters to your task function differently. Please check out the `official docs <https://docs.celeryproject.org/en/stable/reference/celery.app.task.html#celery.app.task.Task.apply_async>`_ for details.
|
||||||
:::
|
:::
|
||||||
|
|
||||||
|
## Rate-Limiting and Smoothing of Task Execution
|
||||||
|
|
||||||
|
Large numbers of installs running the same crontab (ie. `0 * * * *`) can all slam an external service at the same time.
|
||||||
|
|
||||||
|
Consider Artificially smoothing out your tasks with a few methods
|
||||||
|
|
||||||
|
### Offset Crontabs
|
||||||
|
|
||||||
|
Avoid running your tasks on the hour or other nice neat human numbers, consider 23 minutes on the hour instead of at zero (`28 * * * *`)
|
||||||
|
|
||||||
|
### Subset Tasks
|
||||||
|
|
||||||
|
Slice your tasks needed up into more manageable chunks and run them more often. 1/10th of your tasks run 10x more often will return the same end result with less peak loads on external services and your task queue.
|
||||||
|
|
||||||
|
### Celery ETA/Countdown
|
||||||
|
|
||||||
|
Scatter your tasks across a larger window using <https://docs.celeryq.dev/en/latest/userguide/calling.html#eta-and-countdown>
|
||||||
|
|
||||||
|
This example will queue up tasks across the next 10 minutes, trickling them into your workers (and the external service)
|
||||||
|
|
||||||
|
```python
|
||||||
|
for corp in EveCorporationInfo.objects.all().values('corporation_id'):
|
||||||
|
update_corp.apply_async(args=[corp['corporation_id']], priority=TASK_PRIORITY)
|
||||||
|
update_corp.apply_async(
|
||||||
|
args=[corp['corporation_id']],
|
||||||
|
priority=TASK_PRIORITY,
|
||||||
|
countdown=randint(1, 600))
|
||||||
|
```
|
||||||
|
|
||||||
|
### Celery Rate Limits
|
||||||
|
|
||||||
|
Celery Rate Limits come with a small catch, its _per worker_, you may have to be either very conservative or have these configurable by the end user if they varied their worker count.
|
||||||
|
|
||||||
|
<https://docs.celeryq.dev/en/latest/userguide/tasks.html#Task.rate_limit>
|
||||||
|
|
||||||
|
This example of 10 Tasks per Minute will result in ~100 tasks per minute at 10 Workers
|
||||||
|
|
||||||
|
```python
|
||||||
|
@shared_task(rate_limit="10/m")
|
||||||
|
def update_charactercorporationhistory(character_id: int) -> None:
|
||||||
|
"""Update CharacterCorporationHistory models from ESI"""
|
||||||
|
```
|
||||||
|
|
||||||
## What special features should I be aware of?
|
## What special features should I be aware of?
|
||||||
|
|
||||||
Every Alliance Auth installation will come with a couple of special celery related features "out-of-the-box" that you can make use of in your apps.
|
Every Alliance Auth installation will come with a couple of special celery related features "out-of-the-box" that you can make use of in your apps.
|
||||||
@ -192,6 +235,6 @@ You can use it like so:
|
|||||||
|
|
||||||
Please see the [official documentation](https://pypi.org/project/celery_once/) of celery-once for details.
|
Please see the [official documentation](https://pypi.org/project/celery_once/) of celery-once for details.
|
||||||
|
|
||||||
### task priorities
|
### Task Priorities
|
||||||
|
|
||||||
Alliance Auth is using task priorities to enable priority-based scheduling of task execution. Please see [How can I use priorities for tasks?](#how-can-i-use-priorities-for-tasks) for details.
|
Alliance Auth is using task priorities to enable priority-based scheduling of task execution. Please see [How can I use priorities for tasks?](#how-can-i-use-priorities-for-tasks) for details.
|
||||||
|
Loading…
x
Reference in New Issue
Block a user