mirror of
https://gitlab.com/allianceauth/allianceauth.git
synced 2025-07-09 04:20:17 +02:00
Extend developer docs
This commit is contained in:
parent
f28a50f92c
commit
170b246901
3
.gitignore
vendored
3
.gitignore
vendored
@ -72,3 +72,6 @@ celerybeat-schedule
|
||||
|
||||
#transifex
|
||||
.tx/
|
||||
|
||||
#other
|
||||
.flake8
|
||||
|
BIN
docs/_static/images/development/aa_core.png
vendored
Normal file
BIN
docs/_static/images/development/aa_core.png
vendored
Normal file
Binary file not shown.
After Width: | Height: | Size: 193 KiB |
12
docs/conf.py
12
docs/conf.py
@ -18,7 +18,10 @@
|
||||
#
|
||||
import os
|
||||
import sys
|
||||
sys.path.insert(0, os.path.abspath('.'))
|
||||
import django
|
||||
sys.path.insert(0, os.path.abspath('..'))
|
||||
os.environ['DJANGO_SETTINGS_MODULE'] = 'tests.settings_all'
|
||||
django.setup()
|
||||
|
||||
# on_rtd is whether we are on readthedocs.org, this line of code grabbed from docs.readthedocs.org
|
||||
on_rtd = os.environ.get('READTHEDOCS', None) == 'True'
|
||||
@ -38,7 +41,9 @@ from recommonmark.transform import AutoStructify
|
||||
# Add any Sphinx extension module names here, as strings. They can be
|
||||
# extensions coming with Sphinx (named 'sphinx.ext.*') or your custom
|
||||
# ones.
|
||||
extensions = []
|
||||
extensions = [
|
||||
'sphinx.ext.autodoc',
|
||||
]
|
||||
|
||||
# Add any paths that contain templates here, relative to this directory.
|
||||
templates_path = ['_templates']
|
||||
@ -148,6 +153,9 @@ man_pages = [
|
||||
[author], 1)
|
||||
]
|
||||
|
||||
# -- Options for autodoc -------------------------------------------------
|
||||
|
||||
add_module_names = False
|
||||
|
||||
# -- Options for Texinfo output -------------------------------------------
|
||||
|
||||
|
463
docs/development/dev_setup/aa-dev-setup-wsl-vsc-v2.md
Normal file
463
docs/development/dev_setup/aa-dev-setup-wsl-vsc-v2.md
Normal file
@ -0,0 +1,463 @@
|
||||
# Development on Windows 10 with WSL and Visual Studio Code
|
||||
|
||||
This document describes step-by-step how to setup a complete development environment for Alliance Auth apps on Windows 10 with Windows Subsystem for Linux (WSL) and Visual Studio Code.
|
||||
|
||||
The main benefit of this setup is that it runs all services and code in the native Linux environment (WSL) and at the same time can be full controlled from within a comfortable Windows IDE (Visual Studio Code) including code debugging.
|
||||
|
||||
In addition all tools described in this guide are open source or free software.
|
||||
|
||||
```eval_rst
|
||||
.. hint::
|
||||
This guide is meant for development purposes only and not for installing AA in a production environment. For production installation please see chapter **Installation**.
|
||||
```
|
||||
|
||||
## Overview
|
||||
|
||||
The development environment consists of the following components:
|
||||
|
||||
- Visual Studio Code with Remote WSL and Python extension
|
||||
- WSL with Ubunutu 18.04. LTS
|
||||
- Python 3.6 environment on WSL
|
||||
- MySQL server on WSL
|
||||
- Redis on WSL
|
||||
- Alliance Auth on WSL
|
||||
- Celery on WSL
|
||||
|
||||
We will use the build-in Django development webserver, so we don't need to setup a WSGI server or a web server.
|
||||
|
||||
## Requirement
|
||||
|
||||
The only requirement is a PC with Windows 10 and Internet connection in order to download the additional software components.
|
||||
|
||||
## Windows installation
|
||||
|
||||
### Windows Subsystem for Linux
|
||||
|
||||
- Install from here: [Microsoft docs](https://docs.microsoft.com/en-us/windows/wsl/install-win10)
|
||||
|
||||
- Choose Ubuntu 18.04. LTS
|
||||
|
||||
### Visual Studio Code
|
||||
|
||||
- Install from here: [VSC Download](https://code.visualstudio.com/Download)
|
||||
|
||||
- Open the app and install the following VSC extensions:
|
||||
|
||||
- Remote WSL
|
||||
|
||||
- Connect to WSL. This will automatically install the VSC server on the VSC server for WSL
|
||||
|
||||
- Once connected to WSL install the Python extension on the WSL side
|
||||
|
||||
## WSL Installation
|
||||
|
||||
Open a WSL bash and update all software packets:
|
||||
|
||||
```bash
|
||||
sudo apt update && sudo apt upgrade -y
|
||||
```
|
||||
|
||||
### Install Tools
|
||||
|
||||
```bash
|
||||
sudo apt-get install build-essential
|
||||
sudo apt-get install gettext
|
||||
```
|
||||
|
||||
### Install Python
|
||||
|
||||
For AA we want to develop with Python 3.6, because that provides the maximum compatibility with today's AA installations. This also happens to be the default Python 3 version for Ubuntu 18.04. at the point of this writing.
|
||||
|
||||
```eval_rst
|
||||
.. hint::
|
||||
To check your system's Python 3 version you can enter: ``python3 --version``
|
||||
```
|
||||
|
||||
```eval_rst
|
||||
.. note::
|
||||
Should your Ubuntu come with a newer version of Python we recommend to still setup your dev environment with the oldest Python 3 version supported by AA, e.g Python 3.6
|
||||
You an check out this `page <https://askubuntu.com/questions/682869/how-do-i-install-a-different-python-version-using-apt-get/1195153>`_ on how to install additional Python versions on Ubuntu.
|
||||
```
|
||||
|
||||
Use the following command to install Python 3 with all required libraries with the default version:
|
||||
|
||||
```bash
|
||||
sudo apt-get install python3 python3-dev python3-venv python3-setuptools python3-pip python-pip
|
||||
```
|
||||
|
||||
### Installing the DBMS
|
||||
|
||||
Install MySQL and required libraries with the following command:
|
||||
|
||||
```bash
|
||||
sudo apt-get install mysql-server mysql-client libmysqlclient-dev
|
||||
```
|
||||
|
||||
```eval_rst
|
||||
.. note::
|
||||
We chose to use MySQL instead of MariaDB, because the standard version of MariaDB that comes with this Ubuntu distribution will not work with AA.
|
||||
```
|
||||
|
||||
We need to apply a permission fix to mysql or you will get a warning with every startup:
|
||||
|
||||
```bash
|
||||
sudo usermod -d /var/lib/mysql/ mysql
|
||||
```
|
||||
|
||||
Start the mysql server
|
||||
|
||||
```bash
|
||||
sudo service mysql start
|
||||
```
|
||||
|
||||
Create database and user for AA
|
||||
|
||||
```bash
|
||||
sudo mysql -u root
|
||||
```
|
||||
|
||||
```sql
|
||||
CREATE USER 'aa_dev'@'localhost' IDENTIFIED BY 'PASSWORD';
|
||||
CREATE DATABASE aa_dev CHARACTER SET utf8mb4;
|
||||
GRANT ALL PRIVILEGES ON aa_dev . * TO 'aa_dev'@'localhost';
|
||||
CREATE DATABASE test_aa_dev CHARACTER SET utf8mb4;
|
||||
GRANT ALL PRIVILEGES ON test_aa_dev . * TO 'aa_dev'@'localhost';
|
||||
exit;
|
||||
```
|
||||
|
||||
Add timezone info to mysql
|
||||
|
||||
```bash
|
||||
sudo mysql_tzinfo_to_sql /usr/share/zoneinfo | sudo mysql -u root mysql
|
||||
```
|
||||
|
||||
### Install redis and other tools
|
||||
|
||||
```bash
|
||||
sudo apt-get install unzip git redis-server curl libssl-dev libbz2-dev libffi-dev
|
||||
```
|
||||
|
||||
Start redis
|
||||
|
||||
```bash
|
||||
sudo redis-server --daemonize yes
|
||||
```
|
||||
|
||||
```eval_rst
|
||||
.. note::
|
||||
WSL does not have an init.d service, so it will not automatically start your services such as MySQL and Redis when you boot your Windows machine. For convenience we recommend putting the commands for starting these services in a bash script. Here is an example: ::
|
||||
|
||||
#/bin/bash
|
||||
# start services for AA dev
|
||||
sudo service mysql start
|
||||
sudo redis-server --daemonize yes
|
||||
|
||||
In addition it is possible to configure Windows to automatically start WSL services, but that procedure goes beyond the scopes of this guide.
|
||||
```
|
||||
|
||||
### Setup dev folder on WSL
|
||||
|
||||
Setup your folders on WSL bash for your dev project. Our approach will setup one AA project with one venv and multiple apps running under the same AA project, but each in their own folder and git.
|
||||
|
||||
A good location for setting up this folder structure is your home folder or a subfolder of your home:
|
||||
|
||||
```text
|
||||
~/aa-dev
|
||||
|- venv
|
||||
|- myauth
|
||||
|- my_app_1
|
||||
|- my_app_2
|
||||
|- ...
|
||||
```
|
||||
|
||||
Following this approach you can also setup additional AA projects, e.g. aa-dev-2, aa-dev-3 if needed.
|
||||
|
||||
Create the root folder aa-dev.
|
||||
|
||||
### setup virtual Python environment for aa-dev
|
||||
|
||||
Create the virtual environment. Run this in your aa-dev folder:
|
||||
|
||||
```bash
|
||||
python3 -m venv venv
|
||||
```
|
||||
|
||||
And activate your venv:
|
||||
|
||||
```bash
|
||||
source venv/bin/activate
|
||||
```
|
||||
|
||||
### install Python packages
|
||||
|
||||
```bash
|
||||
pip install --upgrade pip
|
||||
pip install wheel
|
||||
```
|
||||
|
||||
## Alliance Auth installation
|
||||
|
||||
## Install and create AA instance
|
||||
|
||||
```bash
|
||||
pip install allianceauth
|
||||
```
|
||||
|
||||
Now we are ready to setup our AA instance. Make sure to run this command in your aa-dev folder:
|
||||
|
||||
```bash
|
||||
allianceauth start myauth
|
||||
```
|
||||
|
||||
Next we will setup our VSC project for aa-dev by starting it directly from the WSL bash:
|
||||
|
||||
```bash
|
||||
code .
|
||||
```
|
||||
|
||||
First you want to make sure exclude the venv folder from VSC as follows:
|
||||
Open settings and go to Files:Exclude
|
||||
Add pattern: `**/venv`
|
||||
|
||||
### Update settings
|
||||
|
||||
Open the settings file with VSC. Its under `myauth/myauth/settings/local.py`
|
||||
|
||||
Make sure to have the settings of your Eve Online app ready.
|
||||
|
||||
Turn on DEBUG mode to ensure your static files get served by Django:
|
||||
|
||||
```python
|
||||
DEBUG = True
|
||||
```
|
||||
|
||||
Update name, user and password of your DATABASE configuration.
|
||||
|
||||
```python
|
||||
DATABASES['default'] = {
|
||||
'ENGINE': 'django.db.backends.mysql',
|
||||
'NAME': 'aa_dev',
|
||||
'USER': 'aa_dev',
|
||||
'PASSWORD': 'PASSWORD',
|
||||
'HOST': '127.0.0.1',
|
||||
'PORT': '3306',
|
||||
'OPTIONS': {'charset': 'utf8mb4'},
|
||||
}
|
||||
```
|
||||
|
||||
For the Eve Online related setup you need to create a SSO app on the developer site:
|
||||
|
||||
- Create your Eve Online SSO App on the [Eve Online developer site](https://developers.eveonline.com/)
|
||||
- Add all ESI scopes
|
||||
- Set callback URL to: `http://localhost:8000/sso/callback`
|
||||
|
||||
Then update local.py with your settings:
|
||||
|
||||
```python
|
||||
ESI_SSO_CLIENT_ID = 'YOUR-ID'
|
||||
ESI_SSO_CLIENT_SECRET = 'YOUR_SECRET'
|
||||
ESI_SSO_CALLBACK_URL = 'http://localhost:8000/sso/callback'
|
||||
```
|
||||
|
||||
Disable email registration:
|
||||
|
||||
```python
|
||||
REGISTRATION_VERIFY_EMAIL = False
|
||||
```
|
||||
|
||||
### Migrations and superuser
|
||||
|
||||
Before we can start AA we need to run migrations:
|
||||
|
||||
```bash
|
||||
cd myauth
|
||||
python manage.py migrate
|
||||
```
|
||||
|
||||
We also need to create a superuser for our AA installation:
|
||||
|
||||
```bash
|
||||
python /home/allianceserver/myauth/manage.py createsuperuser
|
||||
```
|
||||
|
||||
## Running Alliance Auth
|
||||
|
||||
## AA instance
|
||||
|
||||
We are now ready to run out AA instance with the following command:
|
||||
|
||||
```bash
|
||||
python manage.py runserver
|
||||
```
|
||||
|
||||
Once running you can access your auth site on the browser under `http://localhost:8000`. Or the admin site under `http://localhost:8000/admin`
|
||||
|
||||
```eval_rst
|
||||
.. hint::
|
||||
You can start your AA server directly from a terminal window in VSC or with a VSC debug config (see chapter about debugging for details).
|
||||
```
|
||||
|
||||
```eval_rst
|
||||
.. note::
|
||||
**Debug vs. Non-Debug mode**
|
||||
Usually it is best to run your dev AA instance in debug mode, so you get all the detailed error messages that helps a lot for finding errors. But there might be cases where you want to test features that do not exist in debug mode (e.g. error pages) or just want to see how your app behaves in non-debug / production mode.
|
||||
|
||||
When you turn off debug mode you will see a problem though: Your pages will not render correctly. The reason is that Django will stop serving your static files in production mode and expect you to serve them from a real web server. Luckily, there is an option that forces Django to continue serving your static files directly even when not in debug mode. Just start your server with the following option: ``python manage.py runserver --insecure``
|
||||
```
|
||||
|
||||
### Celery
|
||||
|
||||
In addition you can start a celery worker instance for myauth. For development purposed it makes sense to only start one instance and add some additional logging.
|
||||
|
||||
This can be done from the command line with the following command in the myauth folder (where manage.py is located):
|
||||
|
||||
```bash
|
||||
celery -E -A myauth worker -l info -P solo
|
||||
```
|
||||
|
||||
Same as AA itself you can start Celery from any terminal session, from a terminal window within VSC or as a debug config in VSC (see chapter about debugging for details). For convenience we recommend starting Celery as debug config.
|
||||
|
||||
## Debugging setup
|
||||
|
||||
To be able to debug your code you need to add debugging configuration to VSC. At least one for AA and one for celery.
|
||||
|
||||
### Breakpoints
|
||||
|
||||
By default VSC will break on any uncaught exception. Since every error raised by your tests will cause an uncaught exception we recommend to deactivate this feature.
|
||||
|
||||
To deactivate open click on the debug icon to switch to the debug view. Then un-check "Uncaught Exceptions" on the breakpoints window.
|
||||
|
||||
### AA debug config
|
||||
|
||||
In VSC click on Debug / Add Configuration and choose "Django". Should Django not appear as option make sure to first open a Django file (e.g. the local.py settings) to help VSC detect that you are using Django.
|
||||
|
||||
The result should look something like this:
|
||||
|
||||
```python
|
||||
{
|
||||
"name": "Python: Django",
|
||||
"type": "python",
|
||||
"request": "launch",
|
||||
"program": "${workspaceFolder}/myauth/manage.py",
|
||||
"args": [
|
||||
"runserver",
|
||||
"--noreload"
|
||||
],
|
||||
"django": true
|
||||
}
|
||||
```
|
||||
|
||||
### Debug celery
|
||||
|
||||
For celery we need another debug config, so that we can run it in parallel to our AA instance.
|
||||
|
||||
Here is an example debug config for Celery:
|
||||
|
||||
```javascript
|
||||
{
|
||||
"name": "Python: Celery",
|
||||
"type": "python",
|
||||
"request": "launch",
|
||||
"module": "celery",
|
||||
"cwd": "${workspaceFolder}/myauth",
|
||||
"console": "integratedTerminal",
|
||||
"args": [
|
||||
"-E",
|
||||
"-A",
|
||||
"myauth",
|
||||
"worker",
|
||||
"-l",
|
||||
"info",
|
||||
"-P",
|
||||
"solo",
|
||||
],
|
||||
"django": true
|
||||
},
|
||||
```
|
||||
|
||||
### Debug config for unit tests
|
||||
|
||||
Finally it makes sense to have a dedicated debug config for running unit tests. Here is an example config for running all tests of the app `example`.
|
||||
|
||||
```javascript
|
||||
{
|
||||
"name": "Python: myauth unit tests",
|
||||
"type": "python",
|
||||
"request": "launch",
|
||||
"program": "${workspaceFolder}/myauth/manage.py",
|
||||
"args": [
|
||||
"test",
|
||||
"-v 2",
|
||||
"--keepdb",
|
||||
"--debug-mode",
|
||||
"--failfast",
|
||||
"example",
|
||||
],
|
||||
|
||||
"django": true
|
||||
},
|
||||
```
|
||||
|
||||
You can also specify to run just a part of your test suite down to a test method. Just give the full path to the test you want to run, e.g. `example.test.test_models.TestDemoModel.test_this_method`
|
||||
|
||||
### Debugging normal python scripts
|
||||
|
||||
Finally you may also want to have a debug config to debug a non-Django Python script:
|
||||
|
||||
```javascript
|
||||
{
|
||||
"name": "Python: Current File",
|
||||
"type": "python",
|
||||
"request": "launch",
|
||||
"program": "${file}",
|
||||
"console": "integratedTerminal"
|
||||
},
|
||||
```
|
||||
|
||||
## Additional tools
|
||||
|
||||
The following additional tools are very helpful when developing for AA.
|
||||
|
||||
### Code Spell Checker
|
||||
|
||||
Typos in your user facing comments can be quite embarrassing. This spell checker helps you avoid them.
|
||||
|
||||
Install from here: [Code Spell Checker](https://marketplace.visualstudio.com/items?itemName=streetsidesoftware.code-spell-checker)
|
||||
|
||||
### DBeaver
|
||||
|
||||
DBeaver is a free universal database tool and works with many different kinds of databases include MySQL. It can be installed on Windows 10 and will be able to help manage your MySQL databases running on WSL.
|
||||
|
||||
Install from here. [DBeaver](https://dbeaver.io/)
|
||||
|
||||
### django-extensions
|
||||
|
||||
[django-extensions](https://django-extensions.readthedocs.io/en/latest/) is a swiss army knife for django developers with adds a lot of very useful features to your Django site. Here are a few highlights:
|
||||
|
||||
- shell_plus - An enhanced version of the Django shell. It will auto-load all your models at startup so you don't have to import anything and can use them right away.
|
||||
- graph_models - Creates a dependency graph of Django models. Visualizing a model dependency structure can be very useful for trying to understand how an existing Django app works, or e.g. how all the AA models work together.
|
||||
- runserver_plus - The standard runserver stuff but with the Werkzeug debugger baked in. This is a must have for any serious debugging.
|
||||
|
||||
## Adding apps for development
|
||||
|
||||
The idea behind the particular folder structure of aa-dev is to have each and every app in its own folder and git repo. To integrate them with the AA instance they need to be installed once using the -e option that enabled editing of the package. And then added to the INSTALLED_APPS settings.
|
||||
|
||||
To demonstrate let's add the example plugin to our environment.
|
||||
|
||||
Open a WSL bash and navigate to the aa-dev folder. Make sure you have activate your virtual environment. (`source venv/bin/activate`)
|
||||
|
||||
Run these commands:
|
||||
|
||||
```bash
|
||||
git clone https://gitlab.com/ErikKalkoken/allianceauth-example-plugin.git
|
||||
pip install -e allianceauth-example-plugin
|
||||
```
|
||||
|
||||
Add `'example'` to INSTALLED_APPS in your `local.py` settings.
|
||||
|
||||
Run migrations and restart your AA server, e.g.:
|
||||
|
||||
```bash
|
||||
cd myauth
|
||||
python manage.py migrate
|
||||
```
|
10
docs/development/dev_setup/index.md
Normal file
10
docs/development/dev_setup/index.md
Normal file
@ -0,0 +1,10 @@
|
||||
# Setup dev environment for AA
|
||||
|
||||
Here you find guides on how to setup your development environment for AA.
|
||||
|
||||
```eval_rst
|
||||
.. toctree::
|
||||
:maxdepth: 1
|
||||
|
||||
aa-dev-setup-wsl-vsc-v2
|
||||
```
|
@ -8,4 +8,6 @@
|
||||
|
||||
custom/index
|
||||
aa_core/index
|
||||
dev_setup/index
|
||||
tech_docu/index
|
||||
```
|
||||
|
39
docs/development/tech_docu/api/esi.rst
Normal file
39
docs/development/tech_docu/api/esi.rst
Normal file
@ -0,0 +1,39 @@
|
||||
======================
|
||||
django-esi
|
||||
======================
|
||||
|
||||
The django-esi package provides an interface for easy access to the ESI.
|
||||
|
||||
Location: ``esi``
|
||||
|
||||
This is an external package. Please also see `here <https://gitlab.com/allianceauth/django-esi>`_ for it's official documentation.
|
||||
|
||||
clients
|
||||
===========
|
||||
|
||||
.. automodule:: esi.clients
|
||||
:members: esi_client_factory
|
||||
:undoc-members:
|
||||
|
||||
decorators
|
||||
===========
|
||||
|
||||
.. automodule:: esi.decorators
|
||||
:members:
|
||||
:undoc-members:
|
||||
|
||||
errors
|
||||
===========
|
||||
|
||||
.. automodule:: esi.errors
|
||||
:members:
|
||||
:undoc-members:
|
||||
|
||||
|
||||
models
|
||||
===========
|
||||
|
||||
.. automodule:: esi.models
|
||||
:members: Scope, Token
|
||||
:exclude-members: objects, provider
|
||||
:undoc-members:
|
30
docs/development/tech_docu/api/evelinks.rst
Normal file
30
docs/development/tech_docu/api/evelinks.rst
Normal file
@ -0,0 +1,30 @@
|
||||
===============================
|
||||
evelinks
|
||||
===============================
|
||||
|
||||
This package generates profile URLs for eve entities on 3rd party websites like evewho and zKillboard.
|
||||
|
||||
Location: ``allianceauth.eveonline.evelinks``
|
||||
|
||||
dotlan
|
||||
===============
|
||||
|
||||
.. automodule:: allianceauth.eveonline.evelinks.dotlan
|
||||
:members:
|
||||
:undoc-members:
|
||||
|
||||
eveho
|
||||
==============
|
||||
|
||||
.. automodule:: allianceauth.eveonline.evelinks.evewho
|
||||
:members:
|
||||
:undoc-members:
|
||||
|
||||
|
||||
zkillboard
|
||||
===================
|
||||
|
||||
.. automodule:: allianceauth.eveonline.evelinks.zkillboard
|
||||
:members:
|
||||
:undoc-members:
|
||||
|
15
docs/development/tech_docu/api/eveonline.rst
Normal file
15
docs/development/tech_docu/api/eveonline.rst
Normal file
@ -0,0 +1,15 @@
|
||||
======================
|
||||
eveonline
|
||||
======================
|
||||
|
||||
The eveonline package provides models for commonly used Eve Online entities like characters, corporations and alliances. All models have the ability to be loaded from ESI.
|
||||
|
||||
Location: ``allianceauth.eveonline``
|
||||
|
||||
models
|
||||
======
|
||||
|
||||
.. automodule:: allianceauth.eveonline.models
|
||||
:members:
|
||||
:exclude-members: objects, provider
|
||||
:undoc-members:
|
13
docs/development/tech_docu/api/index.md
Normal file
13
docs/development/tech_docu/api/index.md
Normal file
@ -0,0 +1,13 @@
|
||||
# API
|
||||
|
||||
To reduce redundancy and help speed up development we encourage developers to utilize the following packages when developing apps for Alliance Auth.
|
||||
|
||||
```eval_rst
|
||||
.. toctree::
|
||||
:maxdepth: 1
|
||||
|
||||
esi
|
||||
evelinks
|
||||
eveonline
|
||||
testutils
|
||||
```
|
14
docs/development/tech_docu/api/testutils.rst
Normal file
14
docs/development/tech_docu/api/testutils.rst
Normal file
@ -0,0 +1,14 @@
|
||||
=============================
|
||||
tests
|
||||
=============================
|
||||
|
||||
Here you find utility functions and classes, which can help speed up writing test cases for AA.
|
||||
|
||||
Location: ``allianceauth.tests.auth_utils``
|
||||
|
||||
auth_utils
|
||||
===========
|
||||
|
||||
.. automodule:: allianceauth.tests.auth_utils
|
||||
:members:
|
||||
:undoc-members:
|
204
docs/development/tech_docu/celery.md
Normal file
204
docs/development/tech_docu/celery.md
Normal file
@ -0,0 +1,204 @@
|
||||
# Celery FAQ
|
||||
|
||||
**Alliance Auth** uses Celery for asynchronous task management. This page aims to give developers some guidance on how to use Celery when developing apps for Alliance Auth.
|
||||
|
||||
For a complete documentation of Celery please refer to the [official Celery documentation](http://docs.celeryproject.org/en/latest/index.html).
|
||||
|
||||
## When should I use Celery in my app?
|
||||
|
||||
There are two main cases for using celery. Long duration of a process and recurrence of a process.
|
||||
|
||||
### Duration
|
||||
|
||||
Alliance Auth is an online web application and as such the user expects fast and immediate responses to any of his clicks or actions. Same as with any other good web site. Good response times are measures in ms and a user will perceive everything that takes longer than 1 sec as an interruption of his flow of thought (see also [Response Times: The 3 Important Limits](https://www.nngroup.com/articles/response-times-3-important-limits/)).
|
||||
|
||||
As a rule of thumb we therefore recommend to use celery tasks for every process that can take longer than 1 sec to complete (also think about how long your process might take with large amounts of data).
|
||||
|
||||
```eval_rst
|
||||
.. note::
|
||||
Another solution for dealing with long response time in particular when loading pages is to load parts of a page asynchronously, for example with AJAX.
|
||||
```
|
||||
|
||||
### Recurrence
|
||||
|
||||
Another case for using celery tasks is when you need recurring execution of tasks. For example you may want to update the list of characters in a corporation from ESI every hour.
|
||||
|
||||
These are called periodic tasks and Alliance Auth uses [celery beat](https://docs.celeryproject.org/en/latest/userguide/periodic-tasks.html) to implement them.
|
||||
|
||||
## What is a celery task?
|
||||
|
||||
For the most part a celery task is a Python functions that is configured to be executed asynchronously and controlled by Celery. Celery tasks can be automatically retried, executed periodically, executed in work flows and much more. See the [celery docs](https://docs.celeryproject.org/en/latest/userguide/tasks.html) for a more detailed description.
|
||||
|
||||
## How should I use Celery in my app?
|
||||
|
||||
Please use the following approach to ensure your tasks are working properly with Alliance Auth:
|
||||
|
||||
- All tasks should be defined in a module of your app's package called `tasks.py`
|
||||
- Every task is a Python function with has the `@shared_task` decorator.
|
||||
- Task functions and the tasks module should be kept slim, just like views by mostly utilizing business logic defined in your models/managers.
|
||||
- Tasks should always have logging, so their function and potential errors can be monitored properly
|
||||
|
||||
Here is an example implementation of a task:
|
||||
|
||||
```Python
|
||||
import logging
|
||||
from celery import shared_task
|
||||
|
||||
logger = logging.getLogger(__name__)
|
||||
|
||||
|
||||
@shared_task
|
||||
def example():
|
||||
logger.info('example task started')
|
||||
```
|
||||
|
||||
This task can then be started from any another Python module like so:
|
||||
|
||||
```Python
|
||||
from .tasks import example
|
||||
|
||||
example.delay()
|
||||
```
|
||||
|
||||
## How should I use celery tasks in the UI?
|
||||
|
||||
There is a well established pattern for integrating asynchronous processes in the UI, for example when the user asks your app to perform a longer running action:
|
||||
|
||||
1. Notify the user immediately (with a Django message) that the process for completing the action has been started and that he will receive a report once completed.
|
||||
|
||||
2. Start the celery task
|
||||
|
||||
3. Once the celery task is completed it should send a notification containing the result of the action to the user. It's important to send that notification also in case of errors.
|
||||
|
||||
## Can I use long running tasks?
|
||||
|
||||
Long running tasks are possible, but in general Celery works best with short running tasks. Therefore we strongly recommend to try and break down long running tasks into smaller tasks if possible.
|
||||
|
||||
If contextually possible try to break down your long running task in shorter tasks that can run in parallel.
|
||||
|
||||
However, many long running tasks consist of several smaller processes that need to run one after the other. For example you may have a loop where you perform the same action on hundreds of objects. In those cases you can define each of the smaller processes as it's own task and then link them together, so that they are run one after the other. That is called chaining in Celery and is the preferred approach for implementing long running processes.
|
||||
|
||||
Example implementation for a celery chain:
|
||||
|
||||
```Python
|
||||
import logging
|
||||
from celery import shared_task, chain
|
||||
|
||||
logger = logging.getLogger(__name__)
|
||||
|
||||
|
||||
@shared_task
|
||||
def example():
|
||||
logger.info('example task')
|
||||
|
||||
@shared_task
|
||||
def long_runner():
|
||||
logger.info('started long runner')
|
||||
my_tasks = list()
|
||||
for _ in range(10):
|
||||
task_signature = example.si()
|
||||
my_task.append(task_signature)
|
||||
|
||||
chain(my_tasks).delay()
|
||||
```
|
||||
|
||||
In this example we fist add 10 example tasks that need to run one after the other to a list. This can be done by creating a so called signature for a task. Those signature are a kind of wrapper for tasks and can be used in various ways to compose work flow for tasks.
|
||||
|
||||
The list of task signatures is then converted to a chain and started asynchronously.
|
||||
|
||||
```eval_rst
|
||||
.. hint::
|
||||
In our example we use ``si()``, which is a shortcut for "immutable signatures" and prevents us from having to deal with result sharing between tasks.
|
||||
|
||||
For more information on signature and work flows see the official documentation on `Canvas <https://docs.celeryproject.org/en/latest/userguide/canvas.html>`_.
|
||||
|
||||
In this context please note that Alliance Auth currently only supports chaining, because all other variants require a so called results back, which Alliance Auth does not have.
|
||||
```
|
||||
|
||||
## How can I define periodic tasks for my app?
|
||||
|
||||
Periodic tasks are normal celery tasks which are added the scheduler for periodic execution. The convention for defining periodic tasks for an app is to define them in the local settings. So user will need to add those settings manually to his local settings during the installation process.
|
||||
|
||||
Example setting:
|
||||
|
||||
```Python
|
||||
CELERYBEAT_SCHEDULE['structures_update_all_structures'] = {
|
||||
'task': 'structures.tasks.update_all_structures',
|
||||
'schedule': crontab(minute='*/30'),
|
||||
}
|
||||
```
|
||||
|
||||
- `structures_update_all_structures` is the name of the scheduling entry. You can chose any name, but the convention is name of your app plus name of the task.
|
||||
|
||||
- `'task'`: Name of your task (full path)
|
||||
- `'schedule'`: Schedule definition (see Celery documentation on [Periodic Tasks](https://docs.celeryproject.org/en/latest/userguide/periodic-tasks.html) for details)
|
||||
|
||||
## How can I use priorities for tasks?
|
||||
|
||||
In Alliance Auth we have defined task priorities from 0 - 9 as follows:
|
||||
|
||||
```eval_rst
|
||||
====== ========= ===========
|
||||
Number Priority Description
|
||||
====== ========= ===========
|
||||
0 Reserved Reserved for Auth and may not be used by apps
|
||||
1, 2 Highest Needs to run right now
|
||||
3, 4 High needs to run as soon as practical
|
||||
5 Normal default priority for most tasks
|
||||
6, 7 Low needs to run soonish, but is less urgent than most tasks
|
||||
8, 9 Lowest not urgent, can be run whenever there is time
|
||||
====== ========= ===========
|
||||
```
|
||||
|
||||
```eval_rst
|
||||
.. warning::
|
||||
Please make sure to use task priorities with care and especially do not use higher priorities without a good reason. All apps including Alliance Auth share the same task queues, so using higher task priorities excessively can potentially prevent more important tasks (of other apps) from completing on time.
|
||||
|
||||
You also want to make sure to run use lower priorities if you have a large amount of tasks or long running tasks, which are not super urgent. (e.g. the regular update of all Eve characters from ESI runs with priority 7)
|
||||
```
|
||||
|
||||
```eval_rst
|
||||
.. hint::
|
||||
If no priority is specified all tasks will be started with the default priority, which is 5.
|
||||
```
|
||||
|
||||
To run a task with a different priority you need to specify it when starting it.
|
||||
|
||||
Example for starting a task with priority 3:
|
||||
|
||||
```Python
|
||||
example.apply_async(priority=3)
|
||||
```
|
||||
|
||||
```eval_rst
|
||||
.. hint::
|
||||
For defining a priority to tasks you can not use the convenient shortcut ``delay()``, but instead need to start a task with ``apply_async()``, which also requires you to pass parameters to your task function differently. Please check out the `official docs <https://docs.celeryproject.org/en/stable/reference/celery.app.task.html#celery.app.task.Task.apply_async>`_ for details.
|
||||
```
|
||||
|
||||
## What special features should I be aware of?
|
||||
|
||||
Every Alliance Auth installation will come with a couple of special celery related features "out-of-the-box" that you can make use of in your apps.
|
||||
|
||||
### celery-once
|
||||
|
||||
Celery-once is a celery extension "that allows you to prevent multiple execution and queuing of celery tasks". What that means is that you can ensure that only one instance of a celery task runs at any given time. This can be useful for example if you do not want multiple instances of you task to talk to the same external service at the same time.
|
||||
|
||||
We use a custom backend for celery_once in Alliance Auth defined [here](https://gitlab.com/allianceauth/allianceauth/-/blob/master/allianceauth/services/tasks.py#L14)
|
||||
You can import it for use like so:
|
||||
|
||||
```Python
|
||||
from allianceauth.services.tasks import QueueOnce
|
||||
```
|
||||
|
||||
An example of AllianceAuth's use within the `@sharedtask` decorator, can be seen [here](https://gitlab.com/allianceauth/allianceauth/-/blob/master/allianceauth/services/modules/discord/tasks.py#L62) in the discord module
|
||||
You can use it like so:
|
||||
|
||||
```Python
|
||||
@shared_task(bind=True, name='your_modules.update_task', base=QueueOnce)
|
||||
```
|
||||
|
||||
Please see the [official documentation](hhttps://pypi.org/project/celery_once/) of celery-once for details.
|
||||
|
||||
### task priorities
|
||||
|
||||
Alliance Auth is using task priorities to enable priority based scheduling of task execution. Please see [How can I use priorities for tasks?](#how-can-i-use-priorities-for-tasks) for details.
|
5
docs/development/tech_docu/core_models.md
Normal file
5
docs/development/tech_docu/core_models.md
Normal file
@ -0,0 +1,5 @@
|
||||
# Core models
|
||||
|
||||
The following diagram shows the core models of AA and Django and their relationships:
|
||||
|
||||

|
13
docs/development/tech_docu/index.md
Normal file
13
docs/development/tech_docu/index.md
Normal file
@ -0,0 +1,13 @@
|
||||
# Developing apps
|
||||
|
||||
In this section you find topics useful for app developers.
|
||||
|
||||
```eval_rst
|
||||
.. toctree::
|
||||
:maxdepth: 1
|
||||
|
||||
api/index
|
||||
celery
|
||||
core_models
|
||||
templatetags
|
||||
```
|
16
docs/development/tech_docu/templatetags.rst
Normal file
16
docs/development/tech_docu/templatetags.rst
Normal file
@ -0,0 +1,16 @@
|
||||
=============
|
||||
Template Tags
|
||||
=============
|
||||
|
||||
The following template tags are available to be used by all apps. To use them just load the respeetive template tag in your template like so:
|
||||
|
||||
.. code-block:: html
|
||||
|
||||
{% load evelinks %}
|
||||
|
||||
evelinks
|
||||
========
|
||||
|
||||
.. automodule:: allianceauth.eveonline.templatetags.evelinks
|
||||
:members:
|
||||
:undoc-members:
|
@ -83,7 +83,8 @@ apt-get install unzip git redis-server curl libssl-dev libbz2-dev libffi-dev
|
||||
CentOS:
|
||||
|
||||
```bash
|
||||
yum install gcc gcc-c++ unzip git redis curl bzip2-devel
|
||||
yum install gcc gcc-c++ unzip git redis curl
|
||||
2-devel
|
||||
```
|
||||
|
||||
```eval_rst
|
||||
@ -113,7 +114,7 @@ mysql_tzinfo_to_sql /usr/share/zoneinfo | mysql -u root -p mysql
|
||||
```eval_rst
|
||||
.. note::
|
||||
You may see errors when you add the timezone tables. To make sure that they were correctly added run the following commands and check for the ``time_zone`` tables::
|
||||
|
||||
|
||||
mysql -u root -p
|
||||
use mysql;
|
||||
show tables;
|
||||
|
Loading…
x
Reference in New Issue
Block a user