mmolitor87 dc8ed2d510 Supervisor celery simplification (#849)
Combines celery confs under one process group for easy restarting. Updates docs to reflect new commands.
2017-09-01 15:43:54 -04:00

2.8 KiB

Supervisor

Supervisor is a client/server system that allows its users to control a number of processes on UNIX-like operating systems.

What that means is supervisor will take care of ensuring the celery workers are running (and mumble authenticator) and start the automatically on reboot. Handy, eh?

Installation

Most OSes have a supervisor package available in their distribution.

Ubuntu:

sudo apt-get install supervisor

CentOS:

sudo yum install supervisor
sudo systemctl enable supervisord.service
sudo systemctl start supervisord.service

Configuration

Auth provides example config files for the celery workers, the periodic task scheduler (celery beat), and the mumble authenticator. All of these are available in thirdparty/Supervisor.

For most users, all you have to do is copy the config files to /etc/supervisor/conf.d then restart the service. Copy auth.conf for the celery workers, and auth-mumble.conf for the mumble authenticator. For all three just use a wildcard:

sudo cp thirdparty/Supervisor/* /etc/supervisor/conf.d

Ubuntu:

sudo service supervisor restart

CentOS:

sudo systemctl restart supervisor.service

Checking Status

To ensure the processes are working, check their status:

sudo supervisorctl status

Processes will be STARTING, RUNNING, or ERROR. If an error has occurred, check their log files:

  • celery workers: log/worker.log
  • celery beat: log/beat.log
  • authenticator: log/authenticator.log

Restarting Processes

To restart the celery group:

sudo supervisorctl restart auth:*

To restart just celerybeat:

sudo supervisorctl restart auth:celerybeat

To restart just celeryd:

sudo supervisorctl restart auth:celeryd

To restart just mumble authenticator:

sudo supervisorctl restart auth-mumble

Customizing Config Files

The only real customization needed is if running in a virtual environment. The python path will have to be changed in order to start in the venv.

Edit the config files and find the line saying command. Replace python with /path/to/venv/bin/python. For Celery replace celery with /path/to/venv/bin/celery. This can be relative to the directory specified in the config file.

Note that for config changes to be loaded, the supervisor service must be restarted.

Troubleshooting

auth-celerybeat fails to start

Most often this is caused by a permissions issue on the allianceauth directory (the error will talk about celerybeat.pid). The easiest fix is to edit its config file and change the user from allianceserver to root.

Workers are using old settings

Every time the codebase is updated or settings file changed, workers will have to be restarted. Easiest way is to restart the supervisor service (see configuration above for commands)