Basraah 1066e6ac98 The Great Services Refactor (#594)
* Hooks registration, discovery and retrieval module

Will discover @hooks.register decorated functions inside
the auth_hooks module in any installed django app.

* Class to register modular service apps

* Register service modules URLs

* Example service module

* Refactor services into modules

Each service type has been split out into its own django app/module. A
hook mechanism is provided to register a subclass of the ServiceHook
class. The modules then overload functions defined in ServiceHook as
required to provide interoperability with alliance auth. Service modules
provide their own urls and views for user registration and account
management and a partial template to display on the services page. Where
possible, new modules should provide their own models for local data
storage.

* Added menu items hooks and template tags

* Added menu item hook for broadcasts

* Added str method to ServicesHook

* Added exception handling to hook iterators

* Refactor mumble migration and table name

Upgrading will require `migrate mumble --fake-initial` to be run first
and then `migrate mumble` to rename the table.

* Refactor teamspeak3 migration and rename table

Upgrading will require `migrate teamspeak3 --fake-initial`

* Added module models and migrations for refactoring AuthServicesInfo

* Migrate AuthServiceInfo fields to service modules models

* Added helper for getting a users main character

* Added new style celery instance

* Changed Discord from AuthServicesInfo to DiscordUser model

* Switch celery tasks to staticmethods

* Changed Discourse from AuthServicesInfo to DiscourseUser model

* Changed IPBoard from AuthServicesInfo to IpboardUser model

* Changed Ips4 from AuthServicesInfo to Ips4User model

Also added disable service task.

This service still needs some love though. Was always missing a
deactivate services hook (before refactoring) for reasons I'm unsure of
so I'm reluctant to add it without knowing why.

* Changed Market from AuthServicesInfo to MarketUser model

* Changed Mumble from AuthServicesInfo to MumbleUser model

Switched user foreign key to one to one relationship.
Removed implicit password change on user exists.
Combined regular and blue user creation.

* Changed Openfire from AuthServicesInfo to OpenfireUser model

* Changed SMF from AuthServicesInfo to SmfUser model

Added disable task

* Changed Phpbb3 from AuthServicesInfo to Phpbb3User model

* Changed XenForo from AuthServicesInfo to XenforoUser model

* Changed Teamspeak3 from AuthServicesInfo to Teamspeak3User model

* Remove obsolete manager functions

* Standardise URL format

This will break some callback URLs
Discord changes from /discord_callback/ to /discord/callback/

* Removed unnecessary imports

* Mirror upstream decorator change

* Setup for unit testing

* Unit tests for discord service

* Added add main character helper

* Added Discourse unit tests

* Added Ipboard unit tests

* Added Ips4 unit tests

* Fix naming of market manager, switch to use class methods

* Remove unused hook functions

* Added market service unit tests

* Added corp ticker to add main character helper

* Added mumble unit tests

* Fix url name and remove namespace

* Fix missing return and add missing URL

* Added openfire unit tests

* Added missing return

* Added phpbb3 unit tests

* Fix SmfManager naming inconsistency and switch to classmethods

* Added smf unit tests

* Remove unused functions, Added missing return

* Added xenforo unit tests

* Added missing return

* Fixed reference to old model

* Fixed error preventing groups from syncing on reset request

* Added teamspeak3 unit tests

* Added nose as test runner and some test settings

* Added package requirements for running tests

* Added unit tests for services signals and tasks

* Remove unused tests file

* Fix teamspeak3 service signals

* Added unit tests for teamspeak3 signals

Changed other unit tests setUp to inert signals

* Fix password gen and hashing python3 compatibility

Fixes #630

Adds unit tests to check the password functions run on both platforms.

* Fix unit test to not rely on checking url params

* Add Travis CI settings file

* Remove default blank values from services models

* Added dynamic user model admin actions for syncing service groups

* Remove unused search fields

* Add hook function for syncing nicknames

* Added discord hook for sync nickname

* Added user admin model menu actions for sync nickname hook

* Remove obsolete code

* Rename celery config app to avoid package name clash

* Added new style celerybeat schedule configuration

periodic_task decorator is depreciated

* Added string representations

* Added admin pages for services user models

* Removed legacy code

* Move link discord button to correct template

* Remove blank default fields from example model

* Disallow empty django setting

* Fix typos

* Added coverage configuration file

* Add coverage and coveralls to travis config

Should probably use nose's built in coverage, but this works for now.

* Replace AuthServicesInfo get_or_create instances with get

Reflects upstream changes to AuthServicesInfo behaviour.

* Update mumble user table name

* Split out mumble authenticator requirements

zeroc-ice seems to cause long build times on travis-ci and isn't
required for the core projects functionality or testing.
2017-01-25 12:50:16 +10:00

382 lines
12 KiB
Python

from __future__ import unicode_literals
import logging
import requests
import random
import string
import datetime
import json
import re
from django.conf import settings
from django.utils import timezone
from services.models import GroupCache
logger = logging.getLogger(__name__)
class DiscourseError(Exception):
def __init__(self, endpoint, errors):
self.endpoint = endpoint
self.errors = errors
def __str__(self):
return "API execution failed.\nErrors: %s\nEndpoint: %s" % (self.errors, self.endpoint)
# not exhaustive, only the ones we need
ENDPOINTS = {
'groups': {
'list': {
'path': "/admin/groups.json",
'method': requests.get,
'args': {
'required': [],
'optional': [],
},
},
'create': {
'path': "/admin/groups",
'method': requests.post,
'args': {
'required': ['name'],
'optional': ['visible'],
}
},
'add_user': {
'path': "/admin/groups/%s/members.json",
'method': requests.put,
'args': {
'required': ['usernames'],
'optional': [],
},
},
'remove_user': {
'path': "/admin/groups/%s/members.json",
'method': requests.delete,
'args': {
'required': ['username'],
'optional': [],
},
},
'delete': {
'path': "/admin/groups/%s.json",
'method': requests.delete,
'args': {
'required': [],
'optional': [],
},
},
},
'users': {
'create': {
'path': "/users",
'method': requests.post,
'args': {
'required': ['name', 'email', 'password', 'username'],
'optional': ['active'],
},
},
'update': {
'path': "/users/%s.json",
'method': requests.put,
'args': {
'required': ['params'],
'optional': [],
}
},
'get': {
'path': "/users/%s.json",
'method': requests.get,
'args': {
'required': [],
'optional': [],
},
},
'activate': {
'path': "/admin/users/%s/activate",
'method': requests.put,
'args': {
'required': [],
'optional': [],
},
},
'set_email': {
'path': "/users/%s/preferences/email",
'method': requests.put,
'args': {
'required': ['email'],
'optional': [],
},
},
'suspend': {
'path': "/admin/users/%s/suspend",
'method': requests.put,
'args': {
'required': ['duration', 'reason'],
'optional': [],
},
},
'unsuspend': {
'path': "/admin/users/%s/unsuspend",
'method': requests.put,
'args': {
'required': [],
'optional': [],
},
},
'logout': {
'path': "/admin/users/%s/log_out",
'method': requests.post,
'args': {
'required': [],
'optional': [],
},
},
'external': {
'path': "/users/by-external/%s.json",
'method': requests.get,
'args': {
'required': [],
'optional': [],
},
},
},
}
class DiscourseManager:
def __init__(self):
pass
GROUP_CACHE_MAX_AGE = datetime.timedelta(minutes=30)
REVOKED_EMAIL = 'revoked@' + settings.DOMAIN
SUSPEND_DAYS = 99999
SUSPEND_REASON = "Disabled by auth."
@staticmethod
def __exc(endpoint, *args, **kwargs):
params = {
'api_key': settings.DISCOURSE_API_KEY,
'api_username': settings.DISCOURSE_API_USERNAME,
}
silent = kwargs.pop('silent', False)
if args:
endpoint['parsed_url'] = endpoint['path'] % args
else:
endpoint['parsed_url'] = endpoint['path']
data = {}
for arg in endpoint['args']['required']:
data[arg] = kwargs[arg]
for arg in endpoint['args']['optional']:
if arg in kwargs:
data[arg] = kwargs[arg]
for arg in kwargs:
if arg not in endpoint['args']['required'] and arg not in endpoint['args']['optional'] and not silent:
logger.warn("Received unrecognized kwarg %s for endpoint %s" % (arg, endpoint))
r = endpoint['method'](settings.DISCOURSE_URL + endpoint['parsed_url'], params=params, json=data)
try:
if 'errors' in r.json() and not silent:
logger.error("Discourse execution failed.\nEndpoint: %s\nErrors: %s" % (endpoint, r.json()['errors']))
raise DiscourseError(endpoint, r.json()['errors'])
if 'success' in r.json():
if not r.json()['success'] and not silent:
raise DiscourseError(endpoint, None)
out = r.json()
except ValueError:
out = r.text
finally:
try:
r.raise_for_status()
except requests.exceptions.HTTPError as e:
raise DiscourseError(endpoint, e.response.status_code)
return out
@staticmethod
def __generate_random_pass():
return ''.join([random.choice(string.ascii_letters + string.digits) for n in range(16)])
@staticmethod
def __get_groups():
endpoint = ENDPOINTS['groups']['list']
data = DiscourseManager.__exc(endpoint)
return [g for g in data if not g['automatic']]
@staticmethod
def __update_group_cache():
GroupCache.objects.filter(service="discourse").delete()
cache = GroupCache.objects.create(service="discourse")
cache.groups = json.dumps(DiscourseManager.__get_groups())
cache.save()
return cache
@staticmethod
def __get_group_cache():
if not GroupCache.objects.filter(service="discourse").exists():
DiscourseManager.__update_group_cache()
cache = GroupCache.objects.get(service="discourse")
age = timezone.now() - cache.created
if age > DiscourseManager.GROUP_CACHE_MAX_AGE:
logger.debug("Group cache has expired. Triggering update.")
cache = DiscourseManager.__update_group_cache()
return json.loads(cache.groups)
@staticmethod
def __create_group(name):
endpoint = ENDPOINTS['groups']['create']
DiscourseManager.__exc(endpoint, name=name[:20], visible=True)
DiscourseManager.__update_group_cache()
@staticmethod
def __group_name_to_id(name):
cache = DiscourseManager.__get_group_cache()
for g in cache:
if g['name'] == name[0:20]:
return g['id']
logger.debug("Group %s not found on Discourse. Creating" % name)
DiscourseManager.__create_group(name)
return DiscourseManager.__group_name_to_id(name)
@staticmethod
def __group_id_to_name(id):
cache = DiscourseManager.__get_group_cache()
for g in cache:
if g['id'] == id:
return g['name']
raise KeyError("Group ID %s not found on Discourse" % id)
@staticmethod
def __add_user_to_group(id, username):
endpoint = ENDPOINTS['groups']['add_user']
DiscourseManager.__exc(endpoint, id, usernames=[username])
@staticmethod
def __remove_user_from_group(id, username):
endpoint = ENDPOINTS['groups']['remove_user']
DiscourseManager.__exc(endpoint, id, username=username)
@staticmethod
def __generate_group_dict(names):
group_dict = {}
for name in names:
group_dict[name] = DiscourseManager.__group_name_to_id(name)
return group_dict
@staticmethod
def __get_user_groups(username):
data = DiscourseManager.__get_user(username)
return [g['id'] for g in data['user']['groups'] if not g['automatic']]
@staticmethod
def __user_name_to_id(name, silent=False):
data = DiscourseManager.__get_user(name, silent=silent)
return data['user']['id']
@staticmethod
def __user_id_to_name(id):
raise NotImplementedError
@staticmethod
def __get_user(username, silent=False):
endpoint = ENDPOINTS['users']['get']
return DiscourseManager.__exc(endpoint, username, silent=silent)
@staticmethod
def __activate_user(username):
endpoint = ENDPOINTS['users']['activate']
id = DiscourseManager.__user_name_to_id(username)
DiscourseManager.__exc(endpoint, id)
@staticmethod
def __update_user(username, **kwargs):
endpoint = ENDPOINTS['users']['update']
id = DiscourseManager.__user_name_to_id(username)
DiscourseManager.__exc(endpoint, id, params=kwargs)
@staticmethod
def __create_user(username, email, password):
endpoint = ENDPOINTS['users']['create']
DiscourseManager.__exc(endpoint, name=username, username=username, email=email, password=password, active=True)
@staticmethod
def __check_if_user_exists(username):
try:
DiscourseManager.__user_name_to_id(username, silent=True)
return True
except:
return False
@staticmethod
def __suspend_user(username):
id = DiscourseManager.__user_name_to_id(username)
endpoint = ENDPOINTS['users']['suspend']
return DiscourseManager.__exc(endpoint, id, duration=DiscourseManager.SUSPEND_DAYS,
reason=DiscourseManager.SUSPEND_REASON)
@staticmethod
def __unsuspend(username):
id = DiscourseManager.__user_name_to_id(username)
endpoint = ENDPOINTS['users']['unsuspend']
return DiscourseManager.__exc(endpoint, id)
@staticmethod
def __set_email(username, email):
endpoint = ENDPOINTS['users']['set_email']
return DiscourseManager.__exc(endpoint, username, email=email)
@staticmethod
def __logout(id):
endpoint = ENDPOINTS['users']['logout']
return DiscourseManager.__exc(endpoint, id)
@staticmethod
def __get_user_by_external(id):
endpoint = ENDPOINTS['users']['external']
return DiscourseManager.__exc(endpoint, id)
@staticmethod
def __user_id_by_external_id(id):
data = DiscourseManager.__get_user_by_external(id)
return data['user']['id']
@staticmethod
def _sanitize_username(username):
sanitized = username.replace(" ", "_")
sanitized = sanitized.strip(' _')
sanitized = sanitized.replace("'", "")
return sanitized
@staticmethod
def _sanitize_groupname(name):
name = name.strip(' _')
name = re.sub('[^\w]', '', name)
if len(name) < 3:
name = name + "".join('_' for i in range(3-len(name)))
return name[:20]
@staticmethod
def update_groups(user):
groups = []
for g in user.groups.all():
groups.append(DiscourseManager._sanitize_groupname(str(g)[:20]))
logger.debug("Updating discourse user %s groups to %s" % (user, groups))
group_dict = DiscourseManager.__generate_group_dict(groups)
inv_group_dict = {v: k for k, v in group_dict.items()}
username = DiscourseManager.__get_user_by_external(user.pk)['user']['username']
user_groups = DiscourseManager.__get_user_groups(username)
add_groups = [group_dict[x] for x in group_dict if not group_dict[x] in user_groups]
rem_groups = [x for x in user_groups if not x in inv_group_dict]
if add_groups or rem_groups:
logger.info(
"Updating discourse user %s groups: adding %s, removing %s" % (username, add_groups, rem_groups))
for g in add_groups:
DiscourseManager.__add_user_to_group(g, username)
for g in rem_groups:
DiscourseManager.__remove_user_from_group(g, username)
@staticmethod
def disable_user(user):
logger.debug("Disabling user %s Discourse access." % user)
d_user = DiscourseManager.__get_user_by_external(user.pk)
DiscourseManager.__logout(d_user['user']['id'])
DiscourseManager.__suspend_user(d_user['user']['username'])
logger.info("Disabled user %s Discourse access." % user)
return True