The Great Services Refactor (#594)

* Hooks registration, discovery and retrieval module

Will discover @hooks.register decorated functions inside
the auth_hooks module in any installed django app.

* Class to register modular service apps

* Register service modules URLs

* Example service module

* Refactor services into modules

Each service type has been split out into its own django app/module. A
hook mechanism is provided to register a subclass of the ServiceHook
class. The modules then overload functions defined in ServiceHook as
required to provide interoperability with alliance auth. Service modules
provide their own urls and views for user registration and account
management and a partial template to display on the services page. Where
possible, new modules should provide their own models for local data
storage.

* Added menu items hooks and template tags

* Added menu item hook for broadcasts

* Added str method to ServicesHook

* Added exception handling to hook iterators

* Refactor mumble migration and table name

Upgrading will require `migrate mumble --fake-initial` to be run first
and then `migrate mumble` to rename the table.

* Refactor teamspeak3 migration and rename table

Upgrading will require `migrate teamspeak3 --fake-initial`

* Added module models and migrations for refactoring AuthServicesInfo

* Migrate AuthServiceInfo fields to service modules models

* Added helper for getting a users main character

* Added new style celery instance

* Changed Discord from AuthServicesInfo to DiscordUser model

* Switch celery tasks to staticmethods

* Changed Discourse from AuthServicesInfo to DiscourseUser model

* Changed IPBoard from AuthServicesInfo to IpboardUser model

* Changed Ips4 from AuthServicesInfo to Ips4User model

Also added disable service task.

This service still needs some love though. Was always missing a
deactivate services hook (before refactoring) for reasons I'm unsure of
so I'm reluctant to add it without knowing why.

* Changed Market from AuthServicesInfo to MarketUser model

* Changed Mumble from AuthServicesInfo to MumbleUser model

Switched user foreign key to one to one relationship.
Removed implicit password change on user exists.
Combined regular and blue user creation.

* Changed Openfire from AuthServicesInfo to OpenfireUser model

* Changed SMF from AuthServicesInfo to SmfUser model

Added disable task

* Changed Phpbb3 from AuthServicesInfo to Phpbb3User model

* Changed XenForo from AuthServicesInfo to XenforoUser model

* Changed Teamspeak3 from AuthServicesInfo to Teamspeak3User model

* Remove obsolete manager functions

* Standardise URL format

This will break some callback URLs
Discord changes from /discord_callback/ to /discord/callback/

* Removed unnecessary imports

* Mirror upstream decorator change

* Setup for unit testing

* Unit tests for discord service

* Added add main character helper

* Added Discourse unit tests

* Added Ipboard unit tests

* Added Ips4 unit tests

* Fix naming of market manager, switch to use class methods

* Remove unused hook functions

* Added market service unit tests

* Added corp ticker to add main character helper

* Added mumble unit tests

* Fix url name and remove namespace

* Fix missing return and add missing URL

* Added openfire unit tests

* Added missing return

* Added phpbb3 unit tests

* Fix SmfManager naming inconsistency and switch to classmethods

* Added smf unit tests

* Remove unused functions, Added missing return

* Added xenforo unit tests

* Added missing return

* Fixed reference to old model

* Fixed error preventing groups from syncing on reset request

* Added teamspeak3 unit tests

* Added nose as test runner and some test settings

* Added package requirements for running tests

* Added unit tests for services signals and tasks

* Remove unused tests file

* Fix teamspeak3 service signals

* Added unit tests for teamspeak3 signals

Changed other unit tests setUp to inert signals

* Fix password gen and hashing python3 compatibility

Fixes #630

Adds unit tests to check the password functions run on both platforms.

* Fix unit test to not rely on checking url params

* Add Travis CI settings file

* Remove default blank values from services models

* Added dynamic user model admin actions for syncing service groups

* Remove unused search fields

* Add hook function for syncing nicknames

* Added discord hook for sync nickname

* Added user admin model menu actions for sync nickname hook

* Remove obsolete code

* Rename celery config app to avoid package name clash

* Added new style celerybeat schedule configuration

periodic_task decorator is depreciated

* Added string representations

* Added admin pages for services user models

* Removed legacy code

* Move link discord button to correct template

* Remove blank default fields from example model

* Disallow empty django setting

* Fix typos

* Added coverage configuration file

* Add coverage and coveralls to travis config

Should probably use nose's built in coverage, but this works for now.

* Replace AuthServicesInfo get_or_create instances with get

Reflects upstream changes to AuthServicesInfo behaviour.

* Update mumble user table name

* Split out mumble authenticator requirements

zeroc-ice seems to cause long build times on travis-ci and isn't
required for the core projects functionality or testing.
This commit is contained in:
Basraah
2017-01-25 12:50:16 +10:00
committed by GitHub
parent 5738b015c3
commit 1066e6ac98
195 changed files with 8260 additions and 2699 deletions

View File

View File

@@ -0,0 +1,10 @@
from __future__ import unicode_literals
from django.contrib import admin
from .models import MarketUser
class MarketUserAdmin(admin.ModelAdmin):
list_display = ('user', 'username')
search_fields = ('user__username', 'username')
admin.site.register(MarketUser, MarketUserAdmin)

View File

@@ -0,0 +1,7 @@
from __future__ import unicode_literals
from django.apps import AppConfig
class MarketServiceConfig(AppConfig):
name = 'market'

View File

@@ -0,0 +1,59 @@
from __future__ import unicode_literals
from django.conf import settings
from django.template.loader import render_to_string
from services.hooks import ServicesHook
from alliance_auth import hooks
from .urls import urlpatterns
from .tasks import MarketTasks
import logging
logger = logging.getLogger(__name__)
class MarketService(ServicesHook):
def __init__(self):
ServicesHook.__init__(self)
self.name = 'market'
self.urlpatterns = urlpatterns
self.service_url = settings.MARKET_URL
@property
def title(self):
return "Alliance Market"
def delete_user(self, user, notify_user=False):
logger.debug('Deleting user %s %s account' % (user, self.name))
return MarketTasks.delete_user(user, notify_user=notify_user)
def validate_user(self, user):
logger.debug('Validating user %s %s account' % (user, self.name))
if MarketTasks.has_account(user) and self.service_active_for_user(user):
self.delete_user(user)
def service_enabled_members(self):
return settings.ENABLE_AUTH_MARKET or False
def service_enabled_blues(self):
return settings.ENABLE_BLUE_MARKET or False
def render_services_ctrl(self, request):
urls = self.Urls()
urls.auth_activate = 'auth_activate_market'
urls.auth_deactivate = 'auth_deactivate_market'
urls.auth_reset_password = 'auth_reset_market_password'
urls.auth_set_password = 'auth_set_market_password'
return render_to_string(self.service_ctrl_template, {
'service_name': self.title,
'urls': urls,
'service_url': self.service_url,
'username': request.user.market.username if MarketTasks.has_account(request.user) else ''
}, request=request)
@hooks.register('services_hook')
def register_service():
return MarketService()

View File

@@ -0,0 +1,149 @@
from __future__ import unicode_literals
import logging
import random
import string
import re
from django.db import connections
from passlib.hash import bcrypt
# requires yum install libffi-devel and pip install bcrypt
logger = logging.getLogger(__name__)
class MarketManager:
def __init__(self):
pass
SQL_ADD_USER = r"INSERT INTO fos_user (username, username_canonical, email, email_canonical, enabled, salt," \
r"password, locked, expired, roles, credentials_expired, characterid, characterName)" \
r"VALUES (%s, %s, %s, %s, 1,%s, %s, 0, 0, 'a:0:{}', 0, %s, %s) "
SQL_GET_USER_ID = r"SELECT id FROM fos_user WHERE username = %s"
SQL_DISABLE_USER = r"UPDATE fos_user SET enabled = '0' WHERE username = %s"
SQL_ENABLE_USER = r"UPDATE fos_user SET enabled = '1' WHERE username = %s"
SQL_UPDATE_PASSWORD = r"UPDATE fos_user SET password = %s, salt = %s WHERE username = %s"
SQL_CHECK_EMAIL = r"SELECT email FROM fos_user WHERE email = %s"
SQL_CHECK_USERNAME = r"SELECT username FROM fos_user WHERE username = %s"
SQL_UPDATE_USER = r"UPDATE fos_user SET password = %s, salt = %s, enabled = '1' WHERE username = %s"
@staticmethod
def __santatize_username(username):
sanatized = username.replace(" ", "_")
return sanatized.lower()
@staticmethod
def __generate_random_pass():
return ''.join([random.choice(string.ascii_letters + string.digits) for n in range(16)])
@staticmethod
def _gen_pwhash(password):
return bcrypt.encrypt(password.encode('utf-8'), rounds=13)
@staticmethod
def _get_salt(pw_hash):
search = re.compile(r"^\$2[a-z]?\$([0-9]+)\$(.{22})(.{31})$")
match = re.match(search, pw_hash)
return match.group(2)
@classmethod
def check_username(cls, username):
logger.debug("Checking alliance market username %s" % username)
cursor = connections['market'].cursor()
cursor.execute(cls.SQL_CHECK_USERNAME, [cls.__santatize_username(username)])
row = cursor.fetchone()
if row:
logger.debug("Found user %s on alliance market" % username)
return True
logger.debug("User %s not found on alliance market" % username)
return False
@classmethod
def check_user_email(cls, username, email):
logger.debug("Checking if alliance market email exists for user %s" % username)
cursor = connections['market'].cursor()
cursor.execute(cls.SQL_CHECK_EMAIL, [email])
row = cursor.fetchone()
if row:
logger.debug("Found user %s email address on alliance market" % username)
return True
logger.debug("User %s email address not found on alliance market" % username)
return False
@classmethod
def add_user(cls, username, email, characterid, charactername):
logger.debug("Adding new market user %s" % username)
plain_password = cls.__generate_random_pass()
hash = cls._gen_pwhash(plain_password)
salt = cls._get_salt(hash)
username_clean = cls.__santatize_username(username)
if not cls.check_username(username):
if not cls.check_user_email(username, email):
try:
logger.debug("Adding user %s to alliance market" % username)
cursor = connections['market'].cursor()
cursor.execute(cls.SQL_ADD_USER, [username_clean, username_clean, email, email, salt,
hash, characterid, charactername])
return username_clean, plain_password
except:
logger.debug("Unsuccessful attempt to add market user %s" % username)
return "", ""
else:
logger.debug("Alliance market email %s already exists Updating instead" % email)
username_clean, password = cls.update_user_info(username)
return username_clean, password
else:
logger.debug("Alliance market username %s already exists Updating instead" % username)
username_clean, password = cls.update_user_info(username)
return username_clean, password
@classmethod
def disable_user(cls, username):
logger.debug("Disabling alliance market user %s " % username)
cursor = connections['market'].cursor()
cursor.execute(cls.SQL_DISABLE_USER, [username])
return True
@classmethod
def update_custom_password(cls, username, plain_password):
logger.debug("Updating alliance market user %s password" % username)
if cls.check_username(username):
username_clean = cls.__santatize_username(username)
hash = cls._gen_pwhash(plain_password)
salt = cls._get_salt(hash)
cursor = connections['market'].cursor()
cursor.execute(cls.SQL_UPDATE_PASSWORD, [hash, salt, username_clean])
return plain_password
else:
logger.error("Unable to update alliance market user %s password" % username)
return ""
@classmethod
def update_user_password(cls, username):
logger.debug("Updating alliance market user %s password" % username)
if cls.check_username(username):
username_clean = cls.__santatize_username(username)
plain_password = cls.__generate_random_pass()
hash = cls._gen_pwhash(plain_password)
salt = cls._get_salt(hash)
cursor = connections['market'].cursor()
cursor.execute(cls.SQL_UPDATE_PASSWORD, [hash, salt, username_clean])
return plain_password
else:
logger.error("Unable to update alliance market user %s password" % username)
return ""
@classmethod
def update_user_info(cls, username):
logger.debug("Updating alliance market user %s" % username)
try:
username_clean = cls.__santatize_username(username)
plain_password = cls.__generate_random_pass()
hash = cls._gen_pwhash(plain_password)
salt = cls._get_salt(hash)
cursor = connections['market'].cursor()
cursor.execute(cls.SQL_UPDATE_USER, [hash, salt, username_clean])
return username_clean, plain_password
except:
logger.debug("Alliance market update user failed for %s" % username)
return "", ""

View File

@@ -0,0 +1,26 @@
# -*- coding: utf-8 -*-
# Generated by Django 1.10.2 on 2016-12-12 03:27
from __future__ import unicode_literals
from django.conf import settings
from django.db import migrations, models
import django.db.models.deletion
class Migration(migrations.Migration):
initial = True
dependencies = [
('auth', '0008_alter_user_username_max_length'),
]
operations = [
migrations.CreateModel(
name='MarketUser',
fields=[
('user', models.OneToOneField(on_delete=django.db.models.deletion.CASCADE, primary_key=True, related_name='market', serialize=False, to=settings.AUTH_USER_MODEL)),
('username', models.CharField(max_length=254)),
],
),
]

View File

@@ -0,0 +1,16 @@
from __future__ import unicode_literals
from django.utils.encoding import python_2_unicode_compatible
from django.contrib.auth.models import User
from django.db import models
@python_2_unicode_compatible
class MarketUser(models.Model):
user = models.OneToOneField(User,
primary_key=True,
on_delete=models.CASCADE,
related_name='market')
username = models.CharField(max_length=254)
def __str__(self):
return self.username

View File

@@ -0,0 +1,44 @@
from __future__ import unicode_literals
from django.conf import settings
from django.core.exceptions import ObjectDoesNotExist
from notifications import notify
from .models import MarketUser
from .manager import MarketManager
import logging
logger = logging.getLogger(__name__)
class MarketTasks:
def __init__(self):
pass
@classmethod
def delete_user(cls, user, notify_user=False):
if cls.has_account(user):
logger.debug("User %s has a Market account %s. Deleting." % (user, user.market.username))
if MarketManager.disable_user(user.market.username):
user.market.delete()
if notify_user:
notify(user, 'Alliance Market Account Disabled', level='danger')
return True
return False
@staticmethod
def has_account(user):
try:
return user.market.username != ''
except ObjectDoesNotExist:
return False
@staticmethod
def disable():
if settings.ENABLE_AUTH_MARKET:
logger.warn("ENABLE_AUTH_MARKET still True, after disabling users will still be able to activate Market accounts")
if settings.ENABLE_BLUE_MARKET:
logger.warn("ENABLE_BLUE_MARKET still True, after disabling blues will still be able to activate Market accounts")
MarketUser.objects.all().delete()

View File

@@ -0,0 +1,176 @@
from __future__ import unicode_literals
try:
# Py3
from unittest import mock
except ImportError:
# Py2
import mock
from django.test import TestCase, RequestFactory
from django.conf import settings
from django import urls
from django.contrib.auth.models import User
from django.core.exceptions import ObjectDoesNotExist
from alliance_auth.tests.auth_utils import AuthUtils
from .auth_hooks import MarketService
from .models import MarketUser
from .tasks import MarketTasks
MODULE_PATH = 'services.modules.market'
class MarketHooksTestCase(TestCase):
def setUp(self):
self.member = 'member_user'
member = AuthUtils.create_member(self.member)
MarketUser.objects.create(user=member, username=self.member)
self.blue = 'blue_user'
blue = AuthUtils.create_blue(self.blue)
MarketUser.objects.create(user=blue, username=self.blue)
self.none_user = 'none_user'
none_user = AuthUtils.create_user(self.none_user)
self.service = MarketService
def test_has_account(self):
member = User.objects.get(username=self.member)
blue = User.objects.get(username=self.blue)
none_user = User.objects.get(username=self.none_user)
self.assertTrue(MarketTasks.has_account(member))
self.assertTrue(MarketTasks.has_account(blue))
self.assertFalse(MarketTasks.has_account(none_user))
def test_service_enabled(self):
service = self.service()
member = User.objects.get(username=self.member)
blue = User.objects.get(username=self.blue)
none_user = User.objects.get(username=self.none_user)
self.assertTrue(service.service_enabled_members())
self.assertTrue(service.service_enabled_blues())
self.assertEqual(service.service_active_for_user(member), settings.ENABLE_AUTH_MARKET)
self.assertEqual(service.service_active_for_user(blue), settings.ENABLE_BLUE_MARKET)
self.assertFalse(service.service_active_for_user(none_user))
@mock.patch(MODULE_PATH + '.tasks.MarketManager')
def test_delete_user(self, manager):
member = User.objects.get(username=self.member)
service = self.service()
result = service.delete_user(member)
self.assertTrue(result)
self.assertTrue(manager.disable_user.called)
with self.assertRaises(ObjectDoesNotExist):
market_user = User.objects.get(username=self.member).market
def test_render_services_ctrl(self):
service = self.service()
member = User.objects.get(username=self.member)
request = RequestFactory().get('/en/services/')
request.user = member
response = service.render_services_ctrl(request)
self.assertTemplateUsed(service.service_ctrl_template)
self.assertIn(urls.reverse('auth_set_market_password'), response)
self.assertIn(urls.reverse('auth_reset_market_password'), response)
self.assertIn(urls.reverse('auth_deactivate_market'), response)
# Test register becomes available
member.market.delete()
member = User.objects.get(username=self.member)
request.user = member
response = service.render_services_ctrl(request)
self.assertIn(urls.reverse('auth_activate_market'), response)
class MarketViewsTestCase(TestCase):
def setUp(self):
self.member = AuthUtils.create_member('auth_member')
self.member.set_password('password')
self.member.email = 'auth_member@example.com'
self.member.save()
AuthUtils.add_main_character(self.member, 'auth_member', '12345', corp_id='111', corp_name='Test Corporation')
def login(self):
self.client.login(username=self.member.username, password='password')
@mock.patch(MODULE_PATH + '.views.MarketManager')
def test_activate(self, manager):
self.login()
expected_username = 'auth_member'
expected_password = 'password'
expected_id = '1234'
manager.add_user.return_value = (expected_username, expected_password, expected_id)
response = self.client.get(urls.reverse('auth_activate_market'), follow=False)
self.assertTrue(manager.add_user.called)
args, kwargs = manager.add_user.call_args
self.assertEqual(args[0], expected_username)
self.assertEqual(args[1], self.member.email)
self.assertTemplateUsed(response, 'registered/service_credentials.html')
self.assertContains(response, expected_username)
self.assertContains(response, expected_password)
@mock.patch(MODULE_PATH + '.tasks.MarketManager')
def test_deactivate(self, manager):
self.login()
MarketUser.objects.create(user=self.member, username='12345')
manager.disable_user.return_value = True
response = self.client.get(urls.reverse('auth_deactivate_market'))
self.assertTrue(manager.disable_user.called)
self.assertRedirects(response, expected_url=urls.reverse('auth_services'), target_status_code=200)
with self.assertRaises(ObjectDoesNotExist):
market_user = User.objects.get(pk=self.member.pk).market
@mock.patch(MODULE_PATH + '.views.MarketManager')
def test_set_password(self, manager):
self.login()
MarketUser.objects.create(user=self.member, username='12345')
expected_password = 'password'
manager.update_user_password.return_value = expected_password
response = self.client.post(urls.reverse('auth_set_market_password'), data={'password': expected_password})
self.assertTrue(manager.update_custom_password.called)
args, kwargs = manager.update_custom_password.call_args
self.assertEqual(args[1], expected_password)
self.assertRedirects(response, expected_url=urls.reverse('auth_services'), target_status_code=200)
@mock.patch(MODULE_PATH + '.views.MarketManager')
def test_reset_password(self, manager):
self.login()
MarketUser.objects.create(user=self.member, username='12345')
response = self.client.get(urls.reverse('auth_reset_market_password'))
self.assertTrue(manager.update_user_password.called)
self.assertTemplateUsed(response, 'registered/service_credentials.html')
class MarketManagerTestCase(TestCase):
def setUp(self):
from .manager import MarketManager
self.manager = MarketManager
def test_generate_random_password(self):
password = self.manager._MarketManager__generate_random_pass()
self.assertEqual(len(password), 16)
self.assertIsInstance(password, type(''))
def test_gen_pwhash(self):
pwhash = self.manager._gen_pwhash('test')
salt = self.manager._get_salt(pwhash)
self.assertIsInstance(pwhash, str)
self.assertGreaterEqual(len(pwhash), 59)
self.assertIsInstance(salt, str)
self.assertEqual(len(salt), 22)

View File

@@ -0,0 +1,16 @@
from __future__ import unicode_literals
from django.conf.urls import url, include
from . import views
module_urls = [
# Alliance Market Control
url(r'^activate/$', views.activate_market, name='auth_activate_market'),
url(r'^deactivate/$', views.deactivate_market, name='auth_deactivate_market'),
url(r'^reset_password/$', views.reset_market_password, name='auth_reset_market_password'),
url(r'^set_password/$', views.set_market_password, name='auth_set_market_password'),
]
urlpatterns = [
url(r'^market/', include(module_urls))
]

View File

@@ -0,0 +1,107 @@
from __future__ import unicode_literals
from django.contrib import messages
from django.contrib.auth.decorators import login_required
from django.shortcuts import render, redirect
from authentication.decorators import members_and_blues
from services.forms import ServicePasswordForm
from eveonline.managers import EveManager
from .manager import MarketManager
from .models import MarketUser
from .tasks import MarketTasks
import logging
logger = logging.getLogger(__name__)
@login_required
@members_and_blues()
def activate_market(request):
logger.debug("activate_market called by user %s" % request.user)
character = EveManager.get_main_character(request.user)
if character is not None:
logger.debug("Adding market user for user %s with main character %s" % (request.user, character))
result = MarketManager.add_user(character.character_name, request.user.email, character.character_id,
character.character_name)
# if empty we failed
if result[0] != "":
MarketUser.objects.create(user=request.user, username=result[0])
logger.debug("Updated authserviceinfo for user %s with market credentials." % request.user)
logger.info("Successfully activated market for user %s" % request.user)
messages.success(request, 'Activated Alliance Market account.')
credentials = {
'username': result[0],
'password': result[1],
}
return render(request, 'registered/service_credentials.html',
context={'credentials': credentials, 'service': 'Alliance Market'})
logger.error("Unsuccessful attempt to activate market for user %s" % request.user)
messages.error(request, 'An error occurred while processing your Alliance Market account.')
return redirect("auth_services")
@login_required
@members_and_blues()
def deactivate_market(request):
logger.debug("deactivate_market called by user %s" % request.user)
# false we failed
if MarketTasks.delete_user(request.user):
logger.info("Successfully deactivated market for user %s" % request.user)
messages.success(request, 'Deactivated Alliance Market account.')
else:
logger.error("Unsuccessful attempt to activate market for user %s" % request.user)
messages.error(request, 'An error occurred while processing your Alliance Market account.')
return redirect("auth_services")
@login_required
@members_and_blues()
def reset_market_password(request):
logger.debug("reset_market_password called by user %s" % request.user)
if MarketTasks.has_account(request.user):
result = MarketManager.update_user_password(request.user.market.username)
# false we failed
if result != "":
logger.info("Successfully reset market password for user %s" % request.user)
messages.success(request, 'Reset Alliance Market password.')
credentials = {
'username': request.user.market.username,
'password': result,
}
return render(request, 'registered/service_credentials.html',
context={'credentials': credentials, 'service': 'Alliance Market'})
logger.error("Unsuccessful attempt to reset market password for user %s" % request.user)
messages.error(request, 'An error occurred while processing your Alliance Market account.')
return redirect("auth_services")
@login_required
@members_and_blues()
def set_market_password(request):
logger.debug("set_market_password called by user %s" % request.user)
if request.method == 'POST':
logger.debug("Received POST request with form.")
form = ServicePasswordForm(request.POST)
logger.debug("Form is valid: %s" % form.is_valid())
if form.is_valid() and MarketTasks.has_account(request.user):
password = form.cleaned_data['password']
logger.debug("Form contains password of length %s" % len(password))
result = MarketManager.update_custom_password(request.user.market.username, password)
if result != "":
logger.info("Successfully reset market password for user %s" % request.user)
messages.success(request, 'Set Alliance Market password.')
else:
logger.error("Failed to install custom market password for user %s" % request.user)
messages.error(request, 'An error occurred while processing your Alliance Market account.')
return redirect("auth_services")
else:
logger.debug("Request is not type POST - providing empty form.")
form = ServicePasswordForm()
logger.debug("Rendering form for user %s" % request.user)
context = {'form': form, 'service': 'Market'}
return render(request, 'registered/service_password.html', context=context)