Commit 9a0027ca authored by Brion Vibber's avatar Brion Vibber

Merge branch 'yammer-master'

parents 4e5a69ad 5b3de9aa
Yammer Import Plugin
====================
This plugin allows a one-time import pulling user accounts, groups, and
public messages from an existing Yammer instance, using Yammer's public API.
Requirements
------------
* An account on the Yammer network you wish to import from
* An administrator account on the target StatusNet instance, or
command-line administrative access
* This YammerImport plugin enabled on your StatusNet instance
Limitations
-----------
Yammer API key registrations only work for your own network unless you make
arrangements for a 'trusted app' key, so for now users will need to register
the app themselves. There is a helper in the admin panel for this.
In theory any number of users, groups, and messages should be supported, but
it hasn't been fully tested on non-trivial-sized sites.
No provision has yet been made for dealing with conflicting usernames or
group names, or names which are not considered valid by StatusNet. Errors
are possible.
Running via the web admin interface requires having queueing enabled, and is
fairly likely to have problems with the application key registration step in
a small installation at this time.
Web setup
---------
The import process is runnable through an administration panel on your
StatusNet site. The user interface is still a bit flaky, however, and if
errors occur during import the process may stop with no way to restart it
visible.
The admin interface will probably kinda blow up if JS/AJAX isn't working.
You'll be prompted to register the application and authenticate into Yammer,
after which a progress screen will display.
Two big warnings:
* The progress display does not currently auto-refresh.
* If anything fails once actual import has begun, it'll just keep showing
the current state. You won't see an error message, and there's no way
to reset or restart from the web UI yet.
You can continue or reset the import state using the command-line script.
CLI setup
---------
You'll need to register an application consumer key to allow the importer
to connect to your Yammer network; this requires logging into Yammer:
https://www.yammer.com/client_applications/new
Check all the 'read' options; no 'write' options are required, but Yammer
seems to end up setting them anyway.
You can set the resulting keys directly in config.php:
$config['yammer']['consumer_key'] = '#####';
$config['yammer']['consumer_secret'] = '##########';
Initiate authentication by starting up the importer script:
php plugins/YammerImport/scripts/yammer-import.php
Since you haven't yet authenticated, this will request an auth token and
give you a URL to open in your web browser. Once logged in and authorized
there, you'll be given a confirmation code. Pass this back:
php plugins/YammerImport/scripts/yammer-import.php --verify=####
If all is well, the import process will begin and run through the end.
In case of error or manual abort, you should be able to continue the
import from where you left off by running the script again:
php plugins/YammerImport/scripts/yammer-import.php
To reset the Yammer import state -- without removing any of the items
that have already been imported -- you can pass the --reset option:
php plugins/YammerImport/scripts/yammer-import.php --reset
This'll let you start over from the requesting-authentication stage.
Any users, groups, or notices that have already been imported will be
retained.
Subscriptions and group memberships
-----------------------------------
Yammer's API does not expose user/tag subscriptions or group memberships
except for the authenticating user. As a result, users will need to re-join
groups and re-follow their fellow users after the import.
(This limitation may be lifted in future for sites on the Silver or Gold
plans where the import is done by a verified admin, as it should be possible
to fetch the information for each user via the admin account.)
Authentication
--------------
Account passwords cannot be retrieved, but the primary e-mail address is
retained so users can reset their passwords by mail if you're not using a
custom authentication system like LDAP.
Private messages and groups
---------------------------
At this time, only public messages are imported; private direct and group
messages are ignored. (This may change with Silver and Gold plans in future.)
Yammer groups may be either public or private. Groups in StatusNet currently
have no privacy option, so any private groups will become public groups in the
imported site.
Attachments
-----------
Attached image and document files will be copied in as if they had been
uploaded to the StatusNet site. Currently images do not display inline like
they do on Yammer; they will be linked instead.
File type and size limitations on attachments will be applied, so beware some
attachments may not make it through.
Code structure
==============
Standalone classes
------------------
YammerRunner: encapsulates the iterative process of retrieving the various users,
groups, and messages via SN_YammerClient and saving them locally
via YammerImporter.
SN_YammerClient: encapsulates HTTP+OAuth interface to Yammer API, returns data
as straight decoded JSON object trees.
YammerImporter: encapsulates logic to pull information from the returned API data
and convert them to native StatusNet users, groups, and messages.
Web UI actions
-------------
YammeradminpanelAction: web panel for site administrator to initiate and monitor
the import process.
Command-line scripts
--------------------
yammer-import.php: CLI script to start a Yammer import run in one go.
Database objects
----------------
Yammer_state: data object storing YammerRunner's state between iterations.
Yammer_notice_stub: data object for temporary storage of fetched Yammer messages
between fetching them (reverse chron order) and saving them
to local messages (forward chron order).
Yammer_user,
Yammer_group,
Yammer_notice: data objects mapping original Yammer item IDs to their local copies.
<?php
/*
* StatusNet - the distributed open-source microblogging tool
* Copyright (C) 2010, StatusNet, Inc.
*
* This program is free software: you can redistribute it and/or modify
* it under the terms of the GNU Affero General Public License as published by
* the Free Software Foundation, either version 3 of the License, or
* (at your option) any later version.
*
* This program is distributed in the hope that it will be useful,
* but WITHOUT ANY WARRANTY; without even the implied warranty of
* MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
* GNU Affero General Public License for more details.
*
* You should have received a copy of the GNU Affero General Public License
* along with this program. If not, see <http://www.gnu.org/licenses/>.
*/
/**
* @package YammerImportPlugin
* @maintainer Brion Vibber <brion@status.net>
*/
if (!defined('STATUSNET')) { exit(1); }
class YammerImportPlugin extends Plugin
{
/**
* Hook for RouterInitialized event.
*
* @param Net_URL_Mapper $m path-to-action mapper
* @return boolean hook return
*/
function onRouterInitialized($m)
{
$m->connect('admin/yammer',
array('action' => 'yammeradminpanel'));
$m->connect('admin/yammer/auth',
array('action' => 'yammerauth'));
return true;
}
/**
* Set up queue handlers for import processing
* @param QueueManager $qm
* @return boolean hook return
*/
function onEndInitializeQueueManager(QueueManager $qm)
{
$qm->connect('yammer', 'YammerQueueHandler');
return true;
}
/**
* Set up all our tables...
*/
function onCheckSchema()
{
$schema = Schema::get();
$tables = array('Yammer_state',
'Yammer_user',
'Yammer_group',
'Yammer_notice',
'Yammer_notice_stub');
foreach ($tables as $table) {
$schema->ensureTable(strtolower($table), $table::schemaDef());
}
return true;
}
/**
* If the plugin's installed, this should be accessible to admins.
*/
function onAdminPanelCheck($name, &$isOK)
{
if ($name == 'yammer') {
$isOK = true;
return false;
}
return true;
}
/**
* Add the Yammer admin panel to the list...
*/
function onEndAdminPanelNav($nav)
{
if (AdminPanelAction::canAdmin('yammer')) {
$action_name = $nav->action->trimmed('action');
$nav->out->menuItem(common_local_url('yammeradminpanel'),
_m('Yammer'),
_m('Yammer import'),
$action_name == 'yammeradminpanel',
'nav_yammer_admin_panel');
}
return true;
}
/**
* Automatically load the actions and libraries used by the plugin
*
* @param Class $cls the class
*
* @return boolean hook return
*
*/
function onAutoload($cls)
{
$base = dirname(__FILE__);
$lower = strtolower($cls);
switch ($lower) {
case 'sn_yammerclient':
case 'yammerimporter':
case 'yammerrunner':
case 'yammerapikeyform':
case 'yammerauthinitform':
case 'yammerauthverifyform':
case 'yammerprogressform':
case 'yammerqueuehandler':
require_once "$base/lib/$lower.php";
return false;
case 'yammeradminpanelaction':
$crop = substr($lower, 0, strlen($lower) - strlen('action'));
require_once "$base/actions/$crop.php";
return false;
case 'yammer_state':
case 'yammer_notice_stub':
case 'yammer_common':
case 'yammer_user':
case 'yammer_group':
case 'yammer_notice':
require_once "$base/classes/$cls.php";
return false;
default:
return true;
}
}
}
<?php
/**
* StatusNet, the distributed open-source microblogging tool
*
* Yammer import administration panel
*
* PHP version 5
*
* LICENCE: This program is free software: you can redistribute it and/or modify
* it under the terms of the GNU Affero General Public License as published by
* the Free Software Foundation, either version 3 of the License, or
* (at your option) any later version.
*
* This program is distributed in the hope that it will be useful,
* but WITHOUT ANY WARRANTY; without even the implied warranty of
* MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
* GNU Affero General Public License for more details.
*
* You should have received a copy of the GNU Affero General Public License
* along with this program. If not, see <http://www.gnu.org/licenses/>.
*
* @category Settings
* @package StatusNet
* @author Zach Copley <zach@status.net>
* @copyright 2010 StatusNet, Inc.
* @license http://www.fsf.org/licensing/licenses/agpl-3.0.html GNU Affero General Public License version 3.0
* @link http://status.net/
*/
if (!defined('STATUSNET')) {
exit(1);
}
class YammeradminpanelAction extends AdminPanelAction
{
private $runner;
/**
* Returns the page title
*
* @return string page title
*/
function title()
{
return _m('Yammer Import');
}
/**
* Instructions for using this form.
*
* @return string instructions
*/
function getInstructions()
{
return _m('This Yammer import tool is still undergoing testing, ' .
'and is incomplete in some areas. ' .
'Currently user subscriptions and group memberships are not ' .
'transferred; in the future this may be supported for ' .
'imports done by verified administrators on the Yammer side.');
}
function prepare($args)
{
$ok = parent::prepare($args);
$this->subaction = $this->trimmed('subaction');
$this->runner = YammerRunner::init();
return $ok;
}
function handle($args)
{
// @fixme move this to saveSettings and friends?
if ($_SERVER['REQUEST_METHOD'] == 'POST') {
$this->checkSessionToken();
if ($this->subaction == 'change-apikey') {
$form = new YammerApiKeyForm($this);
} else if ($this->subaction == 'apikey') {
if ($this->saveKeys()) {
$form = new YammerAuthInitForm($this, $this->runner);
} else {
$form = new YammerApiKeyForm($this);
}
} else if ($this->subaction == 'authinit') {
// hack
if ($this->arg('change-apikey')) {
$form = new YammerApiKeyForm($this);
} else {
$url = $this->runner->requestAuth();
$form = new YammerAuthVerifyForm($this, $this->runner);
}
} else if ($this->subaction == 'authverify') {
$this->runner->saveAuthToken($this->trimmed('verify_token'));
// Haho! Now we can make THE FUN HAPPEN
$this->runner->startBackgroundImport();
$form = new YammerProgressForm($this, $this->runner);
} else {
throw new ClientException('Invalid POST');
}
return $this->showAjaxForm($form);
}
return parent::handle($args);
}
function saveKeys()
{
$key = $this->trimmed('consumer_key');
$secret = $this->trimmed('consumer_secret');
Config::save('yammer', 'consumer_key', $key);
Config::save('yammer', 'consumer_secret', $secret);
return !empty($key) && !empty($secret);
}
function showAjaxForm($form)
{
$this->startHTML('text/xml;charset=utf-8');
$this->elementStart('head');
$this->element('title', null, _m('Yammer import'));
$this->elementEnd('head');
$this->elementStart('body');
$form->show();
$this->elementEnd('body');
$this->elementEnd('html');
}
/**
* Fetch the appropriate form for our current state.
* @return Form
*/
function statusForm()
{
if (!(common_config('yammer', 'consumer_key'))
|| !(common_config('yammer', 'consumer_secret'))) {
return new YammerApiKeyForm($this);
}
switch($this->runner->state())
{
case 'init':
return new YammerAuthInitForm($this, $this->runner);
case 'requesting-auth':
return new YammerAuthVerifyForm($this, $this->runner);
default:
return new YammerProgressForm($this, $this->runner);
}
}
/**
* Show the Yammer admin panel form
*
* @return void
*/
function showForm()
{
$this->elementStart('fieldset');
$this->statusForm()->show();
$this->elementEnd('fieldset');
}
function showStylesheets()
{
parent::showStylesheets();
$this->cssLink('plugins/YammerImport/css/admin.css', null, 'screen, projection, tv');
}
function showScripts()
{
parent::showScripts();
$this->script('plugins/YammerImport/js/yammer-admin.js');
}
}
<?php
/**
* StatusNet, the distributed open-source microblogging tool
*
* Yammer import administration panel
*
* PHP version 5
*
* LICENCE: This program is free software: you can redistribute it and/or modify
* it under the terms of the GNU Affero General Public License as published by
* the Free Software Foundation, either version 3 of the License, or
* (at your option) any later version.
*
* This program is distributed in the hope that it will be useful,
* but WITHOUT ANY WARRANTY; without even the implied warranty of
* MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
* GNU Affero General Public License for more details.
*
* You should have received a copy of the GNU Affero General Public License
* along with this program. If not, see <http://www.gnu.org/licenses/>.
*
* @category Settings
* @package StatusNet
* @author Zach Copley <zach@status.net>
* @copyright 2010 StatusNet, Inc.
* @license http://www.fsf.org/licensing/licenses/agpl-3.0.html GNU Affero General Public License version 3.0
* @link http://status.net/
*/
if (!defined('STATUSNET')) {
exit(1);
}
class YammerauthAction extends AdminPanelAction
{
/**
* Show the Yammer admin panel form
*
* @return void
*/
function prepare($args)
{
parent::prepare($args);
$this->verify_token = $this->trim('verify_token');
}
/**
* Handle request
*
* Does the subscription and returns results.
*
* @param Array $args unused.
*
* @return void
*/
function handle($args)
{
if ($this->verify_token) {
$runner->saveAuthToken($this->verify_token);
$form = new YammerAuthProgressForm();
} else {
$url = $runner->requestAuth();
$form = new YammerAuthVerifyForm($this, $url);
}
$this->startHTML('text/xml;charset=utf-8');
$this->elementStart('head');
$this->element('title', null, _m('Connect to Yammer'));
$this->elementEnd('head');
$this->elementStart('body');
$form->show();
$this->elementEnd('body');
$this->elementEnd('html');
}
}
<?php
/**
* Data class for remembering Yammer import mappings
*
* PHP version 5
*
* @category Data
* @package StatusNet
* @author Brion Vibber <brion@status.net>
* @license http://www.fsf.org/licensing/licenses/agpl.html AGPLv3
* @link http://status.net/
*
* StatusNet - the distributed open-source microblogging tool
* Copyright (C) 2010, StatusNet, Inc.
*
* This program is free software: you can redistribute it and/or modify
* it under the terms of the GNU Affero General Public License as published by
* the Free Software Foundation, either version 3 of the License, or
* (at your option) any later version.
*
* This program is distributed in the hope that it will be useful,
* but WITHOUT ANY WARRANTY; without even the implied warranty of
* MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
* GNU Affero General Public License for more details.
*
* You should have received a copy of the GNU Affero General Public License
* along with this program. If not, see <http://www.gnu.org/licenses/>.
*/
if (!defined('STATUSNET')) {
exit(1);
}
/**
* Common base class for the Yammer import mappings for users, groups, and notices.
*
* Child classes must override these static methods, since we need to run
* on PHP 5.2.x which has no late static binding:
* - staticGet (as our other classes)
* - schemaDef (call self::doSchemaDef)
* - record (call self::doRecord)
*/
class Yammer_common extends Memcached_DataObject
{
public $__table = 'yammer_XXXX'; // table name
public $__field = 'XXXX_id'; // field name to save into
public $id; // int primary_key not_null
public $user_id; // int(4)
public $created; // datetime
/**
* @fixme add a 'references' thing for the foreign key when we support that
*/
protected static function doSchemaDef($field)
{
return array(new ColumnDef('id', 'bigint', null,
false, 'PRI'),
new ColumnDef($field, 'integer', null,
false, 'UNI'),
new ColumnDef('created', 'datetime', null,
false));
}
/**
* return table definition for DB_DataObject
*
* DB_DataObject needs to know something about the table to manipulate
* instances. This method provides all the DB_DataObject needs to know.
*
* @return array array of column definitions
*/
function table()
{
return array('id' => DB_DATAOBJECT_INT + DB_DATAOBJECT_NOTNULL,
$this->__field => DB_DATAOBJECT_INT + DB_DATAOBJECT_NOTNULL,
'created' => DB_DATAOBJECT_STR + DB_DATAOBJECT_DATE + DB_DATAOBJECT_TIME + DB_DATAOBJECT_NOTNULL);
}
/**
* return key definitions for DB_DataObject
*
* DB_DataObject needs to know about keys that the table has, since it
* won't appear in StatusNet's own keys list. In most cases, this will
* simply reference your keyTypes() function.
*
* @return array list of key field names
*/
function keys()
{
return array_keys($this->keyTypes());
}
/**
* return key definitions for Memcached_DataObject
*
* Our caching system uses the same key definitions, but uses a different
* method to get them. This key information is used to store and clear
* cached data, so be sure to list any key that will be used for static
* lookups.
*
* @return array associative array of key definitions, field name to type:
* 'K' for primary key: for compound keys, add an entry for each component;
* 'U' for unique keys: compound keys are not well supported here.
*/
function keyTypes()
{
return array('id' => 'K', $this->__field => 'U');
}
/**
* Magic formula for non-autoincrementing integer primary keys
*
* If a table has a single integer column as its primary key, DB_DataObject
* assumes that the column is auto-incrementing and makes a sequence table
* to do this incrementation. Since we don't need this for our class, we
* overload this method and return the magic formula that DB_DataObject needs.
*
* @return array magic three-false array that stops auto-incrementing.
*/
function sequenceKey()
{
return array(false, false, false);
}
/**
* Save a mapping between a remote Yammer and local imported user.
*
* @param integer $user_id ID of the status in StatusNet
* @param integer $orig_id ID of the notice in Yammer
*
* @return Yammer_common new object for this value
*/
protected static function doRecord($class, $field, $orig_id, $local_id)
{
$map = parent::staticGet($class, 'id', $orig_id);
if (!empty($map)) {
return $map;
}
$map = parent::staticGet($class, $field, $local_id);
if (!empty($map)) {
return $map;
}