GitLab

Move OMB to a plugin

commit 75d21f00246bcc56d7f854936be1e28395e079a2
Merge: cea0199 d594d07
Author: Zach Copley <zach@status.net>
Date:   Fri Jul 15 11:16:54 2011 -0700

    Merge branch 'kill-omb2' of gitorious.org:~zcopley/statusnet/zcopleys-clone into kill-omb2

    * 'kill-omb2' of gitorious.org:~zcopley/statusnet/zcopleys-clone:
      Fix paths
      Oops, I left out the ability to authorize a token in ApiStatusNetOAuthDataStore
      Some odds and ends
      Remove omb stuff from queuemanager defaults
      Add check to make sure we're not untagging an OMB profile to OMB plugin
      Move some more subscription stuff and peopletag checks to OMB plugin
      Move some OMB-specific unsubscribe stuff to OMB plugin
      Finish removing libomb from core extlibs
      Fix more conflicts
      Fix queuing/queuehandling
      Move some stuff around; fix references
      Fix conflicts
      Move OMB-specific files to OMB plugin
      Move some stuff around; fix references
      Add OMB plugin README and rm references to OMB in mail StatusNet README
      Update paths
      Fix define
      Basic plugin finished
      Move OMB-specific files to OMB plugin
      Remove OMB stuff from router

commit cea019967f343042ebaea14b7bbb0d54289bcc1a
Author: Zach Copley <zach@status.net>
Date:   Wed Jul 13 14:38:40 2011 -0700

    Fix paths

commit d412aa3c0ea0e21e65a72a16c7b9edd64ff373e1
Author: Zach Copley <zach@status.net>
Date:   Tue Jul 12 18:17:06 2011 -0700

    Oops, I left out the ability to authorize a token in ApiStatusNetOAuthDataStore

commit b459c9f10ac283d6e774ef13f3293fc8a6948143
Author: Zach Copley <zach@status.net>
Date:   Wed Jul 6 19:02:08 2011 -0700

    Some odds and ends

commit 895cfbfce58ffb3a05beebf48a90c549e00f1cce
Author: Zach Copley <zach@status.net>
Date:   Wed Jul 6 19:01:23 2011 -0700

    Remove omb stuff from queuemanager defaults

commit b41b9e994f291ff83afb2460d9b37aee8ec1ec2b
Author: Zach Copley <zach@status.net>
Date:   Wed Jul 6 18:46:44 2011 -0700

    Add check to make sure we're not untagging an OMB profile to OMB plugin

commit 94374d26ddd428dac8e4cd4541fd56db748c248b
Author: Zach Copley <zach@status.net>
Date:   Wed Jul 6 18:34:20 2011 -0700

    Move some more subscription stuff and peopletag checks to OMB plugin

commit b91043b7820d5cd8b0ba4e9ee2a9d03c99248f11
Author: Zach Copley <zach@status.net>
Date:   Wed Jul 6 16:37:25 2011 -0700

    Move some OMB-specific unsubscribe stuff to OMB plugin

commit d9430fe52975d9497b4a0d3d54da35b222e207ad
Author: Zach Copley <zach@status.net>
Date:   Wed Jul 6 15:26:30 2011 -0700

    Finish removing libomb from core extlibs

commit bb6257eb85cc7ba392e91468c01503f51faeb989
Author: Zach Copley <zach@status.net>
Date:   Wed Jul 13 12:47:32 2011 -0700

    Fix more conflicts

commit 3c760d0a4b4a083ae5fca2530d22aad5f4a9fdae
Author: Zach Copley <zach@status.net>
Date:   Tue Jul 5 15:49:22 2011 -0700

    Fix queuing/queuehandling

commit ed635fa0c20e150673709c04ecc7f285d12e0ce2
Author: Zach Copley <zach@status.net>
Date:   Tue Jul 5 15:29:35 2011 -0700

    Move some stuff around; fix references

commit cbc553a147941cad16e205a6b66ab4b32a5e3d3d
Author: Zach Copley <zach@status.net>
Date:   Wed Jul 13 12:46:05 2011 -0700

    Fix conflicts

commit 5d77c81f75b57f5d5357d6b46d503650a4b3225d
Author: Zach Copley <zach@status.net>
Date:   Thu Jun 30 19:10:38 2011 -0700

    Move OMB-specific files to OMB plugin

commit 2ed051dbce0ce9b44723b14922026849c39ed603
Author: Zach Copley <zach@status.net>
Date:   Tue Jul 5 15:29:35 2011 -0700

    Move some stuff around; fix references

commit 8809b5e35b1aacb67d70ae3e55a43003b6f591b7
Author: Zach Copley <zach@status.net>
Date:   Tue Jul 5 15:28:59 2011 -0700

    Add OMB plugin README and rm references to OMB in mail StatusNet README

commit 35ced4067c1915baca0b3e184f9533a91a951d2d
Author: Zach Copley <zach@status.net>
Date:   Thu Jun 30 23:50:09 2011 -0700

    Update paths

commit 0ee5bafbce95fc9b8db98c1e828d33d26d08bc73
Author: Zach Copley <zach@status.net>
Date:   Thu Jun 30 23:38:03 2011 -0700

    Fix define

commit e309dd22ffb9087d7fcf9180ede4f531dbd88c3c
Author: Zach Copley <zach@status.net>
Date:   Thu Jun 30 23:30:43 2011 -0700

    Basic plugin finished

commit 00f1e930f27e080b04d1e82952f7886c84e01d97
Author: Zach Copley <zach@status.net>
Date:   Thu Jun 30 19:10:38 2011 -0700

    Move OMB-specific files to OMB plugin

commit 39dcd031a79b49da0b4fe25f1594d2e406b5eb65
Author: Zach Copley <zach@status.net>
Date:   Thu Jun 30 19:10:01 2011 -0700

    Remove OMB stuff from router

commit d594d071be1ec42518dd5465db61e01e7e8ec036
Author: Zach Copley <zach@status.net>
Date:   Wed Jul 13 14:38:40 2011 -0700

    Fix paths

commit 48c1064b4b50e89cf51d2cab388f708f60601247
Author: Zach Copley <zach@status.net>
Date:   Tue Jul 12 18:17:06 2011 -0700

    Oops, I left out the ability to authorize a token in ApiStatusNetOAuthDataStore

commit 1e1168978f38c31dbf0206b3493b2b6dcbe61589
Author: Zach Copley <zach@status.net>
Date:   Wed Jul 6 19:02:08 2011 -0700

    Some odds and ends

commit ac43af2b497d8b9286c49a9469a1dff950e41650
Author: Zach Copley <zach@status.net>
Date:   Wed Jul 6 19:01:23 2011 -0700

    Remove omb stuff from queuemanager defaults

commit 2471af2f8800515a3db544b3a186a18f3e8a43af
Author: Zach Copley <zach@status.net>
Date:   Wed Jul 6 18:46:44 2011 -0700

    Add check to make sure we're not untagging an OMB profile to OMB plugin

commit df974646459ac6d5d97a40d008f1aab66f998226
Author: Zach Copley <zach@status.net>
Date:   Wed Jul 6 18:34:20 2011 -0700

    Move some more subscription stuff and peopletag checks to OMB plugin

commit 8a1427b759e791c14a7a7a22128ba05f0b4b6d12
Author: Zach Copley <zach@status.net>
Date:   Wed Jul 6 16:37:25 2011 -0700

    Move some OMB-specific unsubscribe stuff to OMB plugin

commit bd24220dbb5170af22ea0dea8a3062e6d1aeb6a2
Author: Zach Copley <zach@status.net>
Date:   Wed Jul 6 15:26:30 2011 -0700

    Finish removing libomb from core extlibs

commit 4c3c6f1fabb0f2c92635ccc5e8f38db2293f5456
Author: Zach Copley <zach@status.net>
Date:   Wed Jul 13 12:47:32 2011 -0700

    Fix more conflicts

commit db44deefd731a412685c5669c4c6fa69833de922
Author: Zach Copley <zach@status.net>
Date:   Tue Jul 5 15:49:22 2011 -0700

    Fix queuing/queuehandling

commit ea2d84d2f3d518950d3aa1956ddc8f3a25ca55f3
Author: Zach Copley <zach@status.net>
Date:   Tue Jul 5 15:29:35 2011 -0700

    Move some stuff around; fix references

commit 8ac3e010444b41bd9a78766f5e37e49dff023b45
Author: Zach Copley <zach@status.net>
Date:   Wed Jul 13 12:46:05 2011 -0700

    Fix conflicts

commit 0aad6e10e3637b3189a87b42c24c1d6de1b346bc
Author: Zach Copley <zach@status.net>
Date:   Thu Jun 30 19:10:38 2011 -0700

    Move OMB-specific files to OMB plugin

commit d982d7076c5cb28c7b8e4b1dde8d07d7e58e278f
Author: Zach Copley <zach@status.net>
Date:   Tue Jul 5 15:29:35 2011 -0700

    Move some stuff around; fix references

commit 4b9d39c93562ff4c45c37c940013e8b78197dec1
Author: Zach Copley <zach@status.net>
Date:   Tue Jul 5 15:28:59 2011 -0700

    Add OMB plugin README and rm references to OMB in mail StatusNet README

commit dab0fb6647a85e6835298496d7127a398b6b9293
Author: Zach Copley <zach@status.net>
Date:   Thu Jun 30 23:50:09 2011 -0700

    Update paths

commit 2cb73dac8ad971f1545dcf6ba57746c777e232ef
Author: Zach Copley <zach@status.net>
Date:   Thu Jun 30 23:38:03 2011 -0700

    Fix define

commit 6f226b18a059f175b1bdd3abcb8cb95eedc22ee7
Author: Zach Copley <zach@status.net>
Date:   Thu Jun 30 23:30:43 2011 -0700

    Basic plugin finished

commit 7be304beaa0f39755c3978e0b852fde768950da4
Author: Zach Copley <zach@status.net>
Date:   Thu Jun 30 19:10:38 2011 -0700

    Move OMB-specific files to OMB plugin

commit 5b30da01cfa2802d6e7a4a4a4f39b8043c54f472
Author: Zach Copley <zach@status.net>
Date:   Thu Jun 30 19:10:01 2011 -0700

    Remove OMB stuff from router

Background deletion of user accounts. Notices are deleted in chunks, then the user itself when they're all gone.
While deletion is in progress, the account is locked with the 'deleted' role, which disables all actions with rights control.

Todo:
* Pretty up the notice on the profile page about the pending delete. Show status?
* Possibly more thorough account disabling, such as disallowing all use for login and access.
* Improve error recovery; worst case is that an account gets left locked in 'deleted' state but the queue jobs have gotten dropped out. This would leave the username in use and any undeleted notices in place.

- add event hooks to profile update pings
- send Salmon pings with custom update-profile event to OStatus subscribees and groups (subscribers will see it on your next post)
- fix OStatus queues with overlong transport names, should work on DB queues now
- Ostatus_profile::notifyActivity() and ::notifyDeferred() now can take XML, Notice, or Activity for convenience

- Multiplexing queues into groups and for multiple sites.
- Sharing vs breakout configurable per site and per queue via $config['queue']['breakout']
- Detect how many times a message is redelivered, discard if it's killed too many daemons
 - count configurable with $config['queue']['max_retries']
 - can dump the items to files in $config['queue']['dead_letter_dir']

Queue daemon memory & resource leak fixes:
- avoid unnecessary reconnections to memcached server (switch persistent connections back in on second initialization, assuming it's child process)
- monkey-patch for leaky .ini loads in DB_DataObject::databaseStructure() - was leaking 200k per active switch
- applied leak fixes to Status_network as well, using intermediate base Safe_DataObject for both it and Memcache_DataObject

Misc queue fixes:
- correct handling of child processes exiting due to signal termination instead of regular exit
- shutdown instead of infinite respawn loop if we're already past the soft memory limit at startup
- Added --all option for xmppdaemon... still opens one xmpp connection per site that has xmpp active

Cache updates:
- add Cache::increment() method with native support for memcached atomic increment

Control channel for queue daemons to request graceful shutdown, restart, or update to listen to a newly added or reconfigured site.

  queuectl.php --update -s<site>
  queuectl.php --stop
  queuectl.php --restart

Default control channel is /topic/statusnet-control. For external utilities to send a site update ping direct to the queue server, connect via Stomp and send a message formatted thus:

  update:<nickname>

(Nickname here, *not* server hostname! The rest of the queues will be updated to use nicknames later.)

Note that all currently-connected queue daemons will get these notifications, including both queuedaemon.php and xmppdaemon.php. (XMPP will ignore site update requests for sites that it's not handling.)

Limitations:
* only implemented for stomp queue manager so far
* --update may not yet handle a changed server name properly
* --restart won't reload PHP code files that were already loaded at startup. Still need to stop and restart the daemons from 'outside' when updating code base.

Moved much of the writing that happens when posting a notice to a new
queuehandler, distribqueuehandler. This updates tags, groups, replies
and inboxes at queue time (or at Web time, if queues are disabled).

To make this work well, I had to break up the monolithic
Notice::blowCaches() and make cache blowing happen closer to where
data is updated.

Squashed commit of the following:

commit 5257626c62750ac4ac1db0ce2b71410c5711cfa3
Author: Evan Prodromou <evan@status.net>
Date:   Mon Jan 25 14:56:41 2010 -0500

    slightly better handling of blowing tag memory cache

commit 8a22a3cdf6ec28685da129a0313e7b2a0837c9ef
Author: Evan Prodromou <evan@status.net>
Date:   Mon Jan 25 01:42:56 2010 -0500

    change 'distribute' to 'distrib' so not too long for dbqueue

commit 7a063315b0f7fad27cb6fbd2bdd74e253af83e4f
Author: Evan Prodromou <evan@status.net>
Date:   Mon Jan 25 01:39:15 2010 -0500

    change handle_notice() to handle() in distributqueuehandler

commit 1a39ccd28b9994137d7bfd21bb4f230546938e77
Author: Evan Prodromou <evan@status.net>
Date:   Mon Jan 25 16:05:25 2010 -0500

    error with queuemanager

commit e6b3bb93f305cfd2de71a6340b8aa6fb890049b7
Author: Evan Prodromou <evan@status.net>
Date:   Mon Jan 25 01:11:34 2010 -0500

    Blow memcache at different point rather than one big function for Notice class

commit 94d557cdc016187d1d0647ae1794cd94d6fb8ac8
Author: Evan Prodromou <evan@status.net>
Date:   Mon Jan 25 00:48:44 2010 -0500

    Blow memcache at different point rather than one big function for Notice class

commit 1c781dd08c88a35dafc5c01230b4872fd6b95182
Author: Evan Prodromou <evan@status.net>
Date:   Wed Jan 20 08:54:18 2010 -0500

    move broadcasting and distributing to new queuehandler

commit da3e46d26b84e4f028f34a13fd2ee373e4c1b954
Author: Evan Prodromou <evan@status.net>
Date:   Wed Jan 20 08:53:12 2010 -0500

    Move distribution of notices to new distribute queue handler

Going to use brion's SpawningDaemon instead

This reverts commit bd72e8b9.

Queue handlers for XMPP individual & firehose output now send their XML stanzas
to another output queue instead of connecting directly to the chat server. This
lets us have as many general processing threads as we need, while all actual
XMPP input and output go through a single daemon with a single connection open.

This avoids problems with multiple connected resources:
* multiple windows shown in some chat clients (psi, gajim, kopete)
* extra load on server
* incoming message delivery forwarding issues

Database changes:
* queue_item drops 'notice_id' in favor of a 'frame' blob.
  This is based on Craig Andrews' work branch to generalize queues to take any
  object, but conservatively leaving out the serialization for now.
  Table updater (preserves any existing queued items) in db/rc3to09.sql

Code changes to watch out for:
* Queue handlers should now define a handle() method instead of handle_notice()
* QueueDaemon and XmppDaemon now share common i/o (IoMaster) and respawning
  thread management (RespawningDaemon) infrastructure.
* The polling XmppConfirmManager has been dropped, as the message is queued
  directly when saving IM settings.
* Enable $config['queue']['debug_memory'] to output current memory usage at
  each run through the event loop to watch for memory leaks

To do:
* Adapt XMPP i/o to component connection mode for multi-site support.
* XMPP input can also be broken out to a queue, which would allow the actual
  notice save etc to be handled by general queue threads.
* Make sure there are no problems with simply pushing serialized Notice objects
  to queues.
* Find a way to improve interactive performance of the database-backed queue
  handler; polling is pretty painful to XMPP.
* Possibly redo the way QueueHandlers are injected into a QueueManager. The
  grouping used to split out the XMPP output queue is a bit awkward.

Conflicts:

	scripts/xmppdaemon.php

Queue handlers for XMPP individual & firehose output now send their XML stanzas
to another output queue instead of connecting directly to the chat server. This
lets us have as many general processing threads as we need, while all actual
XMPP input and output go through a single daemon with a single connection open.

This avoids problems with multiple connected resources:
* multiple windows shown in some chat clients (psi, gajim, kopete)
* extra load on server
* incoming message delivery forwarding issues

Database changes:
* queue_item drops 'notice_id' in favor of a 'frame' blob.
  This is based on Craig Andrews' work branch to generalize queues to take any
  object, but conservatively leaving out the serialization for now.
  Table updater (preserves any existing queued items) in db/rc3to09.sql

Code changes to watch out for:
* Queue handlers should now define a handle() method instead of handle_notice()
* QueueDaemon and XmppDaemon now share common i/o (IoMaster) and respawning
  thread management (RespawningDaemon) infrastructure.
* The polling XmppConfirmManager has been dropped, as the message is queued
  directly when saving IM settings.
* Enable $config['queue']['debug_memory'] to output current memory usage at
  each run through the event loop to watch for memory leaks

To do:
* Adapt XMPP i/o to component connection mode for multi-site support.
* XMPP input can also be broken out to a queue, which would allow the actual
  notice save etc to be handled by general queue threads.
* Make sure there are no problems with simply pushing serialized Notice objects
  to queues.
* Find a way to improve interactive performance of the database-backed queue
  handler; polling is pretty painful to XMPP.
* Possibly redo the way QueueHandlers are injected into a QueueManager. The
  grouping used to split out the XMPP output queue is a bit awkward.

Queue handlers for XMPP individual & firehose output now send their XML stanzas
to another output queue instead of connecting directly to the chat server. This
lets us have as many general processing threads as we need, while all actual
XMPP input and output go through a single daemon with a single connection open.

This avoids problems with multiple connected resources:
* multiple windows shown in some chat clients (psi, gajim, kopete)
* extra load on server
* incoming message delivery forwarding issues

Database changes:
* queue_item drops 'notice_id' in favor of a 'frame' blob.
  This is based on Craig Andrews' work branch to generalize queues to take any
  object, but conservatively leaving out the serialization for now.
  Table updater (preserves any existing queued items) in db/rc3to09.sql

Code changes to watch out for:
* Queue handlers should now define a handle() method instead of handle_notice()
* QueueDaemon and XmppDaemon now share common i/o (IoMaster) and respawning
  thread management (RespawningDaemon) infrastructure.
* The polling XmppConfirmManager has been dropped, as the message is queued
  directly when saving IM settings.
* Enable $config['queue']['debug_memory'] to output current memory usage at
  each run through the event loop to watch for memory leaks

To do:
* Adapt XMPP i/o to component connection mode for multi-site support.
* XMPP input can also be broken out to a queue, which would allow the actual
  notice save etc to be handled by general queue threads.
* Make sure there are no problems with simply pushing serialized Notice objects
  to queues.
* Find a way to improve interactive performance of the database-backed queue
  handler; polling is pretty painful to XMPP.
* Possibly redo the way QueueHandlers are injected into a QueueManager. The
  grouping used to split out the XMPP output queue is a bit awkward.

Introduce IoManager::GLOBAL_SINGLE_ONLY which indicates that only one instance of this iomanager will be run, regardless of how many threads/processes and sites there are.

temporary --skip-xmpp flag on queuedaemon.php, allows to run queue daemons but skip subscription to xmpp-based queues
(still working on making these behave gracefully when server is down)

Key changes:
* Initialization code moved from common.php to StatusNet class;
  can now switch configurations during runtime.
* As a consequence, configuration files must now be idempotent...
  Be careful with constant, function or class definitions.
* Control structure for daemons/QueueManager/QueueHandler has been refactored;
  the run loop is now managed by IoMaster run via scripts/queuedaemon.php
  IoManager subclasses are woken to handle socket input or polling, and may
  cover multiple sites.
* Plugins can implement notice queue handlers more easily by registering a
  QueueHandler class; no more need to add a daemon.

The new QueueDaemon runs from scripts/queuedaemon.php:

* This replaces most of the old *handler.php scripts; they've been refactored
  to the bare handler classes.
* Spawns multiple child processes to spread load; defaults to CPU count on
  Linux and Mac OS X systems, or override with --threads=N
* When multithreaded, child processes are automatically respawned on failure.
* Threads gracefully shut down and restart when passing a soft memory limit
  (defaults to 90% of memory_limit), limiting damage from memory leaks.
* Support for UDP-based monitoring: http://www.gitorious.org/snqmon

Rough control flow diagram:
QueueDaemon -> IoMaster -> IoManager
                           QueueManager [listen or poll] -> QueueHandler
                           XmppManager [ping & keepalive]
                           XmppConfirmManager [poll updates]

Todo:

* Respawning features not currently available running single-threaded.
* When running single-site, configuration changes aren't picked up.
* New sites or config changes affecting queue subscriptions are not yet
  handled without a daemon restart.
* SNMP monitoring output to integrate with general tools (nagios, ganglia)
* Convert XMPP confirmation message sends to use stomp queue instead of polling
* Convert xmppdaemon.php to IoManager?
* Convert Twitter status, friends import polling daemons to IoManager
* Clean up some error reporting and failure modes
* May need to adjust queue priorities for best perf in backlog/flood cases

Detailed code history available in my daemon-work branch:
http://www.gitorious.org/~brion/statusnet/brion-fixes/commits/daemon-work

Perhaps it's a little precious, but I took out the switches in
util.php to determine what's supposed to be sent when, and made
a queuemanager class that will just do things when they're supposed to
be done.