1. 25 Jan, 2010 5 commits
    • Evan Prodromou's avatar
      Offload inbox updates to a queue handler to speed up posting online · e26a843c
      Evan Prodromou authored
      Moved much of the writing that happens when posting a notice to a new
      queuehandler, distribqueuehandler. This updates tags, groups, replies
      and inboxes at queue time (or at Web time, if queues are disabled).
      
      To make this work well, I had to break up the monolithic
      Notice::blowCaches() and make cache blowing happen closer to where
      data is updated.
      
      Squashed commit of the following:
      
      commit 5257626c62750ac4ac1db0ce2b71410c5711cfa3
      Author: Evan Prodromou <evan@status.net>
      Date:   Mon Jan 25 14:56:41 2010 -0500
      
          slightly better handling of blowing tag memory cache
      
      commit 8a22a3cdf6ec28685da129a0313e7b2a0837c9ef
      Author: Evan Prodromou <evan@status.net>
      Date:   Mon Jan 25 01:42:56 2010 -0500
      
          change 'distribute' to 'distrib' so not too long for dbqueue
      
      commit 7a063315b0f7fad27cb6fbd2bdd74e253af83e4f
      Author: Evan Prodromou <evan@status.net>
      Date:   Mon Jan 25 01:39:15 2010 -0500
      
          change handle_notice() to handle() in distributqueuehandler
      
      commit 1a39ccd28b9994137d7bfd21bb4f230546938e77
      Author: Evan Prodromou <evan@status.net>
      Date:   Mon Jan 25 16:05:25 2010 -0500
      
          error with queuemanager
      
      commit e6b3bb93f305cfd2de71a6340b8aa6fb890049b7
      Author: Evan Prodromou <evan@status.net>
      Date:   Mon Jan 25 01:11:34 2010 -0500
      
          Blow memcache at different point rather than one big function for Notice class
      
      commit 94d557cdc016187d1d0647ae1794cd94d6fb8ac8
      Author: Evan Prodromou <evan@status.net>
      Date:   Mon Jan 25 00:48:44 2010 -0500
      
          Blow memcache at different point rather than one big function for Notice class
      
      commit 1c781dd08c88a35dafc5c01230b4872fd6b95182
      Author: Evan Prodromou <evan@status.net>
      Date:   Wed Jan 20 08:54:18 2010 -0500
      
          move broadcasting and distributing to new queuehandler
      
      commit da3e46d26b84e4f028f34a13fd2ee373e4c1b954
      Author: Evan Prodromou <evan@status.net>
      Date:   Wed Jan 20 08:53:12 2010 -0500
      
          Move distribution of notices to new distribute queue handler
      e26a843c
    • Brion Vibber's avatar
    • Brion Vibber's avatar
      Use new StatusNetwork->serverName() to get full domain for wildcard config... · cb07fd29
      Brion Vibber authored
      Use new StatusNetwork->serverName() to get full domain for wildcard config until we rebuild queues to be based on nicknames.
      Fixes live bug with new *.status.net sites breaking queuedaemon.php
      cb07fd29
    • Evan Prodromou's avatar
      defaults for nickname and wildcard · 49d500e3
      Evan Prodromou authored
      49d500e3
    • Evan Prodromou's avatar
  2. 24 Jan, 2010 2 commits
  3. 22 Jan, 2010 4 commits
    • Brion Vibber's avatar
      a66181c9
    • Brion Vibber's avatar
      Fix for stuck queue messages: wrap processing in stomp transactions so our... · 6e4cad71
      Brion Vibber authored
      Fix for stuck queue messages: wrap processing in stomp transactions so our lack of an ACK if PHP dies actually triggers redelivery.
      
      Previously, messages once delivered would just get stuck in the queue seemingly forever if they never got ACKed.
      Note this could lead to partial duplication, for instance if the OMB or Twitter queue handlers die after 1/2 of the outgoing sends.
      
      Recommendations:
      * catch exceptions more aggressively within queue handlers (so only PHP fatal errors are likely to kill in the middle)
      * for processing that involves sending to multiple clients, consider a second queue similar to the XMPP output, eg for OMB:
       - first queue gets delivery list and builds message data, enqueueing it for each target address
       - second queue can handle each individual outgoing message (and attempt redelivery etc separately)
      
      This would also protect better against a recurring error preventing delivery in the second part, and could spread out any slow sends over multiple threads.
      6e4cad71
    • Brion Vibber's avatar
      XMPP queued output & initial retooling of DB queue manager to support non-Notice objects. · c7507e7e
      Brion Vibber authored
      Queue handlers for XMPP individual & firehose output now send their XML stanzas
      to another output queue instead of connecting directly to the chat server. This
      lets us have as many general processing threads as we need, while all actual
      XMPP input and output go through a single daemon with a single connection open.
      
      This avoids problems with multiple connected resources:
      * multiple windows shown in some chat clients (psi, gajim, kopete)
      * extra load on server
      * incoming message delivery forwarding issues
      
      Database changes:
      * queue_item drops 'notice_id' in favor of a 'frame' blob.
        This is based on Craig Andrews' work branch to generalize queues to take any
        object, but conservatively leaving out the serialization for now.
        Table updater (preserves any existing queued items) in db/rc3to09.sql
      
      Code changes to watch out for:
      * Queue handlers should now define a handle() method instead of handle_notice()
      * QueueDaemon and XmppDaemon now share common i/o (IoMaster) and respawning
        thread management (RespawningDaemon) infrastructure.
      * The polling XmppConfirmManager has been dropped, as the message is queued
        directly when saving IM settings.
      * Enable $config['queue']['debug_memory'] to output current memory usage at
        each run through the event loop to watch for memory leaks
      
      To do:
      * Adapt XMPP i/o to component connection mode for multi-site support.
      * XMPP input can also be broken out to a queue, which would allow the actual
        notice save etc to be handled by general queue threads.
      * Make sure there are no problems with simply pushing serialized Notice objects
        to queues.
      * Find a way to improve interactive performance of the database-backed queue
        handler; polling is pretty painful to XMPP.
      * Possibly redo the way QueueHandlers are injected into a QueueManager. The
        grouping used to split out the XMPP output queue is a bit awkward.
      
      Conflicts:
      
      	scripts/xmppdaemon.php
      c7507e7e
    • Evan Prodromou's avatar
  4. 21 Jan, 2010 2 commits
  5. 20 Jan, 2010 10 commits
  6. 19 Jan, 2010 4 commits
  7. 16 Jan, 2010 4 commits
  8. 15 Jan, 2010 4 commits
  9. 14 Jan, 2010 5 commits