Difference between revisions of "Workers"

From Dreamwidth Notes
Jump to: navigation, search
(First draft)
 
m (Using worker-manager and workers.conf: Fix markup)
Line 11: Line 11:
 
[http://hg.dwscoalition.org/dw-free/file/tip/bin/worker-manager bin/worker-manager] takes care of starting, stopping, and restarting workers to maintain the numbers of workers set in [http://hg.dwscoalition.org/dw-free/file/tip/etc/workers.conf etc/workers.conf] for each host it runs on. The <code>etc/workers.conf</code> syntax is pretty straightforward and described in the file itself. To start <code>bin/worker-manager</code> for normal use (daemonized), use:
 
[http://hg.dwscoalition.org/dw-free/file/tip/bin/worker-manager bin/worker-manager] takes care of starting, stopping, and restarting workers to maintain the numbers of workers set in [http://hg.dwscoalition.org/dw-free/file/tip/etc/workers.conf etc/workers.conf] for each host it runs on. The <code>etc/workers.conf</code> syntax is pretty straightforward and described in the file itself. To start <code>bin/worker-manager</code> for normal use (daemonized), use:
  
<code>$LJHOME/bin/worker-manager</code>
+
<syntaxhighlight lang="bash">$LJHOME/bin/worker-manager</syntaxhighlight>
  
 
Or, if you want it to stay in the foreground and display progress/debug messages, use:
 
Or, if you want it to stay in the foreground and display progress/debug messages, use:

Revision as of 20:30, 4 October 2011

Workers

Workers are programs that don't run under control of Apache, because they handle tasks that don't need to happen synchronously to a user's request (whether triggered by one or not), or because running them under Apache would consume excessive resources.

Some workers process TheSchwartz or Gearman requests. Those normally run under control of worker-manager (see below). Some workers are manual, and depending on their purpose, can run either under control of worker-manager or be started by cron, or even just once when {re)starting the server. (Manual workers are a mixed lot, and there's no "one size fits all" option for them.)

Starting and stopping workers

Using worker-manager and workers.conf

bin/worker-manager takes care of starting, stopping, and restarting workers to maintain the numbers of workers set in etc/workers.conf for each host it runs on. The etc/workers.conf syntax is pretty straightforward and described in the file itself. To start bin/worker-manager for normal use (daemonized), use:

$LJHOME/bin/worker-manager

Or, if you want it to stay in the foreground and display progress/debug messages, use:

$LJHOME/bin/worker-manager --debug

In both cases, killing it will also kill the workers it started.

Individually

You can also start individual workers from the shell prompt, by typing:

$LJHOME/bin/worker/(worker-name)

Most workers accept the -v or --verbose flag to request extra debugging output and stay in the foreground when started from the shell prompt, so ^C should work to kill them.

Using cron or at server boot time

Please refer to your OS documentation. The crontab(1) and init(8) manpages are probably most relevant.

List of workers

Note: I probably got some of that wrong. Any corrections welcome, especially where I say "not sure".

Workers in alphabetical order of filename
Filename Type Category Function and notes
worker/birthday-notify Manual Fire LJ::Event::Birthday ESN events for users whose birthdays are soon
bin/worker/content-importer TheSchwartz Importer Import all content
bin/worker/content-importer-lite TheSchwartz Importer Import some content (everything except entries and comments)
bin/worker/content-importer-verify TheSchwartz Importer Verify import password
bin/worker/directory-meta Manual Directory/user search Update data used in searching for users
bin/worker/distribute-invites TheSchwartz Invite codes Email distributed invite codes
esn-cluster-subs TheSchwartz ESN Notification delivery stage 2
esn-cluster-subs-mass TheSchwartz ESN Notification delivery stage 2 (mass)
esn-filter-subs TheSchwartz ESN Notification delivery stage 3
esn-filter-subs-mass TheSchwartz ESN Notification delivery stage 3 (mass)
esn-fired-event TheSchwartz ESN Notification delivery stage 1
esn-fired-event-mass TheSchwartz ESN Notification delivery stage 1 (mass)
esn-process-sub TheSchwartz ESN Notification delivery stage 4
esn-process-sub-mass TheSchwartz ESN Notification delivery stage 4 (mass)
bin/worker/expunge-users Manual Account deletion Expunge (permanently) accounts deleted more than 60 days ago
bin/worker/import-scheduler Manual Importer Queue importing jobs in the right order - oneshot as of 20110925, but see bug 1491
bin/worker/incoming-email TheSchwartz Incoming email Handles email posting, email support requests, email support follow-up
bin/worker/latest-feed TheSchwartz Latest things Heavy lifting for the Latest Things page
bin/worker/lazy-cleanup TheSchwartz Journal management Postpone some of the work of entry deletion
bin/worker/load-friends-gm Gearman Old (LJ) friends system Not used anymore, see bug 3971
bin/worker/paidstatus Manual Payments Process paid carts, expired paid accounts, and paid accounts expiring soon
bin/worker/ping-hubbub TheSchwartz Outbound syndication Notify PubSubHubbub of DW journal updates
process-esn TheSchwartz ESN Notification delivery stages 1-4 combined
process-esn-mass TheSchwartz ESN Notification delivery stage 1-4 combined (mass)
bin/worker/process-eventlogrecord TheSchwartz Disabled Not used currently, see bug 3963
bin/worker/process-privacy TheSchwartz Entry privacy Process mass entry privacy changes
bin/worker/resolve-extacct Gearman External accounts Determine the account type (personal/community/syndicated)
bin/worker/schedule-synsuck Manual Syndication Queue TheSchwartz jobs for updating syndicated accounts
bin/worker/search-constraints Gearman Directory/user search Something to do with partial search results? Not sure
[1] Gearman Directory/user search Initiate search? Not sure
bin/worker/search-updater Manual Directory/user search Not sure what it does, still used but has some bitrot. See bug 3968
bin/worker/send-email TheSchwartz Sending email Send email to users
bin/worker/send-email-mass TheSchwartz Sending email Send email to users (mass)
shop-creditcard-charge Gearman Payments Charge a user's credit card
bin/worker/sphinx-copier TheSchwartz Full-text search Keep the full-text search database in sync with actual entries (see bug 3966)
bin/worker/sphinx-search-gm Gearman Full-text search Interface to the core sphinx search - bin/worker/stats-collection Manual Site stats Collect/compile statistics - bin/worker/subscribe-hubbub Manual Syndication Subscribe to hubbub for syndicated accounts so DW is notified of updates to feed sources
bin/worker/support-notify TheSchwartz Support board Email users about support answers they received or support actions they subscribed to
bin/worker/synsuck TheSchwartz Syndication Updates syndicated account from feed source, started by bin/worker/schedule-synsuck or through the hubbub system
bin/worker/sysban-gm Gearman Site administration Retrieve sysban entries by type
bin/worker/t-memlimit Manual Testing Memory stress test
bin/worker/taglib-gm Gearman Journal content Retrieve user's tags
bin/worker/talklib-gm Gearman Journal content Update entry comment count, needed... not sure
bin/worker/userpic-resize-gm Gearman Userpics Resize userpics larger than maximum height or width
bin/worker/xpost TheSchwartz Crossposting Crossposts new or edited entries