Difference between revisions of "Workers"

From Dreamwidth Notes
Jump to: navigation, search
(First draft)
 
(→‎List of workers: dw-send-email VERY ROUGH)
 
(25 intermediate revisions by 9 users not shown)
Line 1: Line 1:
−
Workers
+
Workers are programs that don't run under control of Apache. They are processes which happen asynchronously, either because they use a lot of resources (e.g. search), or because they need to contact external sites (e.g. importer).
  
−
Workers are programs that don't run under control of Apache, because they handle tasks that don't need to happen synchronously to a user's request (whether triggered by one or not), or because running them under Apache would consume excessive resources.
+
Some workers process [[TheSchwartz]] or [[Gearman]] requests. Those normally run under control of worker-manager (see below). Some workers are manual, and depending on their purpose, can run either under control of worker-manager or be started by cron, or even just once when (re)starting the server. (Manual workers are a mixed lot, and there's no "one size fits all" option for them.)
  
−
Some workers process [[TheSchwartz]] or [[Gearman]] requests. Those normally run under control of worker-manager (see below). Some workers are manual, and depending on their purpose, can run either under control of worker-manager or be started by cron, or even just once when {re)starting the server. (Manual workers are a mixed lot, and there's no "one size fits all" option for them.)
+
'''Before using workers that rely on [[TheSchwartz]] or [[Gearman]], you will need to make sure that these services are [http://www.livejournal.com/doc/server/lj.install.workers_setup_install.html installed] and [http://www.livejournal.com/doc/server/lj.install.workers_setup_config.html configured] on your server.'''
  
−
== Starting and stopping workers ==
+
= Starting and stopping workers =
  
−
=== Using worker-manager and workers.conf ===
+
== Individually ==
  
−
[http://hg.dwscoalition.org/dw-free/file/tip/bin/worker-manager bin/worker-manager] takes care of starting, stopping, and restarting workers to maintain the numbers of workers set in [http://hg.dwscoalition.org/dw-free/file/tip/etc/workers.conf etc/workers.conf] for each host it runs on. The <code>etc/workers.conf</code> syntax is pretty straightforward and described in the file itself. To start <code>bin/worker-manager</code> for normal use (daemonized), use:
+
When developing, you can start and debug individual workers from the shell prompt, by typing:
  
−
<code>$LJHOME/bin/worker-manager</code>
+
<syntaxhighlight lang="bash">$LJHOME/bin/worker/(worker-name) --verbose</syntaxhighlight>
  
−
Or, if you want it to stay in the foreground and display progress/debug messages, use:
+
The <code>--verbose</code> or <code>-v</code> flag makes sure that the worker stays in the foreground and provides extra debugging text. To stop the worker when you're done, use ^C.
  
−
<syntaxhighlight lang="bash">$LJHOME/bin/worker-manager --debug</syntaxhighlight>
+
If you run the worker without <code>--verbose</code> flag, it will continue running in the background even after you're done and have logged out. You probably don't want to do this! If you want run multiple workers at the same time, or have them keep running, use the worker-manager described below.
  
−
In both cases, killing it will also kill the workers it started.
+
Most of the time, if you are developing a worker, you will have a separate shell where you run the worker and can see its error messages as you work. Remember that every time you edit the code run by the worker (whether it is directly in the worker or just the libraries the worker uses) you need to kill and restart the worker process itself, or the worker will not see your changes.
  
−
=== Individually ===
 
  
−
You can also start individual workers from the shell prompt, by typing:
+
== Using worker-manager and workers.conf ==
  
−
<syntaxhighlight lang="bash">$LJHOME/bin/worker/(worker-name)</syntaxhighlight>
+
[https://github.com/dreamwidth/dw-free/blob/develop/bin/worker-manager bin/worker-manager] takes care of starting, stopping, and restarting workers to maintain the numbers of workers set in [https://github.com/dreamwidth/dw-free/blob/develop/etc/workers.conf etc/workers.conf] for each host it runs on. The <code>etc/workers.conf</code> syntax is pretty straightforward and described in the file itself. To start <code>bin/worker-manager</code> for normal use (daemonized), use:
  
−
Most workers accept the <code>-v</code> or <code>--verbose</code> flag to request extra debugging output and stay in the foreground when started from the shell prompt, so ^C should work to kill them.
+
<syntaxhighlight lang="bash">$LJHOME/bin/worker-manager</syntaxhighlight>
  
−
=== Using cron or at server boot time ===
+
Or, if you want it to stay in the foreground and display progress/debug messages, use:
 +
 
 +
<syntaxhighlight lang="bash">$LJHOME/bin/worker-manager --debug</syntaxhighlight>
 +
 
 +
In both cases, killing it will also kill the workers it started.
 +
 
 +
If you find that you do not want to use some workers (for testing, temporarily, or long-term), you may turn them off in the configuration. In <code>/etc/workers.conf</code>, setting the count to 0 will not start any of that worker. Commenting out the line on which that worker appears (with a #) will not start any of that worker. These are best for temporary changes. For a permanent change, you may also entirely remove the line on which that worker appears .
 +
 
 +
== Using cron or at server boot time ==
  
 
Please refer to your OS documentation. The crontab(1) and init(8) manpages are probably most relevant.
 
Please refer to your OS documentation. The crontab(1) and init(8) manpages are probably most relevant.
  
−
== List of workers ==
+
= List of workers =
  
−
Note: I probably got some of that wrong. Any corrections welcome, especially where I say "not sure".
+
Below is a list of all the workers in the <code>$LJHOME/bin/worker</code> directory.
  
 
{| border="1"
 
{| border="1"
Line 39: Line 46:
 
! Filename !! Type !! Category !! Function and notes
 
! Filename !! Type !! Category !! Function and notes
 
|-
 
|-
−
! [http://hg.dwscoalition.org/dw-free/file/tip/bin/worker/birthday-notify worker/birthday-notify]
+
! [https://github.com/dreamwidth/dw-free/blob/develop/bin/worker/birthday-notify birthday-notify]
−
| Manual || Fire LJ::Event::Birthday [[ESN#Events|ESN events]] for users whose birthdays are soon
+
| Manual || [[ESN]] || Fire LJ::Event::Birthday [[ESN#Events|ESN events]] for users whose birthdays are soon
 
|-
 
|-
−
! [http://hg.dwscoalition.org/dw-free/file/tip/bin/worker/content-importer bin/worker/content-importer]
+
! [https://github.com/dreamwidth/dw-free/blob/develop/bin/worker/change-poster-id change-poster-id]
 +
| [[TheSchwartz]] || Journal content || Does the backend work associated with claiming an OpenID account
 +
|-
 +
! [https://github.com/dreamwidth/dw-free/blob/develop/bin/worker/content-importer content-importer]
 
| [[TheSchwartz]] || [[Importer]] || Import all content
 
| [[TheSchwartz]] || [[Importer]] || Import all content
 
|-
 
|-
−
! [http://hg.dwscoalition.org/dw-free/file/tip/bin/worker/content-importer-lite bin/worker/content-importer-lite]
+
! [https://github.com/dreamwidth/dw-free/blob/develop/bin/worker/content-importer-lite content-importer-lite]
 
| [[TheSchwartz]] || [[Importer]] || Import some content (everything except entries and comments)
 
| [[TheSchwartz]] || [[Importer]] || Import some content (everything except entries and comments)
 
|-
 
|-
−
! [http://hg.dwscoalition.org/dw-free/file/tip/bin/worker/content-importer-verify bin/worker/content-importer-verify]
+
! [https://github.com/dreamwidth/dw-free/blob/develop/bin/worker/content-importer-verify content-importer-verify]
 
| [[TheSchwartz]] || [[Importer]] || Verify import password
 
| [[TheSchwartz]] || [[Importer]] || Verify import password
 
|-
 
|-
−
! [http://hg.dwscoalition.org/dw-free/file/tip/bin/worker/directory-meta bin/worker/directory-meta]
+
! [https://github.com/dreamwidth/dw-free/blob/develop/bin/worker/directory-meta directory-meta]
−
| Manual || Directory/user search || Update data used in searching for users
+
| Manual || Directory/user search || Update packed user data in mysql, used by search-updater to update the binary data file
 
|-
 
|-
−
! [http://hg.dwscoalition.org/dw-free/file/tip/bin/worker/distribute-invites bin/worker/distribute-invites]
+
! [https://github.com/dreamwidth/dw-free/blob/develop/bin/worker/distribute-invites distribute-invites]
 
| [[TheSchwartz]] || [[Invite code generation#Multiple_user_grants|Invite codes]] || Email distributed invite codes
 
| [[TheSchwartz]] || [[Invite code generation#Multiple_user_grants|Invite codes]] || Email distributed invite codes
 
|-
 
|-
−
! [http://hg.dwscoalition.org/dw-free/file/tip/bin/worker/esn-cluster-subs esn-cluster-subs]
+
! dw-send-email
 +
| ? || emails || or your email won't work
 +
|-
 +
! [https://github.com/dreamwidth/dw-free/blob/develop/bin/worker/embeds embeds]
 +
| [[TheSchwartz]] || Embed codes || Grab titles from embed APIs
 +
|-
 +
! [https://github.com/dreamwidth/dw-free/blob/develop/bin/worker/esn-cluster-subs esn-cluster-subs]
 
| [[TheSchwartz]] || [[ESN]] || [[ESN#Firing_events_and_delivering_notifications|Notification delivery]] stage 2
 
| [[TheSchwartz]] || [[ESN]] || [[ESN#Firing_events_and_delivering_notifications|Notification delivery]] stage 2
 
|-
 
|-
−
! [http://hg.dwscoalition.org/dw-free/file/tip/bin/worker/esn-cluster-subs-mass esn-cluster-subs-mass]
+
! [https://github.com/dreamwidth/dw-free/blob/develop/bin/worker/esn-cluster-subs-mass esn-cluster-subs-mass]
 
| [[TheSchwartz]] || [[ESN]] || [[ESN#Firing_events_and_delivering_notifications|Notification delivery]] stage 2 (mass)
 
| [[TheSchwartz]] || [[ESN]] || [[ESN#Firing_events_and_delivering_notifications|Notification delivery]] stage 2 (mass)
 
|-
 
|-
−
! [http://hg.dwscoalition.org/dw-free/file/tip/bin/worker/esn-filter-subs esn-filter-subs]
+
! [https://github.com/dreamwidth/dw-free/blob/develop/bin/worker/esn-filter-subs esn-filter-subs]
 
| [[TheSchwartz]] || [[ESN]] || [[ESN#Firing_events_and_delivering_notifications|Notification delivery]] stage 3
 
| [[TheSchwartz]] || [[ESN]] || [[ESN#Firing_events_and_delivering_notifications|Notification delivery]] stage 3
 
|-
 
|-
−
! [http://hg.dwscoalition.org/dw-free/file/tip/bin/worker/esn-filter-subs-mass esn-filter-subs-mass]
+
! [https://github.com/dreamwidth/dw-free/blob/develop/bin/worker/esn-filter-subs-mass esn-filter-subs-mass]
 
| [[TheSchwartz]] || [[ESN]] || [[ESN#Firing_events_and_delivering_notifications|Notification delivery]] stage 3 (mass)
 
| [[TheSchwartz]] || [[ESN]] || [[ESN#Firing_events_and_delivering_notifications|Notification delivery]] stage 3 (mass)
 
|-
 
|-
−
! [http://hg.dwscoalition.org/dw-free/file/tip/bin/worker/esn-fired-event esn-fired-event]
+
! [https://github.com/dreamwidth/dw-free/blob/develop/bin/worker/esn-fired-event esn-fired-event]
 
| [[TheSchwartz]] || [[ESN]] || [[ESN#Firing_events_and_delivering_notifications|Notification delivery]] stage 1
 
| [[TheSchwartz]] || [[ESN]] || [[ESN#Firing_events_and_delivering_notifications|Notification delivery]] stage 1
 
|-
 
|-
−
! [http://hg.dwscoalition.org/dw-free/file/tip/bin/worker/esn-fired-event-mass esn-fired-event-mass]
+
! [https://github.com/dreamwidth/dw-free/blob/develop/bin/worker/esn-fired-event-mass esn-fired-event-mass]
 
| [[TheSchwartz]] || [[ESN]] || [[ESN#Firing_events_and_delivering_notifications|Notification delivery]] stage 1 (mass)
 
| [[TheSchwartz]] || [[ESN]] || [[ESN#Firing_events_and_delivering_notifications|Notification delivery]] stage 1 (mass)
 
|-
 
|-
−
! [http://hg.dwscoalition.org/dw-free/file/tip/bin/worker/esn-process-sub esn-process-sub]
+
! [https://github.com/dreamwidth/dw-free/blob/develop/bin/worker/esn-process-sub esn-process-sub]
 
| [[TheSchwartz]] || [[ESN]] || [[ESN#Firing_events_and_delivering_notifications|Notification delivery]] stage 4
 
| [[TheSchwartz]] || [[ESN]] || [[ESN#Firing_events_and_delivering_notifications|Notification delivery]] stage 4
 
|-
 
|-
−
! [http://hg.dwscoalition.org/dw-free/file/tip/bin/worker/esn-process-sub-mass esn-process-sub-mass]
+
! [https://github.com/dreamwidth/dw-free/blob/develop/bin/worker/esn-process-sub-mass esn-process-sub-mass]
 
| [[TheSchwartz]] || [[ESN]] || [[ESN#Firing_events_and_delivering_notifications|Notification delivery]] stage 4 (mass)
 
| [[TheSchwartz]] || [[ESN]] || [[ESN#Firing_events_and_delivering_notifications|Notification delivery]] stage 4 (mass)
 
|-
 
|-
−
! [http://hg.dwscoalition.org/dw-free/file/tip/bin/worker/expunge-users bin/worker/expunge-users]
+
! [https://github.com/dreamwidth/dw-free/blob/develop/bin/worker/expunge-users expunge-users]
−
| Manual || Account deletion || Expunge (permanently) accounts deleted more than 60 days ago
+
| Manual || [[Purging deleted users|Account deletion]] || Expunge (permanently) accounts deleted more than 60 days ago
 
|-
 
|-
−
! [http://hg.dwscoalition.org/dw-free/file/tip/bin/worker/import-scheduler bin/worker/import-scheduler]
+
! [https://github.com/dreamwidth/dw-free/blob/develop/bin/worker/import-eraser import-eraser]
−
| Manual || [[Importer]] || Queue importing jobs in the right order - oneshot as of 20110925, but see [http://bugs.dwscoalition.org/show_bug.cgi?id=1491 bug 1491]
+
| [[TheSchwartz]] || [[Importer]] || Erases all imported content from a specified journal
 
|-
 
|-
−
! [http://hg.dwscoalition.org/dw-free/file/tip/bin/worker/incoming-email bin/worker/incoming-email]
+
! [https://github.com/dreamwidth/dw-free/blob/develop/bin/worker/import-scheduler import-scheduler]
 +
| Manual || [[Importer]] || Queue importing jobs in the right order
 +
|-
 +
! [https://github.com/dreamwidth/dw-free/blob/develop/bin/worker/incoming-email incoming-email]
 
| [[TheSchwartz]] || Incoming email || Handles email posting, email support requests, email support follow-up
 
| [[TheSchwartz]] || Incoming email || Handles email posting, email support requests, email support follow-up
 
|-
 
|-
−
! [http://hg.dwscoalition.org/dw-free/file/tip/bin/worker/latest-feed bin/worker/latest-feed]
+
! [https://github.com/dreamwidth/dw-free/blob/develop/bin/worker/latest-feed latest-feed]
 
| [[TheSchwartz]] || Latest things || Heavy lifting for the Latest Things page
 
| [[TheSchwartz]] || Latest things || Heavy lifting for the Latest Things page
 
|-
 
|-
−
! [http://hg.dwscoalition.org/dw-free/file/tip/bin/worker/lazy-cleanup bin/worker/lazy-cleanup]
+
! [https://github.com/dreamwidth/dw-free/blob/develop/bin/worker/lazy-cleanup lazy-cleanup]
 
| [[TheSchwartz]] || Journal management || Postpone some of the work of entry deletion
 
| [[TheSchwartz]] || Journal management || Postpone some of the work of entry deletion
 
|-
 
|-
−
! [http://hg.dwscoalition.org/dw-free/file/tip/bin/worker/load-friends-gm bin/worker/load-friends-gm]
+
! [https://github.com/dreamwidth/dw-free/blob/develop/bin/worker/paidstatus paidstatus]
−
| [[Gearman]] || Old (LJ) friends system || Not used anymore, see [http://bugs.dwscoalition.org/show_bug.cgi?id=3971 bug 3971]
+
−
|-
+
−
! [http://hg.dwscoalition.org/dw-free/file/tip/bin/worker/paidstatus bin/worker/paidstatus]
+
 
| Manual || Payments || Process paid carts, expired paid accounts, and paid accounts expiring soon
 
| Manual || Payments || Process paid carts, expired paid accounts, and paid accounts expiring soon
 
|-
 
|-
−
! [http://hg.dwscoalition.org/dw-free/file/tip/bin/worker/ping-hubbub bin/worker/ping-hubbub]
+
! [https://github.com/dreamwidth/dw-free/blob/develop/bin/worker/process-esn process-esn]
−
| [[TheSchwartz]] || Outbound syndication || Notify PubSubHubbub of DW journal updates
+
−
|-
+
−
! [http://hg.dwscoalition.org/dw-free/file/tip/bin/worker/process-esn process-esn]
+
 
| [[TheSchwartz]] || [[ESN]] || [[ESN#Firing_events_and_delivering_notifications|Notification delivery]] stages 1-4 combined
 
| [[TheSchwartz]] || [[ESN]] || [[ESN#Firing_events_and_delivering_notifications|Notification delivery]] stages 1-4 combined
 
|-
 
|-
−
! [http://hg.dwscoalition.org/dw-free/file/tip/bin/worker/process-esn-mass process-esn-mass]
+
! [https://github.com/dreamwidth/dw-free/blob/develop/bin/worker/process-esn-mass process-esn-mass]
 
| [[TheSchwartz]] || [[ESN]] || [[ESN#Firing_events_and_delivering_notifications|Notification delivery]] stage 1-4 combined (mass)
 
| [[TheSchwartz]] || [[ESN]] || [[ESN#Firing_events_and_delivering_notifications|Notification delivery]] stage 1-4 combined (mass)
 
|-
 
|-
−
! [http://hg.dwscoalition.org/dw-free/file/tip/bin/worker/process-eventlogrecord bin/worker/process-eventlogrecord]
+
! [https://github.com/dreamwidth/dw-free/blob/develop/bin/worker/process-eventlogrecord process-eventlogrecord]
−
| [[TheSchwartz]] || Disabled || Not used currently, see [http://bugs.dwscoalition.org/show_bug.cgi?id=3963 bug 3963]
+
| [[TheSchwartz]] || Disabled || Not used currently, may want to get rid of it along with LJ::EventLogRecord, LJ::EventLogRecord::*
 
|-
 
|-
−
! [http://hg.dwscoalition.org/dw-free/file/tip/bin/worker/process-privacy bin/worker/process-privacy]
+
! [https://github.com/dreamwidth/dw-free/blob/develop/bin/worker/process-privacy process-privacy]
 
| [[TheSchwartz]] || Entry privacy || Process mass entry privacy changes
 
| [[TheSchwartz]] || Entry privacy || Process mass entry privacy changes
 
|-
 
|-
−
! [http://hg.dwscoalition.org/dw-free/file/tip/bin/worker/resolve-extacct bin/worker/resolve-extacct]
+
! [https://github.com/dreamwidth/dw-free/blob/develop/bin/worker/resolve-extacct resolve-extacct]
 
| [[Gearman]] || External accounts || Determine the account type (personal/community/syndicated)
 
| [[Gearman]] || External accounts || Determine the account type (personal/community/syndicated)
 
|-
 
|-
−
! [http://hg.dwscoalition.org/dw-free/file/tip/bin/worker/schedule-synsuck bin/worker/schedule-synsuck]
+
! [https://github.com/dreamwidth/dw-free/blob/develop/bin/worker/schedule-synsuck schedule-synsuck]
 
| Manual || Syndication || Queue TheSchwartz jobs for updating syndicated accounts
 
| Manual || Syndication || Queue TheSchwartz jobs for updating syndicated accounts
 
|-
 
|-
−
! [http://hg.dwscoalition.org/dw-free/file/tip/bin/worker/search-constraints bin/worker/search-constraints]
+
! [https://github.com/dreamwidth/dw-free/blob/develop/bin/worker/search-constraints search-constraints]
−
| [[Gearman]] || Directory/user search || Something to do with partial search results? Not sure
+
| [[Gearman]] || Directory/user search || Convert a single constraint (e.g., trusted by x) to a set handle, using cached results when possible
 
|-
 
|-
−
! [http://hg.dwscoalition.org/dw-free/file/tip/bin/worker/search-lookup]
+
! [https://github.com/dreamwidth/dw-free/blob/develop/bin/worker/search-lookup search-lookup]
−
| [[Gearman]] || Directory/user search || Initiate search? Not sure
+
| [[Gearman]] || Directory/user search || Initiate search, filtering with each constraint's set handle, returning the intersection; reads the binary data file once at startup so it should be restarted often
 
|-
 
|-
−
! [http://hg.dwscoalition.org/dw-free/file/tip/bin/worker/search-updater bin/worker/search-updater]
+
! [https://github.com/dreamwidth/dw-free/blob/develop/bin/worker/search-updater search-updater]
−
| Manual || Directory/user search || Not sure what it does, still used but has some bitrot. See [http://bugs.dwscoalition.org/show_bug.cgi?id=3968 bug 3968]
+
| Manual || Directory/user search || Copy packed user data from mysql to the binary data file; has some bitrot: refers to bin/ljworkerctl which is ljcomint so it can't actually restart search-lookup
 
|-
 
|-
−
! [http://hg.dwscoalition.org/dw-free/file/tip/bin/worker/send-email bin/worker/send-email]
+
! [https://github.com/dreamwidth/dw-free/blob/develop/bin/worker/send-email send-email]
 
| [[TheSchwartz]] || [[TheSchwartz#Sending_non-ESN_email_to_a_user|Sending email]] || Send email to users
 
| [[TheSchwartz]] || [[TheSchwartz#Sending_non-ESN_email_to_a_user|Sending email]] || Send email to users
 
|-
 
|-
−
! [http://hg.dwscoalition.org/dw-free/file/tip/bin/worker/send-email-mass bin/worker/send-email-mass]
+
! [https://github.com/dreamwidth/dw-free/blob/develop/bin/worker/send-email-mass send-email-mass]
 
| [[TheSchwartz]] || [[TheSchwartz#Sending_non-ESN_email_to_a_user|Sending email]] || Send email to users (mass)
 
| [[TheSchwartz]] || [[TheSchwartz#Sending_non-ESN_email_to_a_user|Sending email]] || Send email to users (mass)
 
|-
 
|-
−
! [http://hg.dwscoalition.org/dw-free/file/tip/bin/worker/shop-creditcard-charge shop-creditcard-charge]
+
! [https://github.com/dreamwidth/dw-free/blob/develop/bin/worker/send-email-ses send-email-ses]
 +
| [[TheSchwartz]] || [[TheSchwartz#Sending_non-ESN_email_to_a_user|Sending email]] || Send email to users through Amazon SES
 +
|-
 +
! [https://github.com/dreamwidth/dw-free/blob/develop/bin/worker/shop-creditcard-charge shop-creditcard-charge]
 
| [[Gearman]] || Payments || Charge a user's credit card
 
| [[Gearman]] || Payments || Charge a user's credit card
 
|-
 
|-
−
! [http://hg.dwscoalition.org/dw-free/file/tip/bin/worker/sphinx-copier bin/worker/sphinx-copier]
+
! [https://github.com/dreamwidth/dw-free/blob/develop/bin/worker/spellcheck-gm spellcheck-gm]
−
| [[TheSchwartz]] || Full-text search || Keep the full-text search database in sync with actual entries (see [http://bugs.dwscoalition.org/show_bug.cgi?id=3966 bug 3966])
+
| [[Gearman]] || Journal content || Gearman worker for running spellcheck on requested text
 +
|-
 +
! [https://github.com/dreamwidth/dw-free/blob/develop/bin/worker/sphinx-copier sphinx-copier]
 +
| [[TheSchwartz]] || Full-text search || Keep the full-text search database in sync with actual entries
 +
|-
 +
! [https://github.com/dreamwidth/dw-free/blob/develop/bin/worker/sphinx-search-gm sphinx-search-gm]
 +
| [[Gearman]] || Full-text search || Interface to the core sphinx search
 
|-
 
|-
−
! [http://hg.dwscoalition.org/dw-free/file/tip/bin/worker/sphinx-search-gm bin/worker/sphinx-search-gm]
+
! [https://github.com/dreamwidth/dw-free/blob/develop/bin/worker/stats-collection stats-collection]
−
| [[Gearman]] || Full-text search || Interface to the core sphinx search
+
−
!-
+
−
! [http://hg.dwscoalition.org/dw-free/file/tip/bin/worker/stats-collection bin/worker/stats-collection]
+
 
| Manual || Site stats || Collect/compile statistics
 
| Manual || Site stats || Collect/compile statistics
−
!-
 
−
! [http://hg.dwscoalition.org/dw-free/file/tip/bin/worker/subscribe-hubbub bin/worker/subscribe-hubbub]
 
−
| Manual || Syndication || Subscribe to hubbub for syndicated accounts so DW is notified of updates to feed sources
 
 
|-
 
|-
−
! [http://hg.dwscoalition.org/dw-free/file/tip/bin/worker/support-notify bin/worker/support-notify]
+
! [https://github.com/dreamwidth/dw-free/blob/develop/bin/worker/support-notify support-notify]
 
| [[TheSchwartz]] || Support board || Email users about support answers they received or support actions they subscribed to
 
| [[TheSchwartz]] || Support board || Email users about support answers they received or support actions they subscribed to
 
|-
 
|-
−
! [http://hg.dwscoalition.org/dw-free/file/tip/bin/worker/synsuck bin/worker/synsuck]
+
! [https://github.com/dreamwidth/dw-free/blob/develop/bin/worker/synsuck synsuck]
−
| [[TheSchwartz]] || Syndication || Updates syndicated account from feed source, started by [http://hg.dwscoalition.org/dw-free/file/tip/bin/worker/schedule-synsuck bin/worker/schedule-synsuck] or through the hubbub system
+
| [[TheSchwartz]] || Syndication || Updates syndicated account from feed source, started by [https://github.com/dreamwidth/dw-free/blob/develop/bin/worker/schedule-synsuck bin/worker/schedule-synsuck] or through the hubbub system
 
|-
 
|-
−
! [http://hg.dwscoalition.org/dw-free/file/tip/bin/worker/sysban-gm bin/worker/sysban-gm]
+
! [https://github.com/dreamwidth/dw-free/blob/develop/bin/worker/sysban-gm sysban-gm]
 
| [[Gearman]] || Site administration || Retrieve sysban entries by type
 
| [[Gearman]] || Site administration || Retrieve sysban entries by type
 
|-
 
|-
−
! [http://hg.dwscoalition.org/dw-free/file/tip/bin/worker/t-memlimit bin/worker/t-memlimit]
+
! [https://github.com/dreamwidth/dw-free/blob/develop/bin/worker/t-memlimit t-memlimit]
 
| Manual || Testing || Memory stress test
 
| Manual || Testing || Memory stress test
 
|-
 
|-
−
! [http://hg.dwscoalition.org/dw-free/file/tip/bin/worker/taglib-gm bin/worker/taglib-gm]
+
! [https://github.com/dreamwidth/dw-free/blob/develop/bin/worker/taglib-gm taglib-gm]
 
| [[Gearman]] || Journal content || Retrieve user's tags
 
| [[Gearman]] || Journal content || Retrieve user's tags
 
|-
 
|-
−
! [http://hg.dwscoalition.org/dw-free/file/tip/bin/worker/talklib-gm bin/worker/talklib-gm]
+
! [https://github.com/dreamwidth/dw-free/blob/develop/bin/worker/talklib-gm talklib-gm]
−
| [[Gearman]] || Journal content || Update entry comment count, needed... not sure  
+
| [[Gearman]] || Journal content || Update entry comment count, needed... not sure
 
|-
 
|-
−
! [http://hg.dwscoalition.org/dw-free/file/tip/bin/worker/userpic-resize-gm bin/worker/userpic-resize-gm]
+
! [https://github.com/dreamwidth/dw-free/blob/develop/bin/worker/userpic-resize-gm userpic-resize-gm]
 
| [[Gearman]] || Userpics || Resize userpics larger than maximum height or width
 
| [[Gearman]] || Userpics || Resize userpics larger than maximum height or width
 
|-
 
|-
−
! [http://hg.dwscoalition.org/dw-free/file/tip/bin/worker/xpost bin/worker/xpost]
+
! [https://github.com/dreamwidth/dw-free/blob/develop/bin/worker/xpost xpost]
 
| [[TheSchwartz]] || Crossposting || Crossposts new or edited entries
 
| [[TheSchwartz]] || Crossposting || Crossposts new or edited entries
 
|}
 
|}
  
−
[[Category: Installation]]
+
[[Category: Dreamwidth Installation]]
 +
[[Category: Development]]

Latest revision as of 00:26, 30 January 2024

Workers are programs that don't run under control of Apache. They are processes which happen asynchronously, either because they use a lot of resources (e.g. search), or because they need to contact external sites (e.g. importer).

Some workers process TheSchwartz or Gearman requests. Those normally run under control of worker-manager (see below). Some workers are manual, and depending on their purpose, can run either under control of worker-manager or be started by cron, or even just once when (re)starting the server. (Manual workers are a mixed lot, and there's no "one size fits all" option for them.)

Before using workers that rely on TheSchwartz or Gearman, you will need to make sure that these services are installed and configured on your server.

Starting and stopping workers

Individually

When developing, you can start and debug individual workers from the shell prompt, by typing: 

$LJHOME/bin/worker/(worker-name) --verbose

The --verbose or -v flag makes sure that the worker stays in the foreground and provides extra debugging text. To stop the worker when you're done, use ^C.

If you run the worker without --verbose flag, it will continue running in the background even after you're done and have logged out. You probably don't want to do this! If you want run multiple workers at the same time, or have them keep running, use the worker-manager described below.

Most of the time, if you are developing a worker, you will have a separate shell where you run the worker and can see its error messages as you work. Remember that every time you edit the code run by the worker (whether it is directly in the worker or just the libraries the worker uses) you need to kill and restart the worker process itself, or the worker will not see your changes.


Using worker-manager and workers.conf

bin/worker-manager takes care of starting, stopping, and restarting workers to maintain the numbers of workers set in etc/workers.conf for each host it runs on. The etc/workers.conf syntax is pretty straightforward and described in the file itself. To start bin/worker-manager for normal use (daemonized), use: 

$LJHOME/bin/worker-manager

Or, if you want it to stay in the foreground and display progress/debug messages, use: 

$LJHOME/bin/worker-manager --debug

In both cases, killing it will also kill the workers it started.

If you find that you do not want to use some workers (for testing, temporarily, or long-term), you may turn them off in the configuration. In /etc/workers.conf, setting the count to 0 will not start any of that worker. Commenting out the line on which that worker appears (with a #) will not start any of that worker. These are best for temporary changes. For a permanent change, you may also entirely remove the line on which that worker appears .

Using cron or at server boot time

Please refer to your OS documentation. The crontab(1) and init(8) manpages are probably most relevant.

List of workers

Below is a list of all the workers in the $LJHOME/bin/worker directory.

Workers in alphabetical order of filename
Filename Type Category Function and notes
birthday-notify Manual ESN Fire LJ::Event::Birthday ESN events for users whose birthdays are soon
change-poster-id TheSchwartz Journal content Does the backend work associated with claiming an OpenID account
content-importer TheSchwartz Importer Import all content
content-importer-lite TheSchwartz Importer Import some content (everything except entries and comments)
content-importer-verify TheSchwartz Importer Verify import password
directory-meta Manual Directory/user search Update packed user data in mysql, used by search-updater to update the binary data file
distribute-invites TheSchwartz Invite codes Email distributed invite codes
dw-send-email  ? emails or your email won't work
embeds TheSchwartz Embed codes Grab titles from embed APIs
esn-cluster-subs TheSchwartz ESN Notification delivery stage 2
esn-cluster-subs-mass TheSchwartz ESN Notification delivery stage 2 (mass)
esn-filter-subs TheSchwartz ESN Notification delivery stage 3
esn-filter-subs-mass TheSchwartz ESN Notification delivery stage 3 (mass)
esn-fired-event TheSchwartz ESN Notification delivery stage 1
esn-fired-event-mass TheSchwartz ESN Notification delivery stage 1 (mass)
esn-process-sub TheSchwartz ESN Notification delivery stage 4
esn-process-sub-mass TheSchwartz ESN Notification delivery stage 4 (mass)
expunge-users Manual Account deletion Expunge (permanently) accounts deleted more than 60 days ago
import-eraser TheSchwartz Importer Erases all imported content from a specified journal
import-scheduler Manual Importer Queue importing jobs in the right order
incoming-email TheSchwartz Incoming email Handles email posting, email support requests, email support follow-up
latest-feed TheSchwartz Latest things Heavy lifting for the Latest Things page
lazy-cleanup TheSchwartz Journal management Postpone some of the work of entry deletion
paidstatus Manual Payments Process paid carts, expired paid accounts, and paid accounts expiring soon
process-esn TheSchwartz ESN Notification delivery stages 1-4 combined
process-esn-mass TheSchwartz ESN Notification delivery stage 1-4 combined (mass)
process-eventlogrecord TheSchwartz Disabled Not used currently, may want to get rid of it along with LJ::EventLogRecord, LJ::EventLogRecord::*
process-privacy TheSchwartz Entry privacy Process mass entry privacy changes
resolve-extacct Gearman External accounts Determine the account type (personal/community/syndicated)
schedule-synsuck Manual Syndication Queue TheSchwartz jobs for updating syndicated accounts
search-constraints Gearman Directory/user search Convert a single constraint (e.g., trusted by x) to a set handle, using cached results when possible
search-lookup Gearman Directory/user search Initiate search, filtering with each constraint's set handle, returning the intersection; reads the binary data file once at startup so it should be restarted often
search-updater Manual Directory/user search Copy packed user data from mysql to the binary data file; has some bitrot: refers to bin/ljworkerctl which is ljcomint so it can't actually restart search-lookup
send-email TheSchwartz Sending email Send email to users
send-email-mass TheSchwartz Sending email Send email to users (mass)
send-email-ses TheSchwartz Sending email Send email to users through Amazon SES
shop-creditcard-charge Gearman Payments Charge a user's credit card
spellcheck-gm Gearman Journal content Gearman worker for running spellcheck on requested text
sphinx-copier TheSchwartz Full-text search Keep the full-text search database in sync with actual entries
sphinx-search-gm Gearman Full-text search Interface to the core sphinx search
stats-collection Manual Site stats Collect/compile statistics
support-notify TheSchwartz Support board Email users about support answers they received or support actions they subscribed to
synsuck TheSchwartz Syndication Updates syndicated account from feed source, started by bin/worker/schedule-synsuck or through the hubbub system
sysban-gm Gearman Site administration Retrieve sysban entries by type
t-memlimit Manual Testing Memory stress test
taglib-gm Gearman Journal content Retrieve user's tags
talklib-gm Gearman Journal content Update entry comment count, needed... not sure
userpic-resize-gm Gearman Userpics Resize userpics larger than maximum height or width
xpost TheSchwartz Crossposting Crossposts new or edited entries
Retrieved from "http://wiki.dreamwidth.net/wiki/index.php?title=Workers&oldid=13190"