Difference between revisions of "Workers"

From Dreamwidth Notes
Jump to: navigation, search
m (HTTPS everywhere; replace links to hg.dwscoalition.org)
(List of workers: dw-send-email VERY ROUGH)
 
(One intermediate revision by one other user not shown)
Line 62: Line 62:
 
|-
 
|-
 
! [https://github.com/dreamwidth/dw-free/blob/develop/bin/worker/directory-meta directory-meta]
 
! [https://github.com/dreamwidth/dw-free/blob/develop/bin/worker/directory-meta directory-meta]
| Manual || Directory/user search || Update data used in searching for users
+
| Manual || Directory/user search || Update packed user data in mysql, used by search-updater to update the binary data file
 
|-
 
|-
 
! [https://github.com/dreamwidth/dw-free/blob/develop/bin/worker/distribute-invites distribute-invites]
 
! [https://github.com/dreamwidth/dw-free/blob/develop/bin/worker/distribute-invites distribute-invites]
 
| [[TheSchwartz]] || [[Invite code generation#Multiple_user_grants|Invite codes]] || Email distributed invite codes
 
| [[TheSchwartz]] || [[Invite code generation#Multiple_user_grants|Invite codes]] || Email distributed invite codes
 +
|-
 +
! dw-send-email
 +
| ? || emails || or your email won't work
 
|-
 
|-
 
! [https://github.com/dreamwidth/dw-free/blob/develop/bin/worker/embeds embeds]
 
! [https://github.com/dreamwidth/dw-free/blob/develop/bin/worker/embeds embeds]
Line 134: Line 137:
 
|-
 
|-
 
! [https://github.com/dreamwidth/dw-free/blob/develop/bin/worker/search-constraints search-constraints]
 
! [https://github.com/dreamwidth/dw-free/blob/develop/bin/worker/search-constraints search-constraints]
| [[Gearman]] || Directory/user search || Something to do with partial search results? Not sure
+
| [[Gearman]] || Directory/user search || Convert a single constraint (e.g., trusted by x) to a set handle, using cached results when possible
 
|-
 
|-
 
! [https://github.com/dreamwidth/dw-free/blob/develop/bin/worker/search-lookup search-lookup]
 
! [https://github.com/dreamwidth/dw-free/blob/develop/bin/worker/search-lookup search-lookup]
| [[Gearman]] || Directory/user search || Initiate search? Not sure
+
| [[Gearman]] || Directory/user search || Initiate search, filtering with each constraint's set handle, returning the intersection; reads the binary data file once at startup so it should be restarted often
 
|-
 
|-
 
! [https://github.com/dreamwidth/dw-free/blob/develop/bin/worker/search-updater search-updater]
 
! [https://github.com/dreamwidth/dw-free/blob/develop/bin/worker/search-updater search-updater]
| Manual || Directory/user search || Not sure what it does, still used but has some bitrot: refers to bin/ljworkerctl which is ljcomint
+
| Manual || Directory/user search || Copy packed user data from mysql to the binary data file; has some bitrot: refers to bin/ljworkerctl which is ljcomint so it can't actually restart search-lookup
 
|-
 
|-
 
! [https://github.com/dreamwidth/dw-free/blob/develop/bin/worker/send-email send-email]
 
! [https://github.com/dreamwidth/dw-free/blob/develop/bin/worker/send-email send-email]

Latest revision as of 00:26, 30 January 2024

Workers are programs that don't run under control of Apache. They are processes which happen asynchronously, either because they use a lot of resources (e.g. search), or because they need to contact external sites (e.g. importer).

Some workers process TheSchwartz or Gearman requests. Those normally run under control of worker-manager (see below). Some workers are manual, and depending on their purpose, can run either under control of worker-manager or be started by cron, or even just once when (re)starting the server. (Manual workers are a mixed lot, and there's no "one size fits all" option for them.)

Before using workers that rely on TheSchwartz or Gearman, you will need to make sure that these services are installed and configured on your server.

Starting and stopping workers

Individually

When developing, you can start and debug individual workers from the shell prompt, by typing:

$LJHOME/bin/worker/(worker-name) --verbose

The --verbose or -v flag makes sure that the worker stays in the foreground and provides extra debugging text. To stop the worker when you're done, use ^C.

If you run the worker without --verbose flag, it will continue running in the background even after you're done and have logged out. You probably don't want to do this! If you want run multiple workers at the same time, or have them keep running, use the worker-manager described below.

Most of the time, if you are developing a worker, you will have a separate shell where you run the worker and can see its error messages as you work. Remember that every time you edit the code run by the worker (whether it is directly in the worker or just the libraries the worker uses) you need to kill and restart the worker process itself, or the worker will not see your changes.


Using worker-manager and workers.conf

bin/worker-manager takes care of starting, stopping, and restarting workers to maintain the numbers of workers set in etc/workers.conf for each host it runs on. The etc/workers.conf syntax is pretty straightforward and described in the file itself. To start bin/worker-manager for normal use (daemonized), use:

$LJHOME/bin/worker-manager

Or, if you want it to stay in the foreground and display progress/debug messages, use:

$LJHOME/bin/worker-manager --debug

In both cases, killing it will also kill the workers it started.

If you find that you do not want to use some workers (for testing, temporarily, or long-term), you may turn them off in the configuration. In /etc/workers.conf, setting the count to 0 will not start any of that worker. Commenting out the line on which that worker appears (with a #) will not start any of that worker. These are best for temporary changes. For a permanent change, you may also entirely remove the line on which that worker appears .

Using cron or at server boot time

Please refer to your OS documentation. The crontab(1) and init(8) manpages are probably most relevant.

List of workers

Below is a list of all the workers in the $LJHOME/bin/worker directory.

Workers in alphabetical order of filename
Filename Type Category Function and notes
birthday-notify Manual ESN Fire LJ::Event::Birthday ESN events for users whose birthdays are soon
change-poster-id TheSchwartz Journal content Does the backend work associated with claiming an OpenID account
content-importer TheSchwartz Importer Import all content
content-importer-lite TheSchwartz Importer Import some content (everything except entries and comments)
content-importer-verify TheSchwartz Importer Verify import password
directory-meta Manual Directory/user search Update packed user data in mysql, used by search-updater to update the binary data file
distribute-invites TheSchwartz Invite codes Email distributed invite codes
dw-send-email  ? emails or your email won't work
embeds TheSchwartz Embed codes Grab titles from embed APIs
esn-cluster-subs TheSchwartz ESN Notification delivery stage 2
esn-cluster-subs-mass TheSchwartz ESN Notification delivery stage 2 (mass)
esn-filter-subs TheSchwartz ESN Notification delivery stage 3
esn-filter-subs-mass TheSchwartz ESN Notification delivery stage 3 (mass)
esn-fired-event TheSchwartz ESN Notification delivery stage 1
esn-fired-event-mass TheSchwartz ESN Notification delivery stage 1 (mass)
esn-process-sub TheSchwartz ESN Notification delivery stage 4
esn-process-sub-mass TheSchwartz ESN Notification delivery stage 4 (mass)
expunge-users Manual Account deletion Expunge (permanently) accounts deleted more than 60 days ago
import-eraser TheSchwartz Importer Erases all imported content from a specified journal
import-scheduler Manual Importer Queue importing jobs in the right order
incoming-email TheSchwartz Incoming email Handles email posting, email support requests, email support follow-up
latest-feed TheSchwartz Latest things Heavy lifting for the Latest Things page
lazy-cleanup TheSchwartz Journal management Postpone some of the work of entry deletion
paidstatus Manual Payments Process paid carts, expired paid accounts, and paid accounts expiring soon
process-esn TheSchwartz ESN Notification delivery stages 1-4 combined
process-esn-mass TheSchwartz ESN Notification delivery stage 1-4 combined (mass)
process-eventlogrecord TheSchwartz Disabled Not used currently, may want to get rid of it along with LJ::EventLogRecord, LJ::EventLogRecord::*
process-privacy TheSchwartz Entry privacy Process mass entry privacy changes
resolve-extacct Gearman External accounts Determine the account type (personal/community/syndicated)
schedule-synsuck Manual Syndication Queue TheSchwartz jobs for updating syndicated accounts
search-constraints Gearman Directory/user search Convert a single constraint (e.g., trusted by x) to a set handle, using cached results when possible
search-lookup Gearman Directory/user search Initiate search, filtering with each constraint's set handle, returning the intersection; reads the binary data file once at startup so it should be restarted often
search-updater Manual Directory/user search Copy packed user data from mysql to the binary data file; has some bitrot: refers to bin/ljworkerctl which is ljcomint so it can't actually restart search-lookup
send-email TheSchwartz Sending email Send email to users
send-email-mass TheSchwartz Sending email Send email to users (mass)
send-email-ses TheSchwartz Sending email Send email to users through Amazon SES
shop-creditcard-charge Gearman Payments Charge a user's credit card
spellcheck-gm Gearman Journal content Gearman worker for running spellcheck on requested text
sphinx-copier TheSchwartz Full-text search Keep the full-text search database in sync with actual entries
sphinx-search-gm Gearman Full-text search Interface to the core sphinx search
stats-collection Manual Site stats Collect/compile statistics
support-notify TheSchwartz Support board Email users about support answers they received or support actions they subscribed to
synsuck TheSchwartz Syndication Updates syndicated account from feed source, started by bin/worker/schedule-synsuck or through the hubbub system
sysban-gm Gearman Site administration Retrieve sysban entries by type
t-memlimit Manual Testing Memory stress test
taglib-gm Gearman Journal content Retrieve user's tags
talklib-gm Gearman Journal content Update entry comment count, needed... not sure
userpic-resize-gm Gearman Userpics Resize userpics larger than maximum height or width
xpost TheSchwartz Crossposting Crossposts new or edited entries