Difference between revisions of "TheSchwartz"

From Dreamwidth Notes
Jump to: navigation, search
(Add info about using it from Dreamwidth code)
(Sending non-ESN email to a user)
 
(8 intermediate revisions by 4 users not shown)
Line 1: Line 1:
TheSchwartz is a worker manager system that runs things like the importer, the private messaging function, the crossposter, and all of the event notifications.  It makes it so that the web servers don't have to spend CPU timing doing work that doesn't have to be done instantly.
+
TheSchwartz is a [[Workers|worker]] manager system that runs things like the importer, the private messaging function, the crossposter, and all of the event notifications.  It makes it so that the servers that serve pages don't have to spend time doing work that doesn't have to be done instantly.
  
 
== Setting up TheSchwartz ==
 
== Setting up TheSchwartz ==
Line 11: Line 11:
 
The following example queues a pony-feeding job, passing it amounts of hay and oats to use.
 
The following example queues a pony-feeding job, passing it amounts of hay and oats to use.
  
<syntaxhighlighting lang="perl">
+
<syntaxhighlight lang="perl">
 
use TheSchwartz;
 
use TheSchwartz;
 
my $sclient = LJ::theschwartz(); # LJ::theschwartz( { role => "mass" } ) for very large stables
 
my $sclient = LJ::theschwartz(); # LJ::theschwartz( { role => "mass" } ) for very large stables
Line 19: Line 19:
 
die "Can't create job" unless $job;
 
die "Can't create job" unless $job;
 
$sclient->insert($job) or die "Can't queue feeding job";
 
$sclient->insert($job) or die "Can't queue feeding job";
</syntaxhighlighting>
+
</syntaxhighlight>
  
<code>LJ::theschwartz</code> is defined in [http://hg.dwscoalition.org/dw-free/file/tip/cgi-bin/ljlib.pl cgi-bin/ljlib.pl], so if you're not in web context (command-line program, Gearman worker, TheSchwartz worker), you'll need to <code>require "ljlib.pl";</code> as well.
+
<code>LJ::theschwartz</code> is defined in [https://github.com/dreamwidth/dw-free/blob/develop/cgi-bin/ljlib.pl cgi-bin/ljlib.pl], so if you're not in web context (command-line program, Gearman worker, TheSchwartz worker, or manual worker), you'll need to <code>use lib "$ENV{LJHOME}/cgi-bin"; require "ljlib.pl";</code> as well.
  
 
You should be able to adapt this to your specific job and data.
 
You should be able to adapt this to your specific job and data.
Line 31: Line 31:
 
=== Sending non-ESN email to a user ===
 
=== Sending non-ESN email to a user ===
  
Use LJ::send_mail from [http://hg.dwscoalition.org/dw-free/file/tip/cgi-bin/ljmail.pl cgi-bin/ljmail.pl]. Note that doing so correctly is fiddly, if you want to respect all of the user's email-related preferences. See [http://bugs.dwscoalition.org/show_bug.cgi?id=3946 bug 3946] for more.
+
Use <code>LJ::send_mail</code> from [https://github.com/dreamwidth/dw-free/blob/develop/cgi-bin/LJ/Sendmail.pm cgi-bin/LJ/Sendmail.pm]. Note that doing so correctly is fiddly, if you want to respect all of the user's email-related preferences, such as preferred email character set/encoding, HTML vs. plain text preference, or language.
  
 
== Writing a worker for Dreamwidth TheSchwartz jobs ==
 
== Writing a worker for Dreamwidth TheSchwartz jobs ==
Line 37: Line 37:
 
The following implements the <code>DW::Worker::FeedPonies</code> worker used in the example above:
 
The following implements the <code>DW::Worker::FeedPonies</code> worker used in the example above:
  
<syntaxhighlighting lang="perl">
+
<syntaxhighlight lang="perl">
 
use LJ::Worker::TheSchwartz;
 
use LJ::Worker::TheSchwartz;
 
schwartz_decl( "DW::Worker::FeedPonies" );
 
schwartz_decl( "DW::Worker::FeedPonies" );
Line 66: Line 66:
 
     $job->completed;
 
     $job->completed;
 
}
 
}
</syntaxhighlighting>
+
</syntaxhighlight>
  
 
You should be able to adapt this to your specific job and data.
 
You should be able to adapt this to your specific job and data.
  
 
[[Category: Development]]
 
[[Category: Development]]

Latest revision as of 10:24, 4 August 2014

TheSchwartz is a worker manager system that runs things like the importer, the private messaging function, the crossposter, and all of the event notifications. It makes it so that the servers that serve pages don't have to spend time doing work that doesn't have to be done instantly.

Setting up TheSchwartz

See TheSchwartz Setup.

Queueing a job for TheSchwartz from Dreamwidth

Generic Job

The following example queues a pony-feeding job, passing it amounts of hay and oats to use.

use TheSchwartz;
my $sclient = LJ::theschwartz(); # LJ::theschwartz( { role => "mass" } ) for very large stables
die "Can't get TheSchwartz client" unless $sclient;
my $job = TheSchwartz::Job->new_from_array("DW::Worker::FeedPonies",
                                           [ hay => $hay, oats => $oats ]);
die "Can't create job" unless $job;
$sclient->insert($job) or die "Can't queue feeding job";

LJ::theschwartz is defined in cgi-bin/ljlib.pl, so if you're not in web context (command-line program, Gearman worker, TheSchwartz worker, or manual worker), you'll need to use lib "$ENV{LJHOME}/cgi-bin"; require "ljlib.pl"; as well.

You should be able to adapt this to your specific job and data.

Notifying users of ESN events they subscribed to

Don't use the above. See Firing ESN events and delivering notifications instead.

Sending non-ESN email to a user

Use LJ::send_mail from cgi-bin/LJ/Sendmail.pm. Note that doing so correctly is fiddly, if you want to respect all of the user's email-related preferences, such as preferred email character set/encoding, HTML vs. plain text preference, or language.

Writing a worker for Dreamwidth TheSchwartz jobs

The following implements the DW::Worker::FeedPonies worker used in the example above:

use LJ::Worker::TheSchwartz;
schwartz_decl( "DW::Worker::FeedPonies" );
schwartz_work(); # Never returns.
 
package DW::Worker::FeedPonies;
use base 'TheSchwartz::Worker';
sub schwartz_capabilities { return ( __PACKAGE__ ); }
sub max_retries { 5 } # Try feeding ponies 5 times before giving up
sub grab_for { 600 } # Give the stable hand 600 seconds (10 minutes) to finish
sub keep_exit_status_for { 86400 } # Keep the result of the feeding attempt for 24 hours
 
sub retry_delay { # How long to wait before retrying on after an attempt fails
    my ($class, $fails) = @_;
 
    return (10, 30, 60, 300, 600)[$fails];
}
 
sub work { # Do the actual work 
    my ($class, $job) = @_;
    my %arg = %{$job->arg};
 
    $job->permanent_failure( "Amount of hay and or oats negative" )
        if $arg{hay} < 0 || $arg{oats} < 0;
    $job->debug( "Feeding the ponies $arg{hay} bales of hay and $arg{oats} bushels of oats" );
    Ponies->feed( %arg ) or return $job->failed( "Unable to feed ponies, retrying" );
 
    $job->completed;
}

You should be able to adapt this to your specific job and data.