Difference between revisions of "Importing"

Revision as of 19:11, 18 January 2009

Start

Basic download script base:

• http://codex.journal-press.com/Ljdump

I don't know anything about scripting, so I don't know if either of the below would be better or worse for our purposes. But

• ljmigrate is based on the ljdump code, captures usericons as well, and has the ability to import to lj-based systems.
• ljsm is a python script which captures usericons and posted images, in addition to entries and comments.

Examples

Good to look at Schwartz workers for reference. Possibilities:

• Worker for privacy conversion -- long running task limited to not kill the DB. bin/worker/process-privacy, LJ::MassPrivacy
• bin/worker/paidstatus - the schwartz 'shell', that declares the job. it imports the module DW::Worker::PaidStatus
• cgi-bin/DW/Worker/PaidStatus.pm - the actual worker; jobs can either succeed, fail permanently, fail temporarily. note the helper subs at the bottom of that module that define the schwartz parameters for retries, etc.
• cgi-bin/DW/Worker/Payment.pm - search for PaidStatus, line 90 -- you'll see the calling convention for creating a new schwartz job and passing arguments
• bin/test-pay - a generic script for testing the payment system

Rough Implementation Guide

1. create a new DW::Worker::ContentDownloader module (or something, name doesn't matter much)
2. create the simple bin/worker/contentdownloader that wraps the module; this is almost a carbon copy of existing bin/worker/paidstatus etc
3. create a test script that lets you insert a job. the test script can be really simple, something like: bin/test-content-downloader --insert-job xb95, and then the script would have four lines of code, one of which is that line 90 from Payment.pm referenced above, i.e., create the job, give it the arguments
4. then, you can test that your pipeline works by running the worker in debug mode... um, bin/worker/contentdownloader --debug I think. You probably have to do perl -I/home/dw/cgi-bin bin/worker/contentdownloader --debug to get the include path setup right
5. anyway, once all that works - your job is working, your module is getting called, etc, then you can pretty easily drop in a call out to the download script. system("/path/to/python", "/path/to/script", "--username", "xb95"); and then worry about error checking, timeout checking, and all the million failure cases

Ideal Process

1. run the content downloader,
2. copy data to mogile
3. mark the user as 'downloaded'
4. run content importer

Other Ideas

• exor674 I think one of the things we'd need to add for the import feature is a external-logid logprop, so if somebody tries to import twice you won't get twice of the entries -- AND the systen can know "oh DW entry 1 was originally livejournal.com-exor674-5827377212221211"
• "Imported from" tag and link
• Options: "import tags, tag prefix, tag suffix" "import comments"? "import friend groups, tag prefix, tag suffix" "import all as *security*"

Importing Comments

• Copyright issues. Idea from azurelunatic:

Supposing comments belonging to others were imported and privately posted, and comments belonging to the user were imported and posted according to their current screening setting in proper threads with the private comments. This would allow a maximum amount of the user's own content to show up.

Supposing also that the comment screening mechanism were modified to handle privately-posted imported comments. This would be done according to the screening settings on the original post: if unscreened on the original, someone OpenID-authenticated as the original comment poster could elect to own and unprivatescreen comments belonging to them.

After this point, the journal owner could screen/unscreen the comment at will.

To make this work better, imported comments should be listed somewhere that the OpenID owner of the comments could find them and possibly mass-unprivatescreen/take ownership.

[Business decision from rahaeli:

If someone is importing their journal directly from another site, by giving us their username and password for that site, import all comments as OpenID comments with the same screening/visibility level that the original comment had.

If someone is importing their journal from downloaded backup file, import all comments as OpenID comments screened, frozen, visible only to the journal owner and the OpenID identity that made the comment, and unable to be unscreened/unfrozen by the journal owner him/herself, only the OpenID identity. That way, the OpenID identity holder can verify that their words haven't been edited offline.

Either way, it would be very very nice if there were a place where an OpenID identity could go and see all their OpenID comments that have been imported and take ownership of them under a specific linked DW account.]

Email templates

These email templates will be sent out when we manage or fail to transfer a journal.

Could not access account

Title

Could not transfer your journal from SERVICE to your account USERNAME on Dreamwidth

Body

Hi, USERNAME--

We were unable to transfer your journal OTHERNAME from SERVICE to your account USERNAME on Dreamwidth. FAILURE MESSAGE.

Possible failure messages

• The username and password you gave us for SERVICE was rejected, so we couldn't download any information.

Journal transferred

Title

About your journal transfer from SERVICE to your account USERNAME on Dreamwidth

Body

Congratulations, USERNAME!

Your journal OTHERNAME on SERVICE has been transferred to your account USERNAME on Dreamwidth.

(OPTIONAL ISSUE/NOTIFICATION TEXT INSERTS: Things you might want to look at:)

Possible issue texts

Icon import

• Because you have more icons on SERVICE than we can import, we have imported your default icon. You can choose which other icons you wish to import at URL.
• Unfortunately, SERVICE doesn't support icon importing. You will have to manually upload your icons from this service.

Entries import

• Some entries couldn't be imported. You can view a list and the reasons at URL.
• We can't automatically transfer polls. You can view a list of entries with polls so you can manually recreate them at URL.
• We can't automatically transfer embedded content, such as YouTube videos. You can view a list of those entries containing embedded content and manual re-insert the item at URL.

Comments import

• Unfortunately, SERVICE doesn't support comment importing. You will have to manually upload your icons from this service.

Ways to enter a service name

(these provided by cheyinka)

• www.somejournal.com WORKS
• http://www.somejournal.com WORKS
• username.somejournal.com WORKS
• http://username.somejournal.com WORKS
• somejournal.com WORKS
• somejournal can't work
• SJ can't work
• username (the ever-popular "didn't read the directions" option) can't work
• internet explorer / semagic / &c. (probably indistinguishable from 'username' as far as the code's concerned) can't work

From John:

• on somejournal can't work
• somejorunal or similar misspellings, though no idea how to fix that. can't work
• AOL (:D) can't work
• username at/on somejournal can't work
• username@somejournal.com WORKS
• username at somejournal dot com WORKS
• SJ (where there are two services with the same initials. Perhaps not so much a consideration now, but in the event of super-federation Dreamwidth extreeeme, a possibility) can't work

From Janine:

• http://www.somejournal.com/users/username WORKS
• http://www.somejournal.com/~username WORKS
• http://users.somejournal.com/username WORKS
• A link to a particular entry in the journal, or really any other page within the journal WORKS

Compatability

Output from test tool found here: http://linode2.andreanall.com/~anall/hidden/check_compat.txt

LiveJournal

Works just fine.

DeadJournal

         Cannocalize: OK (www.deadjournal.com)
     SessionGenerate: OK
             UserURL: OK (redacted)
            Userpics: FAIL
              Groups: OK
                Tags: OK
                 Bio: OK (downloaded)
           SyncItems: OK
           GetEvents: OK
         CommentMeta: FAIL
         CommentMeta: FAIL
www.deadjournal.com Compatable: FAIL

JournalFen

Seems to be all OKs

InsaneJournal

All OK.