Code push

The checklist for updating the code running on the live "dreamwidth.org" production site to the most recent version of the code. Admins of other installations will most likely need to alter this checklist based on your site's unique needs.

To do in advance of the push

Check for pushability

Code pushes can be requested for business reasons or by members of the dev team, but the admin doing the push should take a few minutes to check over what will be pushed to make sure all patches are complete and in a releasable state. Before a push, check with Denise, Mark, or Kareila a few hours in advance to see if there are any pull requests in the review queue that should go into the release for business reasons.

Announce in dw_maintenance that a code push is scheduled

Ideally, the comm should be updated a day or so in advance to let people know the timing of the push; at the very least it should be a few hours' notice. Give times in your local timezone and in GMT, and be sure to include the timezone information. Add a link to <a href="http://www.timeanddate.com/worldclock/fixedform.html">Time and Date's Event Time Announcer</a>.

Preparing to push

Repository prep

For both dw-free and dw-nonfree:

• make sure previous release branch is merged into develop
• make new release branches in both places, incrementing the previous release number
• double check the new branches in Github, making sure each one looks identical to the develop branch for that repo

Production prep

Update to the new release branch on the admin box, being careful not to overwrite any live fixes in production that aren't checked in:

$ sudo su - dw
$ cd $LJHOME # should put you in /home/dw/current
$ git status # you'll see the live code
$ git stash
$ git fetch origin
$ git checkout release-1.xx
$ git stash pop # and handle any merge conflicts

Then repeat the same process in $LJHOME/ext/dw-nonfree (but there are almost never live code changes there).

Once this process is completed, git status in both $LJHOME and $LJHOME/ext/dw-nonfree should be clean (modulo the live changes) and should both be on the expected release branch - but not a bad idea to "git log" and verify the last commit is as expected.

Check for config changes

Do a diff against the previous release branch to see if any config changes were made, and reflect those changes in ext/local/etc/config.pl as needed.

If any workers were added or removed, update and run this script: ext/local/etc/angel/regen-configs.pl (we don’t use workers.conf in production).

Check for new module requirements

Run bin/checkconfig.pl and make sure we don’t require any additional modules. If we do, make sure they get installed before proceeding.

Update Twitter

Update the @dreamwidth Twitter account when beginning the push to let people know the push is starting.

Update dw_maintenance

When beginning the push, update dw_maintenance again to announce that it's starting. Include a downtime estimate, with an extra 20% or so added on top for unexpected problems.

Reload/recompile all the things

At this point you kind of have to begin to move quickly since this phase changes database stuff like styles, text, etc. - if you update those but don't push the code changes for a while, you can end up causing weird behavior on the site.

$ bin/build-static.sh  # this does compass compile, js minify
$ bin/upgrading/texttool.pl load
$ bin/upgrading/update-db.pl -r
$ bin/upgrading/update-db.pl -r --cluster=all --drop
$ bin/upgrading/update-db.pl -r -p

(The --drop is only needed if obsolete tables were removed in this update.)

If the proxy source code has changed, it will also need rebuilding:

$ cd src/proxy
$ go build

Code Push + All The Restarts

If for some reason the site needs to be put into maintenance mode during the push, do that here:

$ sudo su -
$ bin/maint-on

Then to do the actual code push:

$ sudo su -
$ bin/sync-dw-home
$ bin/restart-webs
$ bin/restart-jobs  # to restart the workers

Once every thing is restarted, take things out of maintenance mode if needed:

$ bin/maint-off

It may also be necessary to restart the SSL Image Proxy, but historically whenever that's changed it gets pushed separately.

Spot-check the push

Inform the #dw-dev channel that the push has been completed and the site is back on. Ask them to spot-check a few functions by posting entries, reading their reading page, updating settings, etc. Watch the logs for any errors or problems (by running bin/tail-logs as root).

After the push is complete

Update the dw_maintenance entry

Edit the dw-maintenance entry (don't post a new one) with the date/time (in both your local timezone, clearly marked, and in GMT) with a notice that the push has been completed. Let them know that if they spot any issues, they should contact Support or leave a comment.

Update Twitter account

If needed, update the Twitter account again to let people know the push is complete, and that any problems should go to Support or dw-maintenance.

Highlight new journal themes

If there are any new journal themes in this push, use the /admin/themes interface to add them to the "New" category in the theme chooser.

Monitor dw-maintenance entry for a few hours

Make sure someone can stay available for a few hours as dw_maintenance comments come in, in case there's a major issue that needs to be resolved with another push. (You don't need to reply to the comments -- Denise or Kareila will handle that.)

Commit any needed fixes on the release branch

If there is a secondary push to resolve bugs, these commits should go directly on the release branch. The hotfixes will be merged back to the develop branch before the next push.