Difference between revisions of "Dev Maintenance"

From Dreamwidth Notes
Jump to: navigation, search
(Update your database: Test database update (notably for dreamhacks))
 
(22 intermediate revisions by 11 users not shown)
Line 1: Line 1:
{{Note|text="We're in the middle of moving over to a git workflow. If you haven't yet moved over to git, see the [http://wiki.dwscoalition.org/wiki/index.php?title=Dev_Maintenance&oldid=10581 old instructions]. If you're looking to move over, first make sure you've finished [[Moving your Dreamwidth installation to use Github]]"}}
 
 
 
== Updating the Dreamwidth code on your Dreamhack ==
 
== Updating the Dreamwidth code on your Dreamhack ==
  
Line 9: Line 7:
 
  stop-apache
 
  stop-apache
  
=== Updating dw-free ===
+
=== Getting updates from Github ===
  
Okay.  Let's say you've been running your Dreamwidth install and you want to pull down the latest and greatest in fixes.  This is pretty easy.  First, if you are not somewhere in dw-free, get into that directory:
+
Okay.  Let's say you've been running your Dreamwidth install and you want to pull down the latest and greatest in fixes.  This is pretty easy.  First, if you are not already somewhere in your site's directory, get into that directory:
  
 
  cd $LJHOME
 
  cd $LJHOME
  
Then grab updates from the development and master branches of Dreamwidth's repository:
+
Then grab updates from the development branch of Dreamwidth's repository:
  
 
  git fetch dreamwidth
 
  git fetch dreamwidth
  git pull --ff-only dreamwidth develop:develop
+
  git checkout main && git pull --ff-only dreamwidth main:main
git pull --ff-only dreamwidth master:master
+
  
You will also want to push the Dreamwidth repository changes to your fork on Github:
+
If your installation hasn't been updated to use <code>main</code> instead of <code>master</code> for the primary development branch, see [https://dw-dev.dreamwidth.org/232594.html this post from Mark] detailing the necessary steps to get up to speed.
  
git push origin develop
+
Once you've updated, you will also want to push the Dreamwidth repository changes to your fork on Github:
git push origin master
+
  
Note: if you are working on uncommitted changes in a branch, any automerge will be canceled. Example:
+
  git push origin main
  
mw@memewidth:~/dw/src/jbackup$ git pull dreamwidth develop:develop
+
If you have uncommitted changes, you'll get an error message and will need to use [[Git How To#How_to_stash_your_changes|git stash]].
remote: Counting objects: 180, done.
+
remote: Compressing objects: 100% (54/54), done.
+
remote: Total 116 (delta 90), reused 88 (delta 62)
+
Receiving objects: 100% (116/116), 36.23 KiB, done.
+
Resolving deltas: 100% (90/90), completed with 41 local objects.
+
From https://github.com/dreamwidth/dw-free
+
    53294c1..19b8e73  develop    -> develop
+
error: Your local changes to 'src/jbackup/jbackup.pl' would be overwritten by merge.  Aborting.
+
Please, commit your changes or stash them before you can merge.
+
  
We will get into merging in changes in the following section.
 
 
=== Merge the changes with your branch(es) ===
 
 
{{Warn|text=Pay special attention to stashing unsaved changes!  You can lose work if you are not careful about this!}}
 
 
You should be doing your development in branches separate from the main Dreamwidth branches.  You can check what branches you currently have in your repository with the command:
 
 
git branch
 
 
The current branch will have an asterisk next to it.  Here is an example:
 
 
yourhack@dreamhack.net:~/dw$ git branch
 
  develop
 
* feature/Mobile
 
  master
 
 
You can check to see if you have unsaved changes on your currently checked out branch with:
 
 
git status
 
 
To temporarily save these changes while you make your updates, use:
 
 
git stash
 
 
You can then merge your independent branch with the changes made with Dreamwidth's development branch:
 
 
git merge develop
 
 
And if you have saved changes using stash, you can recover them with:
 
 
git stash pop
 
 
You can change branches with <tt>git checkout BRANCH</tt>, example:
 
 
git checkout feature/Mobile
 
 
=== For those using DW non-free ===
 
 
Repeat for dw-nonfree:
 
 
cd $LJHOME/ext/dw-nonfree
 
 
git pull dreamwidth develop:develop
 
git pull dreamwidth master:master
 
git push origin develop
 
git push origin master
 
 
If you have any branches on dw-nonfree, they will also need to be merged with the development branch of dw-nonfree as described in the above section:
 
 
git merge develop
 
  
 
=== Update your database ===
 
=== Update your database ===
Line 99: Line 35:
 
  $LJHOME/bin/upgrading/update-db.pl -r --cluster=all --innodb
 
  $LJHOME/bin/upgrading/update-db.pl -r --cluster=all --innodb
 
  $LJHOME/bin/upgrading/texttool.pl load
 
  $LJHOME/bin/upgrading/texttool.pl load
 +
# if you have a test database (you have one if you have a dreamhack)
 +
DW_TEST=1 $LJHOME/bin/upgrading/update-db.pl -r -p --innodb
 +
DW_TEST=1 $LJHOME/bin/upgrading/update-db.pl -r --cluster=all --innodb
 +
DW_TEST=1 $LJHOME/bin/upgrading/texttool.pl load
  
 
Remember that different branches might have different text strings you have added, and that you might need to update your database for different branches when you are working on them for those text strings to work.  The same applies for any database changes made by different branches you are working on.
 
Remember that different branches might have different text strings you have added, and that you might need to update your database for different branches when you are working on them for those text strings to work.  The same applies for any database changes made by different branches you are working on.
 +
 +
=== Compile the SCSS and other static files ===
 +
 +
You'll also want to compile any changes to the static files. This will convert SCSS into CSS, as well as recompressing any changed javascript files, into the static build directory:
 +
 +
$LJHOME/bin/build-static.sh
  
 
=== Restart the server ===
 
=== Restart the server ===
Line 130: Line 76:
 
== Scripting ==
 
== Scripting ==
  
{{Warn|text="These scripts have not been throughly tested yet with the new system; they might not be updated entirely or right."}}
+
{{Warn|text="These scripts have not been thoroughly tested yet with the new system; they might not be updated entirely or right."}}
  
 
You can, of course, use scripts to make it easier for you to do some of this.   
 
You can, of course, use scripts to make it easier for you to do some of this.   
  
There is an "omnibus" script available at http://dw-dev.dreamwidth.org/94822.html which incorporates all the individual scripts listed here and some other functions, and includes help information.
+
There is an epic "omnibus" script by <dwuser>jeshyr</dwuser> available at http://dw-dev.dreamwidth.org/94822.html which incorporates all the individual scripts listed here and some other useful functions, and includes help information. It'll save you a heap of angst as a beginner!
  
 
Some simpler example scripts are given below.
 
Some simpler example scripts are given below.
Line 145: Line 91:
  
 
# make sure we are in the right directory;
 
# make sure we are in the right directory;
# save previous directory
 
oldpwd=$PWD
 
 
cd $LJHOME
 
cd $LJHOME
 +
 
git fetch dreamwidth
 
git fetch dreamwidth
  
 
# pull changes from dreamwidth
 
# pull changes from dreamwidth
git checkout develop
 
git pull --ff-only dreamwidth develop
 
 
git checkout master
 
git checkout master
 
git pull --ff-only dreamwidth master
 
git pull --ff-only dreamwidth master
  
 
# push them to Github forks
 
# push them to Github forks
git push origin develop
 
 
git push origin master
 
git push origin master
  
# return to old directory
+
# restore our old position
cd $oldpwd
+
git checkout -
</source>
+
  
Now, when you type 'dwu', this script will update dw-free. If you need to update dw-nonfree as well, then use this code:
+
# check to see if we have a dw-nonfree directory; if we do, update that too
 +
if [ -d "$LJHOME/ext/dw-nonfree" ]; then
 +
  # change to dw-nonfree
 +
  cd $LJHOME/ext/dw-nonfree
  
<source lang="bash">#!/bin/bash
+
  git fetch dreamwidth
  
# make sure we are in the right directory;
+
  # pull changes from dw-nonfree
# save previous directory
+
  git checkout master
oldpwd=$PWD
+
  git pull --ff-only dreamwidth master
cd $LJHOME
+
  
git fetch dreamwidth
+
  # push them to Github forks
 +
  git push origin master
  
# pull changes from dreamwidth
+
  # restore our old position
git checkout develop
+
  git checkout -
git pull --ff dreamwidth develop
+
fi
git checkout master
+
</source>
git pull --ff dreamwidth master
+
  
# push them to Github forks
+
Now, when you type 'dwu', this script will update your dw-free and (if you have it) dw-nonfree repositories.
git push origin develop
+
git push origin master
+
  
# change to dw-nonfree
+
=== dwdb - Updating the database ===
cd $LJHOME/ext/dw-nonfree
+
  
git fetch dreamwidth
+
Put this code in a file called '''~/bin/dwdb''' and make it executable with '''chmod ugo+x ~/bin/dwdb''':
  
# pull changes from dw-nonfree
+
<source lang="bash">#!/bin/bash
git checkout develop
+
$LJHOME/bin/upgrading/update-db.pl -r -p --innodb &&
git pull --ff dreamwidth develop
+
$LJHOME/bin/upgrading/update-db.pl -r --cluster=all --innodb &&
git checkout master
+
$LJHOME/bin/upgrading/texttool.pl load</source>
git pull --ff dreamwidth master
+
  
# push them to Github forks
+
This will update the database when you type 'dwdb'.
git push origin develop
+
git push origin master
+
  
# return to previous directory
+
=== comp - Recompiling the compass files ===
cd $oldpwd
+
</source>
+
  
=== dwdb - Updating the database ===
+
Put this code in a file called "'~/bin/comp'" and make it executable with "'chmod ugo+x ~/bin/comp'":
 
+
Put this code in a file called '''~/bin/dwdb''' and make it executable with '''chmod ugo+x ~/bin/dwdb''':
+
  
 
<source lang="bash">#!/bin/bash
 
<source lang="bash">#!/bin/bash
$LJHOME/bin/upgrading/update-db.pl -r -p --innodb && \
+
cd "$LJHOME" &&
$LJHOME/bin/upgrading/update-db.pl -r --cluster=all --innodb && \
+
compass compile &&
$LJHOME/bin/upgrading/texttool.pl load</source>
+
cd ext/dw-nonfree &&
 +
compass compile</source>
  
This will update the database when you type 'dwdb'.
+
This will recompile the CSS files.
  
 
==Cleaning up your directories==
 
==Cleaning up your directories==
Line 224: Line 158:
  
 
[[Category: Development]]
 
[[Category: Development]]
 +
[[Category: Dreamhack]]

Latest revision as of 22:57, 29 April 2024

Updating the Dreamwidth code on your Dreamhack

Shutting down apache

It is best to shut your Apache instance down before doing the update process, to make sure that everything, especially scripts in cgi-bin/ are reloaded properly:

stop-apache

Getting updates from Github

Okay. Let's say you've been running your Dreamwidth install and you want to pull down the latest and greatest in fixes. This is pretty easy. First, if you are not already somewhere in your site's directory, get into that directory:

cd $LJHOME

Then grab updates from the development branch of Dreamwidth's repository:

git fetch dreamwidth
git checkout main && git pull --ff-only dreamwidth main:main

If your installation hasn't been updated to use main instead of master for the primary development branch, see this post from Mark detailing the necessary steps to get up to speed.

Once you've updated, you will also want to push the Dreamwidth repository changes to your fork on Github:

git push origin main

If you have uncommitted changes, you'll get an error message and will need to use git stash.


Update your database

Now that your code has been updated, update the database:

# order of commands is important
$LJHOME/bin/upgrading/update-db.pl -r -p --innodb
$LJHOME/bin/upgrading/update-db.pl -r --cluster=all --innodb
$LJHOME/bin/upgrading/texttool.pl load
# if you have a test database (you have one if you have a dreamhack)
DW_TEST=1 $LJHOME/bin/upgrading/update-db.pl -r -p --innodb
DW_TEST=1 $LJHOME/bin/upgrading/update-db.pl -r --cluster=all --innodb
DW_TEST=1 $LJHOME/bin/upgrading/texttool.pl load

Remember that different branches might have different text strings you have added, and that you might need to update your database for different branches when you are working on them for those text strings to work. The same applies for any database changes made by different branches you are working on.

Compile the SCSS and other static files

You'll also want to compile any changes to the static files. This will convert SCSS into CSS, as well as recompressing any changed javascript files, into the static build directory:

$LJHOME/bin/build-static.sh

Restart the server

Now you can restart Apache:

start-apache

Of course, in a production environment, this whole process is not too recommended as you never know what kind of code you're going to get. But for the most part, it's fairly straightforward. (And if you're doing development, this is generally safe.)

Additional Information for Non-Dreamhack Users

This section is only for you if you're running your own installation.

Instead of using start-apache/stop-apache, which are Dreamhack-specific scripts, use these commands:

sudo /etc/init.d/apache2 start
sudo /etc/init.d/apache2 stop 

You'll also want to update packages on your system at some point. On Ubuntu, this would be done using:

apt-get update
apt-get upgrade

Or, if you want an easy command to run on your dw account in one swoop:

sudo apt-get update && sudo apt-get upgrade

Scripting

Warning: "These scripts have not been thoroughly tested yet with the new system; they might not be updated entirely or right."

You can, of course, use scripts to make it easier for you to do some of this.

There is an epic "omnibus" script by [info]jeshyr available at http://dw-dev.dreamwidth.org/94822.html which incorporates all the individual scripts listed here and some other useful functions, and includes help information. It'll save you a heap of angst as a beginner!

Some simpler example scripts are given below.

dwu - Updating the repos

Put this code in a file called ~/bin/dwu and make it executable with chmod ugo+x ~/bin/dwu:

#!/bin/bash
 
# make sure we are in the right directory;
cd $LJHOME
 
git fetch dreamwidth
 
# pull changes from dreamwidth
git checkout master
git pull --ff-only dreamwidth master
 
# push them to Github forks
git push origin master
 
# restore our old position
git checkout -
 
# check to see if we have a dw-nonfree directory; if we do, update that too
if [ -d "$LJHOME/ext/dw-nonfree" ]; then
  # change to dw-nonfree
  cd $LJHOME/ext/dw-nonfree
 
  git fetch dreamwidth
 
  # pull changes from dw-nonfree
  git checkout master
  git pull --ff-only dreamwidth master
 
  # push them to Github forks
  git push origin master
 
  # restore our old position
  git checkout -
fi

Now, when you type 'dwu', this script will update your dw-free and (if you have it) dw-nonfree repositories.

dwdb - Updating the database

Put this code in a file called ~/bin/dwdb and make it executable with chmod ugo+x ~/bin/dwdb:

#!/bin/bash
$LJHOME/bin/upgrading/update-db.pl -r -p --innodb &&
$LJHOME/bin/upgrading/update-db.pl -r --cluster=all --innodb &&
$LJHOME/bin/upgrading/texttool.pl load

This will update the database when you type 'dwdb'.

comp - Recompiling the compass files

Put this code in a file called "'~/bin/comp'" and make it executable with "'chmod ugo+x ~/bin/comp'":

#!/bin/bash
cd "$LJHOME" &&
compass compile &&
cd ext/dw-nonfree &&
compass compile

This will recompile the CSS files.

Cleaning up your directories

If you do any amount of work, you'll find that your directories get cluttered with .orig and .rej files everywhere. This script will clean those up:

git clean -f "*.rej" "*.orig"

You can run it from the command line, or put it in a file called ~/bin/tidy and make it executable with "chmod +x ~/bin/tidy". If you do that, you'll be able to just type "tidy" to clean house.