Difference between revisions of "Dreamwidth Scratch Installation"

Revision as of 22:20, 5 December 2018

Firstly, you'll need to set up a Linux system -- here are the Suggested Server Requirements. These instructions assume Ubuntu 16.04. Later versions (as of December 18, 2017) may or may not work, see https://github.com/dreamwidth/dw-free/issues/2248. Please let us know if you run into difficulties--we want to update this if people are having problems! We recommend coming into the #dreamwidth-dev channel on IRC.

After you set up, you can keep updated with the instructions on Dev Maintenance.

If you have difficulties or problems or suggestions for installation procedures, please report them to dw_dev. Also look at Production Setup: Webserver.

Command convention

In this walkthrough, commands to be run as a regular user are prefixed with $. Commands to be run as the super-user are prefixed with #. If you are not the super-user, you can run single commands like this:

$ sudo <command goes here>

Alternately, you can spawn a super-user shell using:

$ sudo -i
# <command goes here>

When done with the super-user shell, exit the shell via:

# exit

Configuring the OS and environment

Bringing the system up to date

Before you start installing anything else, you might want to make sure your system is up to date. You should update the packages on your system:

# apt-get update
# apt-get upgrade

If you want man pages on your system:

# apt-get install man

Installing necessary packages

This will install necessary packages you'll need to run Dreamwidth (and some optional ones). You'll want to be root when running these commands.

# apt-get install git apache2 apache2-bin apache2-data apache2-utils \
  libapache2-mod-perl2 libapache2-mod-apreq2 libapache2-request-perl \
  mysql-server wget unzip links vim libclass-autouse-perl libdatetime-perl \
  libcache-memcached-perl libhash-multivalue-perl libgd-gd2-perl \
  libhtml-template-perl libwww-perl libmime-lite-perl \
  liburi-perl libxml-simple-perl libclass-accessor-perl \
  libclass-data-inheritable-perl libclass-trigger-perl libcrypt-dh-perl \
  libmath-bigint-gmp-perl liburi-fetch-perl libgd-graph-perl \
  libgnupg-interface-perl libmail-gnupg-perl perlmagick \
  libproc-processtable-perl libsoap-lite-perl librpc-xml-perl \
  libstring-crc32-perl libtext-vcard-perl libxml-atom-perl libxml-rss-perl \
  libimage-size-perl libunicode-maputf8-perl libgtop2-dev build-essential \
  libnet-openid-consumer-perl libnet-openid-server-perl libyaml-perl \
  libcaptcha-recaptcha-perl libdbd-sqlite3-perl libtest-simple-perl \
  libtemplate-perl libterm-readkey-perl libmime-base64-urlsafe-perl \
  gcc libtest-most-perl libgearman-client-perl libfile-find-rule-perl \
  libbusiness-creditcard-perl liblwpx-paranoidagent-perl libtheschwartz-perl \
  libfile-type-perl libjson-perl ruby libdbd-mysql-perl libdanga-socket-perl \
  libio-aio-perl libsys-syscall-perl liblog-log4perl-perl libtext-markdown-perl \
  libimage-exiftool-perl libnet-oauth-perl libnet-smtps-perl libxmlrpc-lite-perl

This will download about 106MB of files and use around 511MB of disk space.

Check whether these packages have actually installed. Later on in the process, if you find you are having inexplicable problems, try installing these again.

You will also want to install and configure Postfix so your DW can send out email:

# apt-get install postfix

If you choose not to configure on the install, you can do so later with:

# dpkg-reconfigure postfix

Configure Apache 2

Check whether your system is using threaded MPMs or pre-fork. You will need the latter. Note: if you ran the above command to install Apache, then you need to run these commands to switch to the pre-fork model:

# a2dismod mpm_event
# a2enmod mpm_prefork
# service apache2 restart

Install Perl modules with CPAN

You will also want to install some perl libraries with CPAN. Defaults during CPAN's setup should be okay.

Before installing, check if your CPAN shell has make at the correct location. Run

# which make
/usr/bin/make

then compare the results with running:

# cpan 
...
Would you like to configure as much as possible automatically? [yes] <PRESS ENTER>
...
cpan[1]> o conf make
    make              [/usr/bin/make]
...
cpan[2]> exit

The value printed should match that from the which make command you previously ran.

To install the perl libraries, first set up CPAN:

# cpan Bundle::CPAN

When prompted, press Enter to exit tests. Then, run the following command to install the rest of the necessary libraries:

# cpan GTop Digest::SHA1 Unicode::CheckUTF8 MogileFS::Client \
  TheSchwartz::Worker::SendEmail LWP::UserAgent::Paranoid \
  Mozilla::CA List::Util Paws::S3 Net::DNS Text::Fuzzy

(Net::DNS is required to be installed via CPAN because the version included with current versions of Ubuntu is 0.81, which is too old for LWP::UserAgent::Paranoid to work properly.)

Note that the MogileFS::Client is required even if you are not configuring MogileFS.

Set Timezone to UTC

Dreamwidth will complain if you aren't running in UTC. If you're not configured for UTC already, you can reset it (on Ubuntu/Debian) using

# dpkg-reconfigure tzdata

and choose Etc/UTC.

Configuring MySQL/MariaDB

Newer versions of MySQL and/or MariaDB have imposed stricter requirements than the current Dreamwidth SQL definitions can comply with. These restrictions must be relaxed, or the initial database population scripts will fail.

To do this, run the following 4 commands:

$ sudo -i
# echo "sql_mode = ''" >> /etc/mysql/mysql.conf.d/mysqld.cnf
# service mysql restart
# exit

Setting up the DW user account

Note: If you still find yourself unable to make sudo commands even after usermod, please see Making your DW user a sudo account for alternatives.

Create a user account that will be used for the code. This can be your own personal account if you wish, but it is recommended that you setup a new user that is only going to be used for running the code. That way if you have any insecure code running (i.e., a bad page, or a patch you're working on doesn't have security implemented yet) you don't have to worry about someone getting access to your personal files.

To set up the user account:

# adduser username

Fill out the user's info and put them in the sudo group:

# usermod -a -G sudo username

So, to use dw as a username:

# adduser dw
# usermod -a -G sudo dw

Now, we will get the $LJHOME variable set. Log into your user:

# su - dw

We will assume that you use the bash shell. If you do not, then this section does not apply. You will have to find the shell specific way of setting environment variables.

For bash, you should look in the home directory of the user account you just created for a file .profile. You will want to add one more line to this file:

export LJHOME=/home/dw/dw

Or wherever it is that you want the root of your source code install to be. Yes, the variable is named LJHOME. We'll live with it for now. ;-)

Test this. Log out of the dw user account and log back in, then type:

echo $LJHOME

You should see /home/dw/dw or whatever you set it to. If you don't, then something has gone awry -- maybe the export line isn't in the right place..?

Get the Dreamwidth code

Configure git

Pick up a name and email address you're willing to share with the world (or at least that part of it that pays attention to GitHub and Dreamwidth). Then run the following 2 commands, replacing NAME and EMAIL with what you chose:

$ git config --global user.name "NAME"
$ git config --global user.email "EMAIL"

Optionally, make git use your preferred editor for commit messages (and a few other things). First find the path to your favorite editor (replace EDITOR with the name of your preferred editor, eg nano or vi):

$ which EDITOR

This will print something like: /usr/bin/vi (if your editor of choice is vi). Then, run the following command, replacing /PATH/TO/EDITOR with whatever was output:

$ git config --global core.editor "/PATH/TO/EDITOR"

Set up GitHub

First, create an account on Github. Then, you will need to fork these two repositories:

Go to each of their pages, and click the "Fork" button. This sets up your own copy of Dreamwidth's code which you will use to make and submit your changes.

Here's a quick overview of how the repositories will work together:

dw-free is the main repository and goes into $LJHOME
other repositories will go into $LJHOME/ext
personal config files go into $LJHOME/ext/local
All code from dw-free ($LJHOME) and the additional repos under $LJHOME/ext are automatically live when you start your server. There's no need to run any additional syncing steps anymore.

Set up dw-free

Replace the USERNAME sections of the URL with your Github username. This will make your repository an authenticated version that can push changes back to Github, as opposed to only being able to pull them.

First, clone a copy of the repository onto your machine:

$ cd ~/
$ git clone https://USERNAME@github.com/USERNAME/dw-free.git $LJHOME
$ cd $LJHOME

Now, let's make it aware of the dreamwidth repository so we can grab updates later:

$ git remote add dreamwidth https://github.com/dreamwidth/dw-free
$ git fetch dreamwidth

Finally, let's set up to sync up with the main repository's develop branch:

$ git branch -u dreamwidth/develop develop

Set up dw-nonfree

Warning: For dev servers only. Clone sites should not use dw-nonfree because it contains Dreamwidth-specific branding and files.

$ cd $LJHOME/ext
$ git clone https://USERNAME@github.com/USERNAME/dw-nonfree.git
$ cd dw-nonfree
$ git remote add dreamwidth https://github.com/dreamwidth/dw-nonfree
$ git fetch dreamwidth
$ git branch -u dreamwidth/develop develop
$ cd ../..

Database setup

You should have a local MySQL installation and know the root login to the database. To sign into MySQL as root:

$ mysql -u root -p

These MySQL commands will create your development database:

mysql> create database dw;
mysql> grant all on dw.* to 'dw'@'localhost' identified by 'somePassword';

That last line is made up of a few different sections:

grant all on dw.* - this tells MySQL to grant all permissions to the "dw" database which you created in the first line.
to 'dw'@'localhost' - this tells MySQL to grant the above permissions to a username called "dw" connecting from "localhost" (which means the same computer as the one MySQL is on). You can change "dw" to another username if you like. NB: this is 'dw'@'localhost', not 'dw@localhost' - either cut and paste, or type carefully!
identified by 'somePassword' - since this user doesn't exist yet, we can tell MySQL to create it by adding this. This tells MySQL that the "dw" user will use the password "somePassword" to connect. You probably want to change this password!

While you're here, you probably want to go ahead and create a database for TheSchwartz, too, unless you really know that you won't need it:

mysql> create database dw_schwartz;
mysql> grant all on dw_schwartz.* to 'dw'@'localhost';

(We're leaving out the "identified by" part this time because the user already exists.)

When you're done, exit out of MySQL:

mysql> quit;

Editing the config files

Next you need to configure the site configuration scripts. This is probably the most tricky part of the whole process, since there are so many things you can tweak. However, you can get by with just tweaking etc/config-local.pl and etc/config-private.pl for now.

Copy over your config files, and tell them to take priority over config files in any of the repos:

$ cd $LJHOME
$ mkdir -p ext/local/etc
$ echo "highest"  > ext/local/.dir_scope
$ cp etc/config-local.pl ext/local/etc/config-local.pl
$ vim ext/local/etc/config-local.pl

Or use your editor of choice, of course, instead of vim. Do a search for the phrase 'CHANGE THIS' which occurs once, before the block of human readable text you should use to describe your development installation.

Note that the $IS_DEV_SERVER flag is set to 1 in the template given. Be sure to set this to 0 for a production site, as there are big security issues involved with leaving the flag set to 1.

Below the 'CHANGE THIS' section (unless this is already present somewhere), add:

   $STATDOCS = "$HOME/build/static"

Next, you need to change your local variables in the config-private.pl file. This is where your passwords are configured, as well as many of the variables which define your domain. Do a search for the phrase 'CHANGETHIS'(note the slightly different spacing). You will want to change at least your $DOMAIN and the %DBINFO structure.

$ cd $LJHOME
$ cp etc/config-private.pl ext/local/etc/config-private.pl
$ chmod go-rwx ext/local/etc/config-private.pl
$ vim ext/local/etc/config-private.pl

In this file, uncomment the lines in the %DBINFO blocks (if they are commented out), and place your database password in quotes on the "pass => ," line, before the comma. For example:

   %DBINFO = (
       master => {
           pass => 'mypassword',
       },
   );

Note that there are two DBINFO blocks; get them both.

If you created the database for TheSchwartz earlier, you'll also want to configure it:

   # Schwartz DB configuration
   @THESCHWARTZ_DBS = (
           {
               dsn => 'dbi:mysql:dw_schwartz;host=localhost',
               user => 'dw',
               pass => ,     # CHANGETHIS
           },
       );

Near the bottom of the file is the line you will need to change to tell your server which domain to use.

   package DW::PRIVATE;
   
   $DOMAIN = "ljsite.com";

A note about $DOMAIN: If you are running a local server for development purposes and don't need or want a hostname, you may be tempted to just set $DOMAIN to localhost, but the server won't function correctly without a domain name. One possible workaround for this is to set it to a nonexistent domain (the default ljsite.com is fine) and add a line in your /etc/hosts file that resolves that domain name to your localhost IP (either 127.0.0.1 or ::1.) Another workaround is to install and configure dnsmasq.

The base configuration file is under source control, and is already in etc/config.pl. You should not need to change anything very much in this, but you might have to change some stuff. If you find you do have to touch this, copy it over first, then you can edit it:

$ cd $LJHOME
$ cp etc/config.pl ext/local/etc/config.pl
$ vim ext/local/etc/config.pl

Note that from here on, when any documentation says to edit etc/config*, you'll likely want to edit ext/local/etc/config-* instead.

Make sure things are working with checkconfig.pl

Now, you need to see if everything is working.

If you've setup the files as indicated above, run this command:

$ cd $LJHOME
$ bin/checkconfig.pl --no=ljconfig

If you installed everything given at the top of this page, you should find you have no missing modules. Congratulations! If you don't have all the modules, this is where you need some systems specific knowledge for your system. You will need to install whatever modules are missing. If you get well and truly stuck, find someone on IRC, the mailing list, or make a post to dw_dev.

Populate database with initial data

There are a few commands you can now run to install the database. Just run these and watch for errors.

Note: If make_system.pl says it can't give the system user admin privileges, something has gone wrong with your database population, even if there were no errors.

$ $LJHOME/bin/upgrading/update-db.pl -r --innodb
$ $LJHOME/bin/upgrading/update-db.pl -r --innodb # at least for now we have to run this twice
$ $LJHOME/bin/upgrading/update-db.pl -r --cluster=all --innodb
$ $LJHOME/bin/upgrading/update-db.pl -p

If you created your database for TheSchwartz earlier, you'll also want to populate it with its default tables:

$ mysql -u dw -p dw_schwartz < /usr/share/doc/libtheschwartz-perl/schema.sql

Create a system account:

$ $LJHOME/bin/upgrading/make_system.pl

(That step will ask you for a password for the System account. You can change it later by logging in as system, so just give it something for now.)

And load in the translation strings:

$ $LJHOME/bin/upgrading/texttool.pl load

Compile Foundation stylesheets and static files

Follow the SCSS instructions for installing compass and sass:

$ gem install sass --version 3.2.19 
$ gem install compass --version 0.12.2

Install Java, which will download about 55.9MB and take up 334MB of disk space:

$ sudo apt-get install default-jre

Then compile the stylesheets and static files for dw-free:

$ $LJHOME/bin/build-static.sh

You will see a warning for the compressor not being present, ignore it.

Configure Apache

This step will need to be done as the root user. Follow the instructions below for your version of Apache, and make sure to do service apache2 restart when you are done for your config changes to take effect.

Apache 2.4 configuration

This is the configuration you need for Ubuntu 16.04 and newer.

Add the contents of this config file (name it something like dw.conf), change the hostname, port, and directory if necessary and add to /etc/apache2/sites-available.

Create the apache log directory referenced in the configuration file:

 $ mkdir /home/dw/apache_logs

To activate the site run:

 $ sudo a2ensite dw

(Replacing dw with whatever you named your config file before the .conf bit)

Then disable the default site:

 $ sudo a2dissite 000-default

And activate the Perl Requests and Perl modules:

 $ sudo a2enmod apreq2
 $ sudo a2enmod perl

Finally, restart apache:

 $ sudo service apache2 restart

Some errors may come up here, if they do, don't be disheartened! In nearly all cases Apache provides enough information to fix these.

Apache 2.2 configuration

Apache 2.2 is no longer supported by Dreamwidth code.

Now what?

Have a look at your new DW instance

Congratulations! You now have a working (though minimal) Dreamwidth install. If you point your web browser at your server, you should see a bare-looking welcome page.

Further setup

There are lots of other articles on setting up and customizing your DW install in the DW Installation category, including:

TheSchwartz Setup - TheSchwartz is needed for a handful of features - notably comment posting
MogileFS setup
Subdomain setup
Statistics setup
Generating documentation
Allow users to register
Setting up Search
Dev Testing -- setting up the test db and running the tests

There is a list of wanted how-tos at Installation Wanted How-To, in case you need to add something to the list.

Starting development

If you haven't already, see Dev Getting Started.

@@ Line 372: / Line 372: @@
   $ sudo apt-get install default-jre
-Install YUICompressor:
- $ mkdir -p $LJHOME/ext/yuicompressor
- $ curl -L -o /home/dw/dw/ext/yuicompressor/yuicompressor.jar https://github.com/yui/yuicompressor/releases/download/v2.4.8/yuicompressor-2.4.8.jar
 Then compile the stylesheets and static files for dw-free:
   $ $LJHOME/bin/build-static.sh
+You will see a warning for the compressor not being present, ignore it.
 == Configure Apache ==