View source for Production Puppet

This page describes the steps that the Puppet configuration management system should be responsible for.

== Setup ==

Setup DW user account and add it to sudo.

 adduser dw
 echo "dw ALL=NOPASSWD: ALL" >> /etc/sudoers

Configure /etc/apt/sources.list to use our cache:

 deb http://dw-lb01:9999/ubuntu/ hardy main restricted universe
 deb-src http://dw-lb01:9999/ubuntu/ hardy main restricted universe
 
 deb http://dw-lb01:9999/ubuntu/ hardy-updates main restricted universe
 deb-src http://dw-lb01:9999/ubuntu/ hardy-updates main restricted universe
 
 deb http://dw-lb01:9999/ubuntu hardy-security main restricted universe
 deb-src http://dw-lb01:9999/ubuntu hardy-security main restricted universe

Let's do a system update and upgrade.  This may not be something we can easily do in puppet?  Or is it?  I only want to do this from time to time - i.e., only on initial install, and only manually.  (I don't want to automatically update packages every night, e.g.)

 apt-get update
 apt-get dist-upgrade

Now install packages:

 vim-perl screen mercurial subversion dh-make-perl apache2-mpm-prefork
 libgtop2-dev libapache2-mod-perl2 libapache2-request-perl
 libcaptcha-recaptcha-perl libclass-accessor-perl libclass-autouse-perl
 libclass-data-inheritable-perl libclass-trigger-perl libcompress-zlib-perl
 libcrypt-dh-perl libdbd-mysql-perl libdbi-perl libdatetime-perl
 libdigest-hmac-perl libmd5-perl libdigest-sha1-perl libgd-gd2-perl
 libgd-graph-perl libgnupg-interface-perl libhtml-parser-perl
 libhtml-tagset-perl libhtml-template-perl libio-stringy-perl
 perlmagick libimage-size-perl libwww-perl libwww-perl libmime-lite-perl
 libmime-perl libmailtools-perl libmail-gnupg-perl libmath-bigint-gmp-perl
 libnet-dns-perl libproc-process-perl librpc-xml-perl libsoap-lite-perl
 libstring-crc32-perl libtext-vcard-perl liburi-fetch-perl liburi-perl
 libunicode-maputf8-perl libxml-atom-perl libxml-rss-perl libxml-simple-perl
 mysql-client

Next up, we need to configure Apache.  This only needs to be done on webservers.  On non-webservers, we need to DISABLE "apache2" from starting at boot time.

 $ cat /etc/apache2/conf.d/dw.conf
 User dw
 Group dw
 UseCanonicalName off
 
 StartServers 3
 MaxSpareServers 5
 MinSpareServers 1
 MaxClients 2
 MaxRequestsPerChild 200
 
 DocumentRoot /home/dw/current/htdocs
 PerlSetEnv  LJHOME /home/dw/current
 PerlPassEnv LJHOME
 PerlRequire  /home/dw/current/cgi-bin/modperl.pl

There's some more Apache setup that is required to get rid of the default site and enable the request library:

 sudo a2dissite default
 sudo a2enmod apreq

Last thing is to configure the network.  We need to add the following lines in /etc/network/interfaces:

 # ensure we load our rules 
 pre-up iptables-restore < /etc/iptables.up.rules

Then we need this file created in /etc/iptables.up.rules:

 *filter
 
 # Allows all loopback (lo0) traffic and drop all 
 # traffic to 127/8 that doesn't use lo0
 -A INPUT -i lo -j ACCEPT
 -A INPUT -i ! lo -d 127.0.0.0/8 -j REJECT
 
 # Accept inbound private traffic from one of our servers
 -A INPUT -s 172.19.1.5 -j ACCEPT
 -A INPUT -s 172.19.1.84 -j ACCEPT
 -A INPUT -s 172.19.1.85 -j ACCEPT
 -A INPUT -s 172.19.1.86 -j ACCEPT
 -A INPUT -s 172.19.1.87 -j ACCEPT
 -A INPUT -s 172.19.1.88 -j ACCEPT
 -A INPUT -s 172.19.1.89 -j ACCEPT
 -A INPUT -s 172.19.1.90 -j ACCEPT
 -A INPUT -s 172.19.1.91 -j ACCEPT
 
 # Accepts all established inbound connections
 -A INPUT -m state --state ESTABLISHED,RELATED -j ACCEPT
 
 # Allows all outbound traffic
 -A OUTPUT -j ACCEPT
 
 # Reject all other inbound - default deny unless explicitly allowed policy
 -A INPUT -j REJECT
 -A FORWARD -j REJECT
 
 COMMIT

One thing to note is that this file will get changed each time I add a new server.  Also, every time it's pushed out, you have to run the iptables-restore command mentioned above.  Puppet should handle that.

The source code for the site needs to be copied, but that's something I can do with rsync.  Assuming the dw user is setup from the first step, then all I have to do is rsync the code and start Apache.  Assuming the rest of this is all good.

== Setup Round Two ==

This section contains things I didn't include in the first section, because I forgot about them!  But now that I'm setting up the new production cluster, I have to remember them.

=== other packages ===

System: sysstat dnsutils netcat telnet strace libio-aio-perl bc

libnet-openid-consumer-perl libnet-openid-server-perl

Also, there are packages that I had to build for the system.  They're in our private repository, but I'll move them to dw-ops so puppet has access to them.

Memcache: memcached

Mail: postfix

Database: mysql-server

Perlbal: lighttpd

=== snmpd ===

There's a simple /etc/snmpd/snmpd.conf that needs to be sent out.  This configuration includes a snmp v2 community writable password, so we may have to figure out some way of having a private config.

Also, /etc/default/snmpd needs to be updated to change 127.0.0.1 to the IP address of the internal interface.  (Otherwise snmpd won't ever listen to our requests for help...)

Then make sure snmpd is restarted.

=== Perlbal server ===

The perlbal machines (dfw-lb01, dfw-lb02) need to have heartbeat setup... /etc/heartbeat/{ha.cf,haresources,authkeys} and also be up and running.  (Don't worry about this one yet.  I'm still debating the best way to setup the frontend.  LVS?  Heartbeat?)

=== iptables exceptions ===

Some machines need to have exceptions on various ports.  Uh, lbs need port 80 and 443, the mailserver needs port 25...

=== Downtime Server ===

We run a very simple webserver (lighttpd) that just shows a downtime page when we have that enabled.  Downtime servers run on the perlbals, but in the future I hope to have perlbal do the downtime stuff itself.

=== Admin Server ===

We also run an Apache2 instance that gives us access to cacti, nagios, and other monitoring tools.  This runs on the admin server.

=== Memcache Server ===

Need to edit /etc/memcached.conf to use 450MB of RAM (although this might change depending on the server) plus have it listen on an internal IP only.

=== Database Server ===

Configuration file /etc/mysql/conf.d/dfw-db01.cnf (and dfw-db02.cnf).  The differences are mostly around filenames and such... easy to write a template.


[[Category: Dreamwidth Installation]]