Production Notes

This document is meant to be read by people with sysadmin experience. I'll go back at some point and clean it up, break it down into sections, etc. But for now I'm just trying to dump as much information as possible so that Matthew and Robby have some state on how things are.

Links

• Cacti: http://z.dreamwidth.org/cacti/
• Nagios: http://z.dreamwidth.org/nagios3/
• Healthy?: http://www.dreamwidth.org/admin/healthy.bml

Nagios

The Nagios setup is running on dfw-admin01 in /etc/nagios3, most of the configuration files are in /etc/nagios3/conf.d as you can imagine. You can poke around if you want to change it, it's pretty straightforward.

If you do change things, you probably want to commit them to the operations repository.

   make your changes... etc etc
   
   $ sync-back-nagios
   $ cd /root/dw-ops/nagios/conf.d
   $ hg status
   
   if everything looks good, then:
   
   $ commit -a mark -m "Some commit message."

Replace mark with matthew or alierak as appropriate.

Cacti

Most of the graphs are more or less useful. I spend a lot of time looking at dfw-lb01 which shows all of the incoming site traffic. In particular: eth0 is always the "Internet" interface, on all slices. eth1 is the "Internal/Private" interface. And lo is lo.

The only time lo is really interesting is on the dfw-lb01/dfw-lb02 machines. Look at the SSL configuration to see why, but lo is the measure of how much SSL traffic we're doing.

Traffic Flow

This summarizes the flow of traffic. There are a lot more sections that talk far more in depth about various things, but here you go...