Difference between revisions of "IT/Benchfile"

From CA Greens wiki
Jump to: navigation, search
m (Email server: Postfix and Mailman)
(Slow delivery agent for AOL)
 
(25 intermediate revisions by the same user not shown)
Line 4: Line 4:
  
 
== Provisioning ==
 
== Provisioning ==
The Green Party of California rents a virtual private server (VPS) from Godmama's Forge.  It's Ubuntu-9.0 ("Karmic Koala") based on Debian-5.0 ("Squeeze") plus some work-in-progress from Debian Unstable ("Sid").  The virtual host is named wangari.cagreens.org and runs under Vmware Enterprise.
+
The Green Party of California rents a virtual private server (VPS) from Godmama's Forge.  It's Ubuntu-9.0 ("Karmic Koala") based on Debian-5.0 ("Squeeze") plus some work-in-progress from Debian Unstable ("Sid").  The virtual host is named wangari.cagreens.org and [http://www.debian-administration.org/ runs] under Vmware Enterprise.  The init scripts are in /etc/init.d/.
  
 
== Secrets and Authentication ==
 
== Secrets and Authentication ==
 
There's no root password.  Members of the "admin" user group can get a root shell by
 
There's no root password.  Members of the "admin" user group can get a root shell by
 
  sudo su -
 
  sudo su -
or run other single commands under sudo.  That's how Ubuntu does things.  Passwords such as the MySQL root user's are shown in a text file "notes" in root's home directory.
+
or run other single commands under [http://www.sudo.ws/sudo/sudo.man.html sudo].  That's how Ubuntu does things.  Passwords such as the MySQL root user's are shown in a text file "notes" in root's home directory.
 +
 
 +
Several users not involved with server administration maintain files in /w/cal appearing on the "[http://files.cagreens.org  old web site]".  They connect via SSH2.  There is no FTP nor Telnet server.  Almost everyone uses interactive keyboard authentication, plain old passwords.  A few users have installed RSA or DSA public keys.
  
 
== Domain Name Service ==
 
== Domain Name Service ==
The cagreens.org domain is served by two DNS vendors with five public name servers.  The master is "amybiehl.greens.org" running on Cameron's "bract" VPS (bract.truffula.us, bract.cagreens.org...) at Chunkhost.  The four slaves are on Dyn.com ("The Dynamic DNS") where Cameron has a "VIP" account.
+
The ''cagreens.org'' domain is served by two DNS vendors with five public name servers.  The master is ''amybiehl.greens.org'' running on Cameron's "bract" VPS (''bract.truffula.us,'' ''bract.cagreens.org''...) at Chunkhost.  The four slaves are on Dyn.com ("The Dynamic DNS") where Cameron has a "VIP" account.
  
About fifty other domains depend on a name server "cesarchavez.cagreens.org", which is running right now on Cameron's "rachel" dedicated server.  These domains include addictedtowar.com, interfaithpeacecoalition.org, sjvgreens.org, migreens.org, etc.
+
About fifty other domains depend on a name server ''cesarchavez.cagreens.org'', which is running right now on Cameron's ''rachel'' dedicated server.  These domains include ''addictedtowar.com'', ''interfaithpeacecoalition.org'', ''sjvgreens.org'', ''migreens.org'', etc.  The Green Party of California lent its domain name to this effort in about 1998, to serve the larger Green movement and commemorate Cesar Chavez.  I don't know whether there was a formal decision to do it or it seemed a "no brainer" at the time.
 +
 
 +
The ''cesarchavez.cagreens.org'' name server is not authoritative for the ''cagreens.org'' domain.  That's because it's considered poor practice (in some circles, anyway) to put a name server in its own domain.  If you don't do that, you don't need your registrar to handle a "glue record" for you.
  
 
== LAMP Stack ==
 
== LAMP Stack ==
 
  apt-get install apache2-mpm-prefork php5-pear php5-gd php5-cli libapache2-mod-php5 mysql-server mysql-client
 
  apt-get install apache2-mpm-prefork php5-pear php5-gd php5-cli libapache2-mod-php5 mysql-server mysql-client
 +
Look in the /etc/apache2/sites-enabled, /etc/php5, and /var/log/apache2 directories.
  
 
== Backups ==
 
== Backups ==
Line 24: Line 29:
  
 
== Local name service ==
 
== Local name service ==
  apt-get install maradns
+
  apt-get install [http://www.maradns.org/ maradns]
 
A lightweight caching name server, itself authoritative for no domains, listens on address 127.0.0.53.  It queries customer name servers belonging to got.net and Godmama's Forge.  It's mentioned first in our resolver configuration file /etc/resolv.conf which is obliquely mentioned (as "files") in /etc/nsswitch.conf.
 
A lightweight caching name server, itself authoritative for no domains, listens on address 127.0.0.53.  It queries customer name servers belonging to got.net and Godmama's Forge.  It's mentioned first in our resolver configuration file /etc/resolv.conf which is obliquely mentioned (as "files") in /etc/nsswitch.conf.
 
To see its configuration,
 
To see its configuration,
Line 31: Line 36:
  
 
== DNS block list ==
 
== DNS block list ==
  apt-get install rbldnsd
+
  apt-get install [http://www.corpit.ru/mjt/rbldnsd.html rbldnsd]
 
The blocklist file is /var/lib/rbldns/iparanges.  It's maintained on another server, and pushed here by rsync via ssh.  Configuration is in /etc/default/rbldnsd.  This is one of two servers of the "dnsbl.cagreens.org" blocklist.  At least four hosts refer to this DNSBL.
 
The blocklist file is /var/lib/rbldns/iparanges.  It's maintained on another server, and pushed here by rsync via ssh.  Configuration is in /etc/default/rbldnsd.  This is one of two servers of the "dnsbl.cagreens.org" blocklist.  At least four hosts refer to this DNSBL.
  
 
== Email server: Postfix and Mailman ==
 
== Email server: Postfix and Mailman ==
The "Debian Way" has Postfix' configuration files in /etc/postfix and the mapfiles they use in /etc/postfix/maps.  Tradition has the email aliases file in /etc/, so we have a symlink there pointing at the real one.   
+
=== Postfix ===
 +
[http://www.postfix.org/BASIC_CONFIGURATION_README.html Postfix] is set up the "Debian Way"Postfix' configuration files are in /etc/postfix and the mapfiles they use are in /etc/postfix/maps.  Tradition has the email aliases file in /etc/, so we have a symlink there pointing at the real one.   
 
  apt-get install postfix postfix-pcre mailman
 
  apt-get install postfix postfix-pcre mailman
 +
 +
There's a [http://www.gnu.org/s/make/manual/make.html#Introduction <code>Makefile</code>] in /etc/postfix.  Most of the time you can edit a file and type
 +
make
 +
and it will do whatever's needed.  If you're not sure, run
 +
make -n
 +
first and decide.  The Makefile also knows how to pull in updates to the map files used for spam reduction, currently from a public rsync server on Cameron's garage PC.  We use maps in Postfix mapfile formats "hash", PCRE, and CIDR.
 +
 +
In a wide terminal window,
 +
tail -f /var/log/mail.info
 +
 +
=== Mailman ===
 
Mailman runs as the "list" user.  GNU Mailman installs into its data directories and runs there.  Debian installs the software in /usr/lib/cgi-bin/mailman and /usr/lib/mailman/, with data files in /var/lib/mailman/.  They've done a nice job of separating the files their package maintainer updates from those Mailman updates as it runs, so you can do
 
Mailman runs as the "list" user.  GNU Mailman installs into its data directories and runs there.  Debian installs the software in /usr/lib/cgi-bin/mailman and /usr/lib/mailman/, with data files in /var/lib/mailman/.  They've done a nice job of separating the files their package maintainer updates from those Mailman updates as it runs, so you can do
 
  apt-get install mailman
 
  apt-get install mailman
when there's an update, without losing/breaking anything.  The /var/lib/mailman run directory has a bunch of symlinks under it, pointing into the software location.
+
when there's an update, without losing/breaking anything.  The /var/lib/mailman run directory has a bunch of symlinks under it, pointing into the software location.  There's a <code>qrunner</code> daemon with eight processes, started and stopped by <code>/var/lib/mailman/bin/mailmanctl</code>.
  
Mailman generates two files /var/lib/mailman/data/aliases and /var/lib/mailman/data/aliases.db which are mentioned in /etc/postfix/main.cf.  That's where the aliases for Mailman posting and admin functions are.
+
Mailman generates two files /var/lib/mailman/data/aliases and /var/lib/mailman/data/aliases.db which are mentioned in /etc/postfix/main.cf.  That's where the posting and admin email addresses are defined.
  
 
There's a popular script for further integration between Postfix and Mailman.  Debian distributes it with Mailman but we're not using it.  It makes Postfix depend on Python.  The main advantage would be pre-queue rejection of rejectable messages, reducing backscatter and administrative noise.
 
There's a popular script for further integration between Postfix and Mailman.  Debian distributes it with Mailman but we're not using it.  It makes Postfix depend on Python.  The main advantage would be pre-queue rejection of rejectable messages, reducing backscatter and administrative noise.
 +
 +
=== Slow delivery agent for AOL ===
 +
 +
Roughly one in ten subscribers to our Mailman lists are at AOL.com addresses.  AOL blocks incoming email from any sender whose non-deliverable rate goes over about 1% on a given day.  That's close to the noise level due to addresses aging out.  AOL users are allowed to create multiple, anonymous "screen name" identities.  An unused, forgotten, or abandoned screen name contributes noise.  A few of them at any one time are enough to get AOL to block incoming email from our server.  We need to send to AOL addresses slowly, one by one, so we can identify bad screen names from the bounces and log entries, and remove them.  It's also helpful when political adversaries sign up for Mailman lists and then report the list traffic as spam.
 +
 +
Postfix has a mechanism for transmitting some messages through a transport besides its own built-in smtp sender.  A transport named "slow" is described at the end of /etc/postfix/master.cf and mentioned ("<code>slow_destination_recipient_limit = 1</code>") in main.cf.  It's a shell script that queues messages for later delivery.  The list of addresses to be diverted is the hash:/etc/postfix/transport mapfile mentioned ("<code>transport_maps =</code>") in main.cf.  You can edit <code>transport</code> and type <code>make</code>.  The syntax is obvious.
 +
 +
A job in Cameron's crontab examines this queue every few minutes, and relays a few messages out through Cameron's "bract.truffula.us" host in Los Angeles.  A post to a large Mailman list will be broadcast everywhere within a few seconds, except for the AOL destinations which will take half an hour to two hours to trickle out.  Messages the relay won't accept (high spam score) pile up in ~cls/stuck where they must be examined and discarded or forwarded manually.  The script uses a command-line smtp sender available in Debian.
 +
 +
apt-get install msmtp
 +
 +
You are invited to write an improved version of our slow AOL transport that doesn't require an external relay and a cron job.
  
 
== Drupal and carve-outs ==
 
== Drupal and carve-outs ==

Latest revision as of 16:25, 29 January 2012

Benchfile

A "benchfile" is the book you leave in your desk drawer for your replacement in case you suddenly get a better job.

Provisioning

The Green Party of California rents a virtual private server (VPS) from Godmama's Forge. It's Ubuntu-9.0 ("Karmic Koala") based on Debian-5.0 ("Squeeze") plus some work-in-progress from Debian Unstable ("Sid"). The virtual host is named wangari.cagreens.org and runs under Vmware Enterprise. The init scripts are in /etc/init.d/.

Secrets and Authentication

There's no root password. Members of the "admin" user group can get a root shell by

sudo su -

or run other single commands under sudo. That's how Ubuntu does things. Passwords such as the MySQL root user's are shown in a text file "notes" in root's home directory.

Several users not involved with server administration maintain files in /w/cal appearing on the "old web site". They connect via SSH2. There is no FTP nor Telnet server. Almost everyone uses interactive keyboard authentication, plain old passwords. A few users have installed RSA or DSA public keys.

Domain Name Service

The cagreens.org domain is served by two DNS vendors with five public name servers. The master is amybiehl.greens.org running on Cameron's "bract" VPS (bract.truffula.us, bract.cagreens.org...) at Chunkhost. The four slaves are on Dyn.com ("The Dynamic DNS") where Cameron has a "VIP" account.

About fifty other domains depend on a name server cesarchavez.cagreens.org, which is running right now on Cameron's rachel dedicated server. These domains include addictedtowar.com, interfaithpeacecoalition.org, sjvgreens.org, migreens.org, etc. The Green Party of California lent its domain name to this effort in about 1998, to serve the larger Green movement and commemorate Cesar Chavez. I don't know whether there was a formal decision to do it or it seemed a "no brainer" at the time.

The cesarchavez.cagreens.org name server is not authoritative for the cagreens.org domain. That's because it's considered poor practice (in some circles, anyway) to put a name server in its own domain. If you don't do that, you don't need your registrar to handle a "glue record" for you.

LAMP Stack

apt-get install apache2-mpm-prefork php5-pear php5-gd php5-cli libapache2-mod-php5 mysql-server mysql-client

Look in the /etc/apache2/sites-enabled, /etc/php5, and /var/log/apache2 directories.

Backups

apt-get install rsync

Backup is a nightly rsync via ssh to a host in Cameron's garage. Authentication is by a key in ~root/.ssh/authorized_keys.

Local name service

apt-get install maradns

A lightweight caching name server, itself authoritative for no domains, listens on address 127.0.0.53. It queries customer name servers belonging to got.net and Godmama's Forge. It's mentioned first in our resolver configuration file /etc/resolv.conf which is obliquely mentioned (as "files") in /etc/nsswitch.conf. To see its configuration,

egrep -v '^#|^$' /etc/maradns/mararc

A local caching name server, also known as a "DNS forwarder", means you don't have to use the network for most name queries. For an email server that's looking up sender's PTR records, it's a real speed-up.

DNS block list

apt-get install rbldnsd

The blocklist file is /var/lib/rbldns/iparanges. It's maintained on another server, and pushed here by rsync via ssh. Configuration is in /etc/default/rbldnsd. This is one of two servers of the "dnsbl.cagreens.org" blocklist. At least four hosts refer to this DNSBL.

Email server: Postfix and Mailman

Postfix

Postfix is set up the "Debian Way". Postfix' configuration files are in /etc/postfix and the mapfiles they use are in /etc/postfix/maps. Tradition has the email aliases file in /etc/, so we have a symlink there pointing at the real one.

apt-get install postfix postfix-pcre mailman

There's a Makefile in /etc/postfix. Most of the time you can edit a file and type

make

and it will do whatever's needed. If you're not sure, run

make -n

first and decide. The Makefile also knows how to pull in updates to the map files used for spam reduction, currently from a public rsync server on Cameron's garage PC. We use maps in Postfix mapfile formats "hash", PCRE, and CIDR.

In a wide terminal window,

tail -f /var/log/mail.info

Mailman

Mailman runs as the "list" user. GNU Mailman installs into its data directories and runs there. Debian installs the software in /usr/lib/cgi-bin/mailman and /usr/lib/mailman/, with data files in /var/lib/mailman/. They've done a nice job of separating the files their package maintainer updates from those Mailman updates as it runs, so you can do

apt-get install mailman

when there's an update, without losing/breaking anything. The /var/lib/mailman run directory has a bunch of symlinks under it, pointing into the software location. There's a qrunner daemon with eight processes, started and stopped by /var/lib/mailman/bin/mailmanctl.

Mailman generates two files /var/lib/mailman/data/aliases and /var/lib/mailman/data/aliases.db which are mentioned in /etc/postfix/main.cf. That's where the posting and admin email addresses are defined.

There's a popular script for further integration between Postfix and Mailman. Debian distributes it with Mailman but we're not using it. It makes Postfix depend on Python. The main advantage would be pre-queue rejection of rejectable messages, reducing backscatter and administrative noise.

Slow delivery agent for AOL

Roughly one in ten subscribers to our Mailman lists are at AOL.com addresses. AOL blocks incoming email from any sender whose non-deliverable rate goes over about 1% on a given day. That's close to the noise level due to addresses aging out. AOL users are allowed to create multiple, anonymous "screen name" identities. An unused, forgotten, or abandoned screen name contributes noise. A few of them at any one time are enough to get AOL to block incoming email from our server. We need to send to AOL addresses slowly, one by one, so we can identify bad screen names from the bounces and log entries, and remove them. It's also helpful when political adversaries sign up for Mailman lists and then report the list traffic as spam.

Postfix has a mechanism for transmitting some messages through a transport besides its own built-in smtp sender. A transport named "slow" is described at the end of /etc/postfix/master.cf and mentioned ("slow_destination_recipient_limit = 1") in main.cf. It's a shell script that queues messages for later delivery. The list of addresses to be diverted is the hash:/etc/postfix/transport mapfile mentioned ("transport_maps =") in main.cf. You can edit transport and type make. The syntax is obvious.

A job in Cameron's crontab examines this queue every few minutes, and relays a few messages out through Cameron's "bract.truffula.us" host in Los Angeles. A post to a large Mailman list will be broadcast everywhere within a few seconds, except for the AOL destinations which will take half an hour to two hours to trickle out. Messages the relay won't accept (high spam score) pile up in ~cls/stuck where they must be examined and discarded or forwarded manually. The script uses a command-line smtp sender available in Debian.

apt-get install msmtp

You are invited to write an improved version of our slow AOL transport that doesn't require an external relay and a cron job.

Drupal and carve-outs

Most of the www.cagreens.org site is on Drupal, installed by the mbrennan user in /w/d7 Sept. 2011. Margot Brennan is a principal of the Radical Designs web design shop. Many directories and individual files are "carved out" of the www.cagreens.org virtual web server. These "carve-outs" are listed as Alias directives in the file /etc/apache2/sites-available/d7.cagreens.org. They refer to files on the pre-Drupal web site, in /w/cal/. That site appears intact at http://files.cagreens.org.

CiviCRM

A demonstration instance of CiviCRM was installed in March 2011. This instance was abandoned because no first-tier technical support was available for its lone user.