I believe in having a completely distributed system. This means
each machine contains a copy of the operating system. Installing the
OS on each machine manually is cumbersome. To optimise this process,
what I do is first set up and install one machine exactly the way I
want to. I then create a tar and gzipped file of the entire system
and place it on a bootable CD-ROM which I then clone on each machine
in my cluster.
The commands I use to create the tar file are as follows:
tar -czvlps --same-owner --atime-preserve -f /maxa/slash.tgz /
I use a script called go that takes a machine
number as its argument and untars the slash.tgz file on the
CD-ROM and replaces the hostname and IP address in the appropriate
locations. A version of the go script and the input files for
it can be accessed at:
http://www.ram.org/computing/linux/linux/cluster/. This script
will have to be edited based on your cluster design.
To make this work, I use Martin Purschke's Custom Rescue Disk
http://www.phenix.bnl.gov/~purschke/RescueCD/) to create a
bootable CD image containing the .tgz file representing the cloned
system, as well as the go script and other associated files.
This is burned onto a CD-ROM.
There are several documents that describe how to create your own
custom bootable CD, including the Linux Bootdisk HOWTO (
http://www.linuxdoc.org/HOWTO/Bootdisk-HOWTO/), which also
contains links to other pre-made boot/root disks.
Thus you have a system where all you have to do is insert a CDROM,
turn on the machine, have a cup of coffee (or a can of coke) and come
back to see a full clone. You then repeat this process for as many
machines as you have. This procedure has worked extremely well for me
and if you have someone else actually doing the work (of inserting and
removing CD-ROMs) then it's ideal. In my system, I specify the IP
address by specifying the number of the machine, but this could be
completely automated through the use of DHCP.
http://www.informatik.uni-koeln.de/fai/) is an automated
system to install a Debian GNU/Linux operating system on a PC
cluster. You can take one or more virgin PCs, turn on the power and
after a few minutes Linux is installed, configured and running on the
whole cluster, without any interaction necessary.
http://systemimager.org) is software that automates Linux
installs, software distribution, and production deployment.
DHCP vs. hard-coded IP addresses
If you have DHCP set up, then you don't need to reset the IP
address and that part of it can be removed from the go
DHCP has the advantage that you don't muck around with IP
addresses at all provided the DHCP server is configured
appropriately. It has the disadvantage that it relies on a centralised
server (and like I said, I tend to distribute things as much as
possible). Also, linking hardware ethernet addresses to IP addresses can
make it inconvenient if you wish to replace machines or change
The hardware in general has worked really well for us. Specific
issues are listed below:
The AMD dual 1.2 GHz machines run really hot. Two of them in a
room increase the temperature significantly. Thus while they might be
okay as desktops, the cooling and power consumption when using them as
part of a large cluster is a consideration. The AMD Palmino
configuration described previously seems to work really well, but I
definitely recommend getting two fans in the case--this solved all our
Some tar executables apparently don't create a tar file the nice
way they're supposed to (especially in terms of referencing and
de-referencing symbolic links). The solution to this I've found is to
use a tar executable that does, like the one from RedHat 7.0.