%%To do:
%%	all program,file names in \tt
%%	references?
%%	conclusion?

\documentstyle[12pt]{article}
\title{Suns and Others; Accommodating Heterogeneity in a Sun Network}

\author{	Steven M. Romig\\
	Computer and Information Science Dept.\\
	The Ohio State University\\
	Columbus, Ohio 43210\\
	614-292-8018\\
	romig@cis.ohio-state.edu\\
\\
	Elizabeth D. Zwicky\\
	SRI International\\
	333 Ravenswood Ave.\\
	Menlo Park, CA 94025\\
	415-859-3290\\
	zwicky@erg.sri.com
}

\begin{document}

\pagestyle{empty}
\oddsidemargin 1.5in
\evensidemargin 1.5in
\textwidth 6in

\topmargin 0pt
\textheight 9in

\maketitle

\begin{abstract}
Sun provides lots of tools for administering machines in a network
environment. Many of these tools are also available from other
vendors. Unfortunately, adding machines of other
architectures to a Sun network, even when those machines apparently
come with all the right tools (NFS, NIS, rdist) is not as easy as this
might suggest. This paper discusses some of the common problems with
heterogeneous setups, and some solutions.
\end{abstract}

Many sites consist of heterogeneous environments in the sense that they
consist of computers of several different architectures, from multiple
vendors, running different software platforms.  There
are many software tools that make it easier to maintain such an
environment, including NFS, NIS and rdist.  These serve to present a
more homogeneous view of the heterogeneous systems.  For example, NFS
allows one to mount file systems back and forth, without regard for
where they came from or where they are being mounted.  These are
great, but differences between the systems often make these tools less
useful than they might be.

There are differences in 2 regards: bugs or misfeatures that are
present on one system but not others, and intentional features that
are present on one system but not the others.  This forces system
administrators to choose between taking a ``least common denominator''
view of the net (forsaking some features to accommodate all of the
systems), and reintroducing heterogeneity into a psuedo-homogeneous
world.  Sometimes even giving up features is difficult simply because
it can be hard to identify just what the least common denominator is.
Sometimes it is not clear that there is a viable least common
denominator, as witnessed by the frequently quoted statement by the
IEEE 1003.7 committee that the only standard administration tool they
could find was {\tt sync}, a least common denominator well below the
threshold of usefulness.

This paper describes some of the tools that can be used in a
heterogeneous environment and the problems that one can encounter in
using them.

\section{Remote Procedure Call}

RPC (and XDR) forms one of the main bases of communications in
heterogeneous Unix systems today.  RPC is used in many packages, such
as NFS and NIS.  Initially RPC supported 2 forms of authentication -
``none'' and ``unix''.  None really means no authentication at all (as
the name implies), and the Unix style of authentication is little
better than no authentication at all.  It is fairly easy to ``spoof''
NFS or NIS packets and pretend to be a particular user (to read or
write their files) or to provide fake NIS service if one has root
access to a UNIX system, or any access to a PC on your LAN.

Consequently, Sun introduced a third form of authentication that uses
DES to pass more secure authentication information with each RPC
interchange.  Proper use of secure RPC makes it much more difficult to
spoof RPC transactions.  

For example, secure RPC can be used with NFS.  There is an option to
mount(8) that allows one to specify that secure RPC should be used in
all NFS transactions for a particular file system.  You can also
export a file system with exportfs(8) so that it can only be used from
systems that use secure RPC.  If you export a file system with secure
RPC enabled, then it is more difficult to access that file system from
an unauthorized host than it would normally be, and this is the crux
of the problem. Any host that is not capable of the same secure RPC
that the server knows is automatically an unauthorized host, no matter
what your opinion of its trustability is. Thus, if you turn on secure
RPC you may be unable to mount filesystems on machines that you would
like to trust. In this case, the least common denominator is NFS
without secure RPC, and the choice is between increased security and
increased homogeneity.

There are differences in the details of RPC implementation as well,
which may cause difficulties even with apparently straightforward RPC
calls. For instance, the maximum allowable size of an RPC response
varies, but is not a queryable parameter. People writing RPC programs
would be well advised to be conservative.

\section{Network File System - NFS}

The Network File System (NFS) is one of the prime tools for making
large networks homogeneous. A reasonably clever naming scheme and
possibly an automounter can make every machine appear to have the same
files as every other machine. Unfortunately, there are both technical
and logistical problems involved in achieving this effect.
The technical problems have two causes. First, NFS is designed as a
stateless protocol, and there are a number of operations on the file
system that require state information. Compensating for this requires
adding in new services to go along with NFS. Second, while most
versions of UNIX are capable of hiding the facts about their network
implementations and their filesystems from their users, many of them
achieve this effect by coding assumptions into programs. These
assumptions are often invalidated by NFS mounts.

\subsection{Failures of Statelessness}

The two famous places where the design of NFS conflicts with the uses
of file systems are quotas and file locking. Sun provides {\tt rpc.quotad}
and {\tt rpc.lockd} as work-arounds to allow the use of hard quotas and file
locks. {\tt rpc.quotad} actually works - for Suns - but few other vendors
are  capable of using it. {\tt rpc.lockd}, on the other hand, is infamous
for being flaky and bug ridden, even when every machine involved is a
Sun. Neither of these problems are particularly amenable to
work-arounds. Using quotas under NFS is particularly dangerous, since
the most frequent failure mode is that quotas are silently enforced;
writes fail once you are over quota, but no error messages are
produced.

\subsection{The Invisible Becomes Visible}

Many programs need to know what size a block is. For instance, {\tt
du} attempts to provide the size of a file in kilobytes, while reading
the size of the file in blocks. Unfortunately, on many machines {\tt
stat} does not return the size of a block; thus, {\tt du} determines
block size at compile time, not at run time. NFS mounts between Suns
and Pyramids, for instance, cause {\tt du} to be off by a factor of 4,
since one {\tt du} believes that all blocks are 512 bytes, and the
other that all blocks are 2 kilobytes. This causes users who receive
disk space warnings either to be elated, because you think they have a
quarter the space they do, or furious, since you are claiming they use
4 times as much space as they think they do.

\subsection{Soft Mounts versus Hard Mounts}

NFS provides two styles of mounting things; in a hard mount, all
operations are retried until they succeed, whereas in a soft mount,
operations that fail return errors and do not retry. Soft mounts are
often preferred, because they reduce the extent to which machines hang
when other machines go down. Unfortunately, because soft mounts give
up when there are problems, and return errors which many programs do
not check for, file systems that are soft-mounted with write
permission tend to experience interesting forms of corruption. Writes
are often buffered until file closure, which means that an NFS failure
may cause a close to return errors. Many programs do not check the
results of close, and fail to take appropriate measures to safeguard
data before exiting, which can result in truncated or mangled files.
In other situations, apparently successful writes replace whole blocks
of files with nulls.

The meaning of a soft mount also changes from vendor to vendor.
Pyramid, in some versions of OSx, assumes that you simply do not care
at all about soft-mounted file systems, and fails to propagate error
messages (for instance, ``Write failed'') at all. This can and does
lead to horrible surprises, as people discover that when they saved
their file and exited the editor, it was silently truncated to 0
length. This makes soft mounts an even riskier proposition on
heterogeneous networks than on Sun-only networks, where you usually
know when your files are being eaten. The rule of thumb is that soft
mounts are only advisable for file systems being mounted read-only;
unfortunately, vendor manuals usually fail to mention that writeable
soft mounts may result in data corruption.

\subsection{Extending File System Semantics}

More and more vendors have taken to using ``unused'' permission bits
to add new features to the file system. Sun, for instance, uses the
setgid bit on directories as an indicator for how to set group
ownership on objects created in the directory. Hewlett-Packard uses
the setuid bit to determine whether or not directories should be
visible. There are other, odder file system extensions: Pyramid OSx
extends symbolic links to provide conditional symbolic links, which
have different targets depending on which OSx universe you are running
in; several separate vendors have added named pipes to the file
systems; Hewlett Packard has added access control lists.

All of these extra features may show up through NFS mounts, and have
to be interpreted or ignored by the NFS client side. Some of them fail
relatively harmlessly, like conditional symbolic links, which point
nowhere on non-Pyramid NFS clients. Some of them fail
catastrophically, like named pipes, which cause several Encore
operating systems to crash on any attempt to access them.
They all tend to make NFS mounts less transparent.

\subsection{A Maze of Twisty Little Parameters, All Different}

Most implementations of {\tt mount} that work with NFS allow you to
adjust NFS parameters such as the maximum read and write
buffer sizes, timeout period, or number of retransmissions.  We have
found that in heterogeneous environments it is often necessary to
adjust the read and write buffer sizes for NFS mounts to get the best
possible NFS performance.  The reason for this is that when a host
sends an 8 kilobyte NFS packet out on the Ethernet, it is actually
fragmented into 6 Ethernet packets.  On some hosts (such as Suns),
those packets are sent in quick succession, and can overwhelm the
receive buffers on the Ethernet interfaces on other hosts.  This
results in lost fragments, which means that the NFS packet cannot be
rebuilt, and the sending end will time out and retransmit the NFS
packet.  The timeouts can degrade NFS performance by an amazing
amount in some cases.  We have found that decreasing the read or write
buffer sizes to one or two kilobytes can noticeably improve NFS
performance (since it reduces the chances of overrunning a receive
buffer). 

The problem here is that to do the job right, you have to tune the NFS
mount attributes for every type of NFS client and server pair that you
have.  If you have 13 NFS clients (Sun 3s running SunOS 4.1, Sun 4s
running SunOS 4.1.1, HP 9000/3XXs running HP/UX 7.0 and 8.0, Pyramid
98XEs and 9825s running OSx 4.4c, etc.) and 5 types of NFS
servers, then there are 65 client-server pairs whose NFS mount
attributes you need to optimize.  

%% move hard and soft mounts here?
%%	-- nah, leave it where it is...
%% mention Sun white paper on adjusting NFS parameters?
%%	-- what white paper?

%%*	another strange group access problem that j will have to
%%*	    remind me of 

\subsection{Root and Group Access Problems}

Most NFS users are quite familiar with the way that NFS servers
usually disallow root access on file systems from NFS clients.  This
is fairly consistent on most systems now, although some still do not
provide a means of restricting root access to only specific hosts (its
either all or none).  

Sun's NFS server also effectively removes any group access rights that
the client might have if the user ID for the request is 0 (root).
This means that a program running as user root and group backup cannot
write a file that is owned by group backup, and writeable only by
group backup.  The NFS servers from some other vendors (HP, Pyramid,
Encore and BBN, among others) would allow this sort of group access.
This sort of group-write scenario comes up most often in
system administration tools, which frequently must run as root (to do
backups on one server, for example) and still retain the ability to
write a group-writeable file or directory to update a log or database.
Disallowing group-write access for NFS requests from root makes it
much more difficult to write generic sysadmin tools like backup
systems. 

The usual work-around for this sort of problem is to allow root access
to any of the hosts that would need to run those sorts of programs.
You can only specify root access for 10 hosts on Sun NFS servers, so
large sites still tend to run into problems with this. The work-around
that has been suggested by at least one vendor, is simply to export
everything with root access, because ``everybody else does it that
way.'' 

Furthermore, it is extremely easy for system administrators to confuse
the possible types of granted trust, because root access to
NFS-mounted file systems is granted on a per-file-system basis,
completely separately from  root rlogin access or tape drive access
such as is granted by {\tt /.rhosts}. The usual problem arises when {\tt tar
-cf - foo | (cd bar; tar -xf -)} does not work, while {\tt tar -cf - foo
| rsh bar-host "(cd bar; tar -xf -)"} does work. Since the operations
are effectively equivalent, people expect them to require the same
permissions, but the first requires NFS export with root, while the
second requires {\tt /.rhosts} entries. 

\subsection{Mount Points and Naming}

It is extremely easy to set up NFS so that it's hard to find files. So
easy, in fact, that I have had vendors assure me that it is impossible
to use NFS and have files appear in the same places on all the
machines. (This appeared to be a result of assuming that one of Sun's
default automounter configurations was the only possible NFS
configuration.) Since install scripts usually want to put all software
in /usr, and put home directory space someplace with a fixed name that
does not depend on the machine name, following instructions results in
17 machines with /home partitions, 17 more with /export/home
partitions, a few more with /user partitions, and heaven only knows
what else.  If you have 17 machines with /home partitions, you
obviously can't export them all so that they mount under their
original names. Furthermore, if you have some machines with /home,
some with /export/home, and some with /user, you will be lucky if you
can remember where things are supposed to be on any given machine at
any given moment. 

Throwing out the default naming scheme, and settling on one that names
partitions by host or by function will usually reduce hassles. For
instance, one of our sites names user partitions ``/n/machinem'',
where m is the 0 for the first partition on the machine, 1 for the
second, and so on; the other site uses ``/machine/letter'' where
``letter'' is ``a'' for the first partition on the machine, ``b'' for
the second, and so on. Both schemes work relatively well (although
working on both systems at the same time induces near-terminal
confusion). This scheme allows every machine to look the same, without
concealing from users what machine their home directories are on.
Other sites prefer to make directories visible to users as
/home/username, regardless of where the files live. This can be done
with a similar naming scheme, and clever use of symbolic links and/or
and automounter. 

%% Refs: Arch, depot

\subsection{Automounting}

\section{Keeping Clocks Consistent}

The proper operation of NFS requires that the clocks on the clients
and the servers be within a few seconds of each other, since otherwise
you can run into strange cache consistency problems.  Unfortunately,
the vendors do not always supply tools for maintaining clock
consistency in a networked environment.  Even when they do, they do
not always provide the same tools.  Some supply {\tt rdate}, some
supply programs that use the Time Synchronization Protocol from 4.3
BSD, some supply both, some supply neither.

The easiest solution to this problem is to retrieve the sources for
{\tt xntpd} and install that on all of your hosts.  {\tt xntpd} is a
UNIX implementation of the Network Time Protocol (RFC 1119).   

\section{NIS}

%%    Designing NIS files so the right users are in the right places
%% Unifying UIDs, simplifying automount maps; not to mention the
%% group file is in NIS but machines need different ones because
%% of the 8 group problems.
%%	-- mentioned 8 group stuff later
%% The little-used login restriction netgroup options.
%%	-- is that really relevant to this discussion?  are there
%%	   vendors that don't support it?
%% NIS on Irix is completely different for passwd file. (Bleah!!!)

\subsection{NIS Host Lookup Problems}

One has four options for host name/address lookups on Suns.  You can
use the /etc/hosts file alone (if you are not running NIS), you can
use NIS and use a hosts map that is common to your entire NIS domain,
you can use the NIS hosts map and back it up with Domain Name Service
(DNS) lookups when needed, or you can create your own copy of libc.so
that uses only DNS for host name/address lookups.  If your site is
connected to the Internet, then you pretty much need to use the third
or fourth options.

The third option works as follows.  If the hosts NIS map was created
with {\tt makedbm -b} then Sun's {\tt ypserv} will use DNS lookups to try to
resolve NIS requests that were not found in the hosts NIS map.  In
other words, if a request for {\tt foo.bar} is made, and that host is not
found in the hosts NIS map, DNS will be used to try to resolve that
hostname and get information about it.  This provides a convenient
mechanism to make use of the DNS without having to change much about
your systems, and it works well in a homogeneous environment where all
of the {\tt ypserv} processes operate in that fashion.

It fails in some heterogeneous environments where the {\tt ypserv} processes
do not necessarily act in the same manner.  For example, if a Sun NIS
client happens to bind to an HP {\tt ypserv}, then host name lookups are
only resolved against the host's NIS map, and not against the DNS as
well.  This means that whether you will be able to resolve hostnames
that are not in the local hosts NIS map will depend on whether your
client is bound to a Sun {\tt ypserv} or not.  

The fourth option solves this problem by forcing all hostname lookups
to go through DNS, avoiding NIS entirely.  This is a real solution to
the problem, and Sun does provide the means for accomplishing this
(through the {\tt shlib.etc} package that can optionally be installed on
your system from the distribution CD).

\subsection{NIS Compatibility Problems}

As another example of NIS compatibility problems, consider shadow
passwords.  Suns allow you to use a shadow password file
and to propagate that information as a NIS map.
Some other vendors support shadow password files in other ways, or not
at all.  On top of the complication of maintaining password
information consistent between the set of hosts that support NIS and
those that do not, we now have the added complications of dealing with
the various ways of supporting shadow passwords, so that we might end
up with some set of hosts that all get their normal password
information through NIS but the shadow password information from
different sources.  Additional complexity is incurred when you try to
integrate password aging into the picture.

An interesting compatibility problem from the users' point of view is
that Sun uses the {\tt passwd} command to modify passwords in NIS or
on the local machine, as appropriate. Other vendors that provide NIS
follow Sun's previous (and annoying) behavior of using {\tt yppasswd}
to change NIS passwords, and {\tt passwd} to change local passwords. 

\subsection{Unifying Data For NIS}

In order for NFS to work correctly, the information about users and
groups must be consistent throughout the network. This is primarily an
administrator's configuration problem, but it presents new challenges
in a multi-vendor environment. For instance, group 1 is daemon by
default on Suns, but is ``reserved for 'other''' under SGI's Irix.
Creating a consistent group file is not that hard in itself, but you
also need to make sure that files retain the correct group
associations when you change the group file.  If you change the kmem
group from group ID 5 to 10, you also have to ensure that the {\tt
/dev/kmem} device and any programs that are setgid kmem are also
changed accordingly. Since most vendors consider group ids under 100
reserved, they feel free to make use of them however they choose.

The addition of multiple vendors to a network can increase the number
of groups that your users need to belong to.  You have to take care
then not to exceed the minimum of the group membership limits on your
hosts, since otherwise users who are in 11 groups may have problems
using the hosts that only support 8 groups. These problems will
include mysterious failures of NFS access, as NFS servers that support
only 8 groups are unable to validate users in more than 8 groups. This
is primarily a problem between vendors, but may also be a problem
between operating system releases; SunOS 3.5 supports only 8 groups,
while 4.0 and above support up to 16. Users who need more than 8
groups may be forced to resort to having multiple accounts, or using
passwords on groups (a procedure which, while theoretically possible,
is clumsy and unpleasant at best).

%%Network printing
%% Try Irix, which has lp and lpr, and getting them to talk to each
%% other let alone remote queues is unfun. Try new permissions
%% (again!) via hosts.lpd (is network problem, not really het.
%% problem). Printing is pretty disgusting, really.
%% -- I'll let you write about printing.

\section{Rdist}

It is often useful to be able to distribute files from system to
system using tools such as {\tt rdist}.  Some vendors supply {\tt
rdist}, which is convenient.  The sources are also available for free
from a number of locations, which means that you can install it on the
hosts that do not come supplied with {\tt rdist} by the vendor.  But
one can run into interesting problems regardless.

{\tt Rdist} requires that it be installed with the same pathname on
every host - if it is not, then an {\tt rdist} running on one host
will not be able to start an {\tt rdist} process on remote hosts that
use a different pathname.  This is easily fixable once you know that
the problem exists, although the cause of your problem may not be
readily apparent when {\tt rdist} fails.

When {\tt rdist} establishes a connection to an {\tt rdist} on a
remote host, it also checks its version number and aborts the
connection if the numbers do not match, to avoid protocol errors.
Unfortunately, some vendors ship different versions of {\tt rdist}
which makes it impossible to use it as is for updating files.  It is
easy to work around this by retrieving the {\tt rdist} sources from
one of its locations and installing the same version on all of your
systems.

%% rdist doesn't cut it for some things that you want to do
%% - such as?  relevant to the topic?

\section{Dump, Restore and Tape Drives}

One of the authors has been working on a generic version of the BSD
{\tt dump} program that supports (among other features) end of tape
(EOT) handling.  One of the interesting results of this was
discovering that the tape devices on our various systems handle EOT in
myriad ways.  Some Sun tape devices (1/2 inch and 8mm) usually return 0
on the write that hits EOT.  Sun 1/4 inch tapes return an error,
though bugs in the driver lead to indeterminate results (sometimes the
write succeeds, but error is returned, sometimes it returns an error,
and the write failed.  Sometimes it writes a partial block and returns
a partial count without failing.  This makes it hard enough to support
EOT handling on Suns, but consider that other vendors handle EOT in
other ways.  For example, the 1/2 inch tape devices on Pyramids return
ENOSPC (No space left on device) at EOT.

There are also the well known problems with byte ordering on dump
tapes.  Robust versions of {\tt restore} exist, and will automatically
handle byte and quad swapping to allow you to read most dump tapes.
However, you can still run into problems when trying to read tapes
written with {\tt dump} on systems from other vendors, especially when
the {\tt restore} program you are using does not support byte or quad
swapping. Byte ordering and quad swapping aside, the {\tt dump} format
is not guaranteed to be portable, so you may not be able to restore
files created by another vendor's dump even if the byte ordering is
the same.

There are other complications as well.  Not all vendors ship {\tt
dump}. Irix gives you {\tt BRU}, which is better than {\tt tar} but
which is not generally available on other systems unless you buy it.
HP supplies a version of {\tt dump} that does not support their file system
extensions (access control lists and context dependent files).  They
do supply a program called {\tt fbackup}, but it is not available for
other platforms (so you cannot restore those backups onto other
systems). 

\section{Differences in Programs}

Another source of anomalies is in how programs behave on various
systems.  Some of this is expected - for example, System V {\tt ps}
takes different arguments than the BSD {\tt ps}.  The differences are
not always as simple as knowing whether a system is based on system V
or BSD UNIX.  How do you use find to search an entire file system
without crossing into other file systems?  On a Sun, you use {\tt find
path -xdev}. On an HP, you would use {\tt find path -hidden
-mountstop}. Some versions do not even have options like {\tt -xdev}
or {\tt -depth}.

Differences like these make it difficult to write shell scripts that
will run correctly on more than one platform.  Another example is the
{\tt at} command.  On some systems (Pyramid OSx) {\tt at} assumes that
commands given to it through stdin are to be executed with {\tt
/bin/sh}.  HP's {\tt at} uses /bin/sh by default, but also allows you
to specify it through the SHELL environment variable.  Sun's work the
same way, but also allow you to use an option on the command line to
override the normal behavior.  It would seem safe to use Bourne shell
scripts with {\tt at}, but you would also need to ensure that SHELL is
set correctly as well.  Details like these are not always obvious and
further increase the difficulty in creating portable scripts.

In some cases, a sufficiently determined administrator can solve these
problems by buying source from one vendor, or obtaining Berkeley or
GNU project sources, and installing the same program everywhere. For
programs like {\tt rdist} and {\tt lpd} which are not too deeply
intertwined with the rest of the operating system, and which must
interoperate in order to function usefully, this is a reasonably
achievable solution. On the other hand, it is a maintenance nightmare,
as you have to recompile and fix dozens of utilities for every
operating system upgrade. Some programs are also extremely difficult
to port.

A related problem is how to figure out what kind of system that you
are running on.  It is all well and good to be able to use the {\tt sun3}
command to tell whether you are on a sun3 or not, but it is difficult
to find out what type of system you are on unless you enumerate all of
the possibilities and run the appropriate commands (which may not even
exist). System V and its look alikes provide {\tt uname}, but not all
versions of {\tt uname} provide the same information. Furthermore, you
may be interested in distinctions that the vendor does not make - for
instance, {tt uname} will not tell you whether a machine is a diskless
client, or who is responsible for it.

Programs like {\tt configure} attempt to make guesses based on known
programs like {\tt uname} and known peculiarities of given systems.
Many sites also have local systems for maintaining the type
information that is useful to them. OSU CIS maintains a file in
/etc that specifies host type information; SRI ITAD maintains extra
parts of the host information in their name server records. Such systems are
relatively easy to devise, but they form strong limits on the
portability of software between sites. Furthermore, they tend to
evolve and shift over time, resulting in situations like that at SRI
ITAD where most of the programs pull host type information from the
name server, but most of the staff pull host type information out of
comment fields in the NIS map for ethers. The two databases contain
overlapping but non-identical fields, and are approximately equally
correct, at wrong for 10 percent or more of the hosts. Finally, only
locally written software is capable of using the information. Software
written elsewhere will still have to rely on either asking the
installer, or making educated guesses.

\section{The Users}

A heterogeneous environment can be extremely confusing to the users.
Part of this lies in the differences between the systems: if you are
used to an HP system, a Sun feels strange (and visa versa).  These
problems can be exacerbated if you support a more homogeneous heterogeneous
system by giving everyone a single home directory that they use on all
of the different platforms.  Now the users have to take care to use
the right binaries on each system.  Rather than maintain a single
{\tt ~/bin} directory, sites like this tend to advocate separate bin
directories for each architecture that a user deals with (for
instance, {\tt ~/bin.sun3}, {\tt ~/bin.sun4}, {\tt ~/bin.pyr} and so on).  It is
easy to set up one's dot files to accommodate this, but the methods
that we are familiar with rely on site-specific mechanisms to provide
a consistent means for the users to find their host type when they
login. {\tt Tcsh} provides a built-in variable, {\tt HOSTTYPE}, which solves
the problem nicely if you happen to be running tcsh on all your hosts;
if you aren't, it merely moves the problem to one of figuring out
whether or not you have HOSTTYPE available to you.

For users that have to build multiple versions of a software package
on each of several architectures, we generally suggest that they
create separate object directories to store the architecture
specific files in.  This can be done fairly painlessly with either
makefiles or with liberal use of symbolic links to make a common set
of sources appear in several directories.

\end{document}
