5. Compiling and Invoking `TOP-C' Applications

A `TOP-C' application can be compiled once, and then linked to your choice of a run-time library for either a sequential, distributed memory or shared memory architecture. The two shell scripts `bin/topcc' and `bin/topc++' are used instead of `gcc' and `g++' (or other C/C++ compilers).

5.1 Compiling TOP-C Applications
5.2 Command Line Options in TOP-C Applications
5.3 Invoking a TOP-C Application in Sequential Memory
5.4 Invoking a TOP-C Application in Distributed Memory
5.5 Invoking a TOP-C Application in Shared Memory

5.1 Compiling TOP-C Applications

The TOP-C application file must contain

#include <topc.h>

It must make calls to

TOPC_init(...); TOPC_master_slave(...); TOPC_finalize();

as describe in 3.1.1 Structure of a TOP-C Program. The application file is compiled by one of:

topcc --seq myfile.c topcc --mpi myfile.c topcc --pthread myfile.c

according to whether the target computer architecture will be sequential (`--seq': single processor), distributed memory (`--mpi': networked CPU's), or shared memory (`--pthread': SMP or other shared memory architecture with a POSIX threads interface). topcc is a substitute for cc or gcc, and creates an `a.out' file. (Similarly, topc++ exists as a substitute for c++ or g++.) There are man files,

`doc/topcc.1', `doc,topc++.1'

with further information on topcc and topc++. If installed, man topcc and man topc++ exist.

The same object file may be relinked to use different `TOP-C' memory models without recompiling the object file.

topcc -c myapp.c topcc --seq -o myapp-seq myapp.o topcc --mpi -o myapp-mpi myapp.o

For large applications, it may be preferable to directly invoke the `TOP-C' libraries and include files. For such situations, topc-config exists. The following is exactly equivalent to topcc --mpi myfile.c (assuming you configured `TOP-C' using gcc).

gcc `topc-config --cflags` --mpi myfile.c `topc-config --libs`

Type topc-config with no arguments for a full set of command line options.

For the rest of this chapter, we standardize our description for topcc. However, topc++ is equally valid wherever topcc is mentioned.

5.2 Command Line Options in TOP-C Applications

TOP-C searches for TOP-C parameters in the following locations, in order:

1. values of variables TOPC_OPT_xxx in the application code before TOPC_init()
2. the file `.topcrc' in the home directory
3. the environment variable TOPC_OPTS
4. arguments on the command line in the form --TOPC-xxx

For a brief synopsis of application command line options, type:

topcc myapp.c ./a.out --TOPC-help [ OR FOR MORE INFORMATION: ./a.out --TOPC-help --TOPC-verbose ]

Currently, this will display the following.

Usage: ./a.out [TOPC_OPTION ...] [OTHER_OPTIONS ...] where TOPC_OPTION is one of: --TOPC-help[=<0/1>] display this message [default: false] --TOPC-stats[=<0/1>] print stats before & after [default: false] --TOPC-verbose[=<0/1>] set verbose mode [default: false] --TOPC-num-slaves=<int> number of slaves (sys-defined default) [default: -1] --TOPC-aggregated-tasks=<int> number of tasks to aggregate [default: 1] --TOPC-slave-wait=<int> secs before slave starts (use w/ gdb attach) [default: 0] --TOPC-slave-timeout=<int> dist mem: secs to die if no msgs, 0=never [default: 1800] --TOPC-trace=<int> trace (0: notrace, 1: trace, 2: user trace fncs.) [default: 2] --TOPC-procgroup=<string> procgroup file (--mpi) [default: "./procgroup"] --TOPC-topc-log=<string> NOT IMPL: log file for TOPC output ("-" = stdout) [default: "-"] --TOPC-safety=<int> [0..20]: higher turns off optimizations, (try with --TOPC-verbose) [default: 0]

For each option, `--TOPC-PARAM', there is a corresponding C/C++ variable, TOPC_OPT_PARAM. This variable is of type int or (char *). If the application program sets the value before a call to TOPC_init(), these variables will act as defaults. For example, to turn off tracing by default, write:

int main( int argc, char *argv[] ) { TOPC_OPT_trace = 0; TOPC_init( &argc, &argv ); ... }

The option `--TOPC-trace' causes the task input and task output to be traced and printed as they are passed across the network. The action of a task is also printed. If an application is called with `--TOPC-trace=2' (default) and if the variables TOPC_OPT_trace_input and TOPC_OPT_trace_result are set to pointers to functions then those functions are called, and they may print additional information. TOPC_OPT_trace_input must be set to a function of one variable: void * input. TOPC_OPT_trace_result must be set to a function of two variables: void * input, void * output. When using C++, the function pointers must be cast to TOPC_trace_input_ptr or TOPC_trace_result_ptr before being assigned to TOPC_OPT_trace_input or TOPC_OPT_trace_result, respectively. For an example, see `examples/parfactor.c' in the TOP-C distribution.

The option `--TOPC-stats' prints statistics (running times, etc.) and information about the conditions of an invocation of a TOP-C application before and after a run. The option `--TOPC-verbose' (set by default) displays TOP-C warnings. With --TOPC-help, it provides additional information.

For the usage of `--TOPC-procgroup', see 5.4 Invoking a TOP-C Application in Distributed Memory. That section also explains on what hosts the slaves run when `--TOPC-num-slaves' indicates a different number of slaves than the `procgroup' file.

For the usage of `--TOPC-aggregated-tasks', see 7.3 Improving Performance. For the usage of `--TOPC-slave-wait', see 6.5 Stepping Through a Slave Process with `gdb'. For the usage of `--TOPC-safety', see 5.2 Command Line Options in TOP-C Applications. For the usage of the other options, see the Concept Index.

`TOP-C' recognizes -- as terminating all option processing, according to standard UNIX conventions.

5.3 Invoking a TOP-C Application in Sequential Memory

For example,

topcc --seq -g -O0 myfile.c

compiles a sequential version for debugging using gdb (see section `Summary' in The GNU debugger), for example. This is usually a first step in debugging a TOP-C application, since sequential debugging is easier than parallel debugging.

5.4 Invoking a TOP-C Application in Distributed Memory

Linking using the `--mpi' option (default) allows an application to execute using a distributed memory model of networked processors. The `TOP-C' distribution includes a subset MPI(1) implementation `MPINU', sufficient to run `TOP-C' applications.

topcc --mpi myapp.c ./a.out

The application must then create the remote slave processes at runtime. If you use `MPINU' (the default configuration of `TOP-C', then the remote slave processes are specified by a `procgroup' file. Otherwise, the startup mechanism depends on your `MPI' implementation.

5.4.1 Writing Procgroup Files for `MPINU'
5.4.2 If Slaves Fail to Start

5.4.1 Writing Procgroup Files for `MPINU'

`MPINU' is built into the default configuration of the `TOP-C' library and uses the `procgroup' mechanism to specify slave processes. (If you prefer to use a different `MPI' dialect, `TOP-C' will work, but `src/Makefile.in' must be modified, and that `MPI' dialect may use a different mechanism for introducing slave processes.)

When the application binary is executed under the default, it looks at the current directory for a file,

`procgroup'

The procgroup file determines the number and location of the slave processes. The procgroup file need only be visible from the master process. If one prefers, one can specify an alternate procgroup file via the syntax as in the following example:

./a.out --TOPC-procgroup=../myprocgroup.big

The `TOP-C' distribution includes a file `bin/procgroup' as an example of the procgroup format. The file must contain a line:

local 0

for the master process. It must also contain a line for each slave process in one of the following forms:

hostname 1 full_pathname hostname 1 - hostname 1 ./relative_pathname hostname 1 ../relative_pathname

where hostname is the remote host. The pathname - means to use the same pathname for the slave on the remote host as for the master on the current host. A relative pathname, such as ./a.out or ../new_arch/a.out, specifies a pathname relative to the pathname of the binary of the master on the current host.

Most of the time, it is simplest to just include a full pathname or else - in the `procgroup' file. The relative pathnames are useful for a shared filesystem with binaries compiled for different architectures. For example, the procgroup file might include relative paths `../sparc/a.out', `../alpha/a.out' and `../linux/a.out'. If you invoke `full_path/sparc/a.out', this will yield a master running `full_path/sparc/a.out' and three slaves running each of the three architectures.

The full principles are as follows. Let SLAVE_PATH be the path of the slave as given in the procgroup file, and let MASTER_DIR be the directory of the master process as invoked on on the command line.

SLAVE_PATH absolute: slave binary image is SLAVE_PATH SLAVE_PATH relative and MASTER_DIR absolute: slave binary image is MASTER_DIR/SLAVE_PATH SLAVE_PATH relative and MASTER_DIR relative: slave binary image is $PWD/MASTER_DIR/SLAVE_PATH SLAVE_PATH is - and master process invoked on command line as MASTER_BIN: slave binary image is MASTER_BIN (if MASTER_BIN is absolute path) or $PWD/MASTER_BIN (if MASTER_BIN is relative path) If the procgroup line contains command line arguments, those command line arguments are passed to the slave application as its first arguments, and any arguments on the master command line are appended to the list of arguments.

TOP-C assumes that your application does not change the working directory before calling TOPC_init().

By default, `TOP-C' uses the procgroup file in the current directory. You can choose an explicit procgroup file via a program variable, TOPC_OPT_procgroup="/project/myprocgroup";, or via a command-line option, /project/sparc/app --TOPC-procgroup=/project/myprocgroup. See section 5.2 Command Line Options in TOP-C Applications.

If the command line option `--TOPC-num-slaves=int' is given, and if int is less than the number of slaves in the `procgroup' file, then `TOP-C' will use the first int slaves from the file. If int is more than the number of slaves in the `procgroup' file, then `TOP-C' will use all of the given slaves, and then create additional processes on the remote hosts, by returning to the beginning of the `procgroup' file and re-reading the list of slave host/binaries until int slaves have been created in all.

It is recommended to use only localhost during initial development.

The environment variable, SSH (default value ssh) is used to invoke the remote host. If, for example, your site uses rsh instead of ssh, the following code, when executed before TOPC_init() will produce this effect.

putenv("SSH=rsh");

Alternatively, type SSH=rsh (sh/bash, etc.) or setenv SSH rsh (csh/tcsh, etc.) in the UNIX shell before invoking the TOP-C application.

5.4.2 If Slaves Fail to Start

If some slave processes start, but not others, then try executing the following simple program.

#include "topc.h" int main(int argc, char *argv) { char host[100]; printf("%s connecting ...\n", gethostname(host, 99)); TOPC_init(&argc, &argv); printf("... %s connected.\n", gethostname(host, 99)); TOPC_finalize(); }

If the slave processes fail to start up or fail to respond to the master and if you are using `MPINU' (default configuration of `TOP-C', one other debugging resource is available. If an application fails to start up, then `TOP-C' leaves in the `/tmp' directory a file

`/tmp/mpinu-ssh.$$'

where $$ is the process id. The file shows the commands that `TOP-C/mpinu' tried to use to start up the slave process. By examining and even manually executing those commands from the terminal, one can often deduce the difficulty in creating the slave processes.

Test the ability of your computer facility to execute remote processes without passwords by typing: ssh REMOTE_HOST pwd. If the problem is that ssh is not working, try setting the environment variable SSH to ssh or other site-specific setting. See section 5.4.1 Writing Procgroup Files for `MPINU'.

If you are using ssh (default if the environment variable SSH is not set), and if it requires a password then the following commands within UNIX may allow ssh to operate on your local cluster without passwords.

ssh-keygen -t dsa [accept default values] ssh-keygen -t rsa [accept default values] cat ~/.ssh/id*.pub >> ~/.ssh/authorized_keys

For security, be sure that ~/.ssh has no read permission for other users.

5.5 Invoking a TOP-C Application in Shared Memory

Linking using the `--pthread' option allows an application to execute using POSIX threads. Note that the `TOP-C' memory model for shared memory has some small variations from the distributed memory model. The largest potential source of incompatibility is if your DoTask() routine modifies any global or static variables. If so, such variables will need to be declared thread private. Check your source code to see if this affects you.

Second, if you encounter insufficient performance, you may consider experimenting with fine grain thread parallelism. The default `TOP-C' algorithm for shared memory allows UpdateSharedData() to begin executing only after each current invocation of DoTask() completes. This can be modified by an application for finer grain parallelism. See section Optimizing TOP-C Code for the Shared Memory Model, for details in either of the above cases.

Note also that while a `TOP-C' application object file can usually be linked using an arbitrary `TOP-C' memory model without recompiling, there are some circumstances where you may first need to recompile the application source using topcc --pthread.