---

Manual Reference Pages  - UPCRUN (1)

---

upcrun - a portable parallel job launcher for UPC programs

CONTENTS

Synopsis
Options

upcrun [options] program-name [ program-arguments ... ]

-h -? -help
	See this message
-help-gasnet [network_name]
	See the GASNet documentation, or the GASNet documentation for a particular network conduit.
-n <num>
	Spawn <num> UPC threads. If the executable was compiled for dynamic thread count then this flag is required. When compiled for a static thread count, this flag is optional, but must agree with the compiled-in setting if present. (-np is a synonym for -n)
-nodes <num>
	Specifies the number of compute nodes to use for execution. See the THREAD LAYOUT section of the man page for more details. (-N is a synonym for -nodes)
-c -cpus-per-node <num>
	Specifies the number of UPC threads to execute on each compute node. See the THREAD LAYOUT section of the man page for more details.
-p -pthreads <num>
	If the UPC executable was compiled with pthreads support then this option overrides the compiled-in default number of pthreads per process. A value of zero resets to the compiled-in default or the UPC_PTHREADS_PER_PROC environment variable. This flag is not legal with an executable not compiled with pthreads support.
-shared-heap <sz>
	Requests the given amount of shared memory (per UPC thread). Units of <sz> default to megabytes; use ’2GB’ to request 2 gigabytes.
-heap-offset <sz>
	Request the given offset between the starting addresses of the regular and shared heaps. Units of <sz> default to megabytes; use ’2GB’ to request 2 gigabytes.
-[no]trace
	Enable tracing. This option is only effective if the executable was build against a runtime with tracing enabled.
-tracefile <file>
	Override the default destination for tracing output. If present, an optional ‘%’ character in the filename will expand into a distinct integer for each process. This option implies -trace.
-q -[no]quiet
	Suppress initialization messages from UPC runtime.
-v -[no]verbose
	Verbose: display commands invoked, environment variables set and other diagnostics.
-t -[no]show
	Testing: don’t actually start the job, just output the system commands that would have been used to do so.
-i -[no]info
	Display useful information about the executable and exit
-version
	Show version information for upcrun

upcrun uses a site-wide ‘upcrun.conf’ file to get some of its settings. You may override any of the settings found in the global upcrun.conf file by creating a ‘.upcrunrc’ file in your $HOME directory.

To specify flags to pass to upcrun each time it is invoked set ‘default_options’:

default_options = -n 4 -shared-heap 256MB

To specify flags to pass to upcrun only when it is invoked for an executable compiled for a given network, set the ‘<conduit>_options’ parameter:

mpi_options = -shared-heap 192MB

To specify the number of UPC threads to start on each node of a cluster of SMPs, set the ‘default_cpus_per_node’ paramater. If this value is unset, or is set to zero, then upcrun will rely on the underlying spawner to correctly layout processes on nodes. When running a UPC executable that was not compiled to use pthreads, this default is likely to be acceptible. However, when using pthreads the underlying spawner is unaware of the number of threads per process and therefore may start more UPC threads per node than available CPUs. Your site-wide ‘upcrun.conf’ should have a correct setting for this parameter, but if using pthreads you are strongly encouraged to verify the value is correct.

default_cpus_per_node = 2

To modify the command line that upcrun uses to run applications for a particular network type (a.k.a. conduit), set the ‘<conduit>_spawn’ and/or ‘<conduit>_spawn_nodes’ parameters, where <conduit> is one of the network types listed in ‘upcc -version’. These parameters are templates used to execute a command to launch the necesssary processes. When writing templates, the following subsitutions are available:

%N	number of processes to launch (might not equal UPC threads when using pthreads)
%M	number of "nodes" on which to launch processes
%P	program file
%A	program arguments
%C	alias for "%P %A"
%D	current working directory
%L	UPC-specific environment variable names (comma separated list) which the spawner will propogate to the application
%V	expands to "-v" if the user passed -v to upcrun, or to nothing otherwise
%%	expands to a single % character

Arguments are split on whitespace, but single- or double-quotes may be used to prevent this. Backslash ‘#146; is not special.

The ‘<conduit>_spawn’ parameter is a template for launching a spawner similar to most implementations of ‘mpirun’ -- a single parameter specifies the number of processes to launch and these are assigned to available nodes in an implementation-specific manner. This spawner is used when the number of requested nodes and number of cpus per node are both unspecified (or has been set to zero). Examples include:

mpi_spawn = gasnetrun_mpi -n %N -E %L %P %A
elan_spawn = prun -n %N %P %A
smp_spawn = %P %A

The ‘<conduit>_spawn_nodes’ parameter is a template for launching a spawner such as ‘prun’ or ‘poe’, in which one specifies both the number of processes to launch and the number of nodes to use. Given sufficient information, upcrun will compute the number of processes and nodes required as described in the section THREAD LAYOUT. For example:

elan_spawn_nodes = prun -N %M -n %N %P %A

The UPCRUN_FLAGS environment variable can be set to pass any flags/arguments that you wish to use for every invocation of upcrun. This is in addition to the ‘default_options’ parameter described above.

The UPC_<CONDUIT>_SPAWNCMD and UPC_<CONDUIT>_SPAWN_NODESCMD environment variables can be set to override the spawner templates found in the upcrun.conf and .upccrc files.

The ’udp’ network type allows UPC programs to run on any machine that supports the ubiquitous UDP network layer. But it needs to be told which machines are involved in the job in order to bootstrap the job. There are three methods for doing this:

1.	If you wish to run a UDP job entirely on your local machine, simply set the GASNET_SPAWNFN environment variable to "L" (for ’localhost’). This is not a high-performance method (all communications between threads involve localhost UDP send/receives), but can be useful if you do not wish to use pthreads (which use faster shared memory for communication).
2.	To run on multiple nodes, set the GASNET_SPAWNFN environment variable to "S", and SSH_SERVERS to a space-separated list of hostnames (with the first host corresponding to thread 0’s host, the second to thread 1’s, etc.). Use duplicate host names to put multiple threads on the same node. This method uses ssh (by default: rsh can also be used by setting SSH_CMD to ’rsh’) to connect to the nodes and set up the job. Logging into the nodes must not require an interactive password, and so you must use a method that allows this (ssh-agent, or passwordless ssh keys, or a passwordless rsh setup). For information on using ssh-agent (generally the most secure of these methods), see http://upc.lbl.gov/docs/user/sshagent.html
3.	You may use your own custom spawner to launch UDP jobs. See the README.udp file for details.

The layout of UPC threads to network nodes depends on the setings of two parameters, the ‘cpus_per_node’ and the -nodes flag. The ‘cpus_per_node’ setting comes from the -cpus_per_node flag if present, or else from the ‘default_cpus_per_node’ setting in ‘upcrun.conf’ or ‘$HOME/.upccrc’. There are three distinct mechanisms for thread layout depending on the values of ‘cpus_per_node’ and the -nodes flag.

If ‘cpus_per_node’ and -nodes are both set to zero (or are not set) then the layout of UPC threads is left to the underlying mpirun-style spawner (the ‘<conduit>_spawn’ configuration setting or the UPC_<CONDUIT>_SPAWNCMD environment variable). This is the only case in which the mpirun-style spawner is used by default. If the executable has been compiled with pthreads enabled, the UPC threads are first grouped into processes which are in turn laid out by the spawner. With the possible exception of the last process, each such process includes the same number of pthreads. This number defaults to the value compiled-in to the executable. This can be overridden at upcrun time by either the UPC_PTHREADS_PER_PROC evironment variable or with the -pthreads flag. For unusual cases, the UPC_PTHREADS_MAP environment variable can be used to specify the grouping of threads into processes.

If ‘cpus_per_node’ is zero while -nodes has a non-zero value then UPC threads are spread as evenly as possible over the given number of nodes without regard to possible overcommit of CPUs. When using pthreads, this may result in some processes having fewer threads than others if the threads do not divide evenly among the processes and nodes.

If ‘cpus_per_node’ is non-zero and -nodes is zero (or not set), then UPC threads are laid out to use the fewest number of nodes possible without exceeding ‘cpus_per_node’. Then UPC threads are spread as evenly as possible over the given number of nodes. When using pthreads, this may result in some processes having fewer threads than others if the threads do not divide evenly among the processes and nodes.

Options are read from the site-wide and user-specific configuration files, the environment and the command-line . The precedence of options is equivalent to parsing the options in the following order:

default_options <conduit>_options UPCRUN_FLAGS command-line

For options which set a value (such as -n and -shared-heap), the last value seen is the one used. Thus values on the command-line always take precedence over any others.

The ‘default_options’ and ‘<conduit>_options’ are taken from your $HOME/.upcrunrc file if present there, or from the site-wide upcrun.conf otherwise. If a given setting is present in both files only the settings in $HOME/.upcrunrc are used; they are not additive.

Arguments in ‘default_options’, ‘<conduit>_options’ and UPCRUN are split on whitespace, but single- or double-quotes will suppress splitting. The backslash character ‘#146; does not have any special meaning.

We are very interested in fixing any bugs in upcrun. For bug reporting instructions, please go to http://upc.lbl.gov.

upcc(1)

The Berkeley UPC User’s Guide (available at http://upc.lbl.gov)