DPLACE(1) DPLACE(1)
dplace - a NUMA memory placement tool
dplace [-place placement_file]
[-data_pagesize n-bytes]
[-data_lpage_wait [off|on]]
[-stack_pagesize n-bytes]
[-stack_lpage_wait [off|on]]
[-text_pagesize n-bytes]
[-text_lpage_wait [off|on]]
[-migration [off|on|threshold]]
[-migration_level threshold]
[-propagate]
[-mustrun]
[-v[erbose]]
program [program-arguments]
The given program is executed after placement policies are set up
according to command line arguments and the specifications described in
placement_file.
-place placement_file
Placement information is read from placement_file. If this argument
is omitted, no input file is read. See dplace(5) for correct
placement file format.
-data_pagesize n-bytes
Data and heap page sizes will be of size n-bytes. Valid page sizes
are 16k multiplied by a non negative integer powers of 4 up to a
maximum size of 16m. Valid page sizes are 16k, 64k, 256k, 1m, 4m,
and 16m.
-data_lpage_wait [off|on]
Normal behavior in the event of large page shortages is to continue
running utilizing smaller pages instead. If this option is specified
as on then the process will wait until large pages become available
for use by the data segment.
-stack_pagesize n-bytes
Stack page sizes will be of size n-bytes. Valid page sizes are 16k
multiplied by a non negative integer powers of 4 up to a maximum
size of 16m. Valid page sizes are 16k, 64k, 256k, 1m, 4m, and 16m.
-stack_lpage_wait [off|on]
Normal behavior in the event of large page shortages is to continue
running utilizing smaller pages instead. If this option is specified
as on then the process will wait until large pages become available
Page 1
DPLACE(1) DPLACE(1)
for use by the stack segment.
-text_pagesize n-bytes
Text page sizes will be of size n-bytes. Valid page sizes are 16k
multiplied by a non negative integer powers of 4 up to a maximum
size of 16m. Valid page sizes are 16k, 64k, 256k, 1m, 4m, and 16m.
-text_lpage_wait [off|on]
Normal behavior in the event of large page shortages is to continue
running utilizing smaller pages instead. If this option is specified
as on then the process will wait until large pages become available
for use by the stack segment.
-migration [off|on|threshold]
Page migration is turned on or off. If a threshold is specified then
page migration will be turned on and the migration threshold will be
set in the same manner as when -migration_level is specified (see
below).
-migration_level threshold
Page migration threshold is set to threshold. This value specifies
the maximum percentage difference between the number of remote
memory accesses and local memory accesses (relative to maximum
counter values ) for a given page, before a migration request event
occurs. A special argument of 0 will turn page migration off. This
option is provided for backward compatibility only, new scripts
should use the -migration option (see above) instead.
-propagate
Migration and page size information will be inherited by descendents
which are exec'ed.
-mustrun
When threads are attached to memories or cpus, the threads are
attached to CPUs on the node using process_cpulink with a request
mode of mandatory.
-verbose or -v
Detailed diagnostic information is written to standard error.
To place data according to the file placement_file for the executable
a.out that would normally be run by:
% a.out < in > out
one would simply
% dplace -place placement_file a.out < in > out .
An example placement file placement_file, when a.out is two threaded
might look like:
# placement_file
memories 2 in topology cube # set up 2 memories which are close
Page 2
DPLACE(1) DPLACE(1)
threads 2 # number of threads
run thread 0 on memory 1 # run the first thread on the 2nd memory
run thread 1 on memory 0 # run the 2nd thread on the first memory
This specification, would request 2 nearby memories from the operating
system. At creation, the threads are requested to run on an available cpu
which is local to the specified memory. As data and stack space is
touched or faulted in, physical memory is allocated from the memory which
is local to the thread which initiated the fault.
This can be written in a scalable way for a variable number of threads
using the environment variable NP as follows:
# scalable placement_file
memories $NP in topology cube # set up memories which are close
threads $NP # number of threads
# run the last thread on the first memory etc.
distribute threads $NP-1:0:-1 across memories
Since most MPI implementations use $MPI_NP+1 threads; where the first
thread is mainly inactive. One might use the placement file:
# scalable placement_file for MPI
memories ($MPI_NP + 1)/2 in topology cube # set up memories which are close
threads $MPI_NP + 1 # number of threads
# ignore the lazy thread
distribute threads 1:$MPI_NP across memories
When using MPI with dplace, syntax similar to the following should be
used:
mpirun -np <number_of_processes> dplace <dplace_args> a.out
Some applications run more efficiently using large pages. To run a
program a.out utilizing 64k pages for both stack and data, a placement
file is not necessary. One need only invoke the command:
dplace -data_pagesize 64k -stack_pagesize 64k a.out
from the shell.
Physical placement can also be accomplished using dplace. The following
placement file:
# physical placement_file for 3 specific memories and 6 threads
Page 3
DPLACE(1) DPLACE(1)
memories 3 in topology physical near \
/hw/module/2/slot/n4/node \
/hw/module/3/slot/n2/node \
/hw/module/4/slot/n3/node
threads 6
#the first two threads (0 & 1 ) will run on /hw/module/2/slot/n4/node
#the second two threads (2 & 3 ) will run on /hw/module/3/slot/n2/node
#the last two threads (4 & 5 ) will run on /hw/module/4/slot/n3/node
distribute threads across memories
specifies three physical nodes using the proper /hw path. To find out the
names of the memory nodes on the machine you are using, type "find /hw
-name node -print" at the shell command prompt.
The mustrun option will bind a thread to a particular CPU on the node.
In cases where threads are distributed across memories, the CPU selection
will attempt to schedule CPUs as to maximize CPU to memory bandwidth.
For example, on an Origin 3000, CPUs sharing a memory will be scheduled
on two separate processor busses available on the node, which serves to
increase available memory bandwidth for each processor. The following
placement file will demonstrate this behaviour:
# placement file for 4 memories and 8 threads
threads 8
memories 4
distribute threads across memories
By running this placement file using the option -mustrun the CPU
selection on each memory of an Origin 3000 will attempt to select CPU
numbers 0 and 2, or 1 and 3, or 0 and 3, or 1 and 2. The selection will
avoid pairing 0 and 1, or 2 and 3, as these CPUs share a processor
interface bus. In a situation where a processor selection cannot be done
optimally as described, then the next available CPU regardless of bus
attachment will be selected.
If command line arguments are omitted, dplace chooses the following set
of defaults:
place /dev/null
data_pagesize 16k
stack_pagesize 16k
text_pagesize 16k
migration off
propagate off
Page 4
DPLACE(1) DPLACE(1)
mustrun off
verbose off
Programs must be dynamic executables; non shared executables behavior are
are unaffected by dplace. Placement files will only affect direct
descendents of dplace. Parallel applications must be based on the
sproc(2) or fork(2) mechanism. Page sizes for regions which are not
stack, text, or data can not be specified with dplace (eg: SYSV shared
memory). Regions shared by multiple processes (eg: DSO text) are faulted
in with the pagesize settings of the faulting process. Dplace sets the
environment variable _DSM_OFF which will disable libmp's own DSM
directives and environment variables.
Dplace recognizes and uses the environment variables PAGESIZE_DATA,
PAGESIZE_STACK and PAGESIZE_TEXT. When using these variables it is
important to note that the units are in kilobytes. The command line
option will override environment variable setting.
If errors are encountered in the placement_file, the default procedure
for dplace is to print a diagnostic message to standard error specifying
where the error occurred in the placement_file and abort execution. If
errors are encountered in the libdplace.so library during the run-time
execution of program, then a diagnostic message is sent to standard
error, a default signal of SIGKILL is sent to all members of the process
group, and execution is aborted.
The mode signal expr statement allows a selection of a specific signal
number to be generated upon error. If the mode signal expr is specified,
the action taken when libdplace.so detects a run-time error is to send
the signal number derived from expr to the program invoked by dplace.
Under this condition the control is returned to the caller, which is the
program. The signal number can range from 1 to 32.
An example of how to set the signal number:
mode signal 16
Upon detecting an error in libdplace.so during run-time, signal 16
(defined as SIGUSR1) is sent to the calling process (in this case the
program and control is returned to the caller.
dplace(3), dplace(5), dprof(1), numa(5), mmci(5), dlook(1), numa_view(1),
mld(3).
PPPPaaaaggggeeee 5555 [ Back ]
|
