*nix Documentation Project
·  Home
 +   man pages
·  Linux HOWTOs
·  FreeBSD Tips
·  *niX Forums

  man pages->HP-UX 11i man pages -> gang_sched (7)              


 gang_sched(7)                                                 gang_sched(7)

 NAME    [Toc]    [Back]
      gang_sched - Gang Scheduler

 DESCRIPTION    [Toc]    [Back]
      The gang scheduler permits a set of MPI (Message Passing Interface)
      processes, or multiple threads from a single process, to be scheduled
      concurrently as a group.

      Gang scheduling is enabled and disabled by setting the MP_GANG
      environment variable to ON or OFF.

      The gang scheduling feature can significantly improve parallel
      application performance in loaded timeshare environments that are
      oversubscribed.  Oversubscription occurs when the total number of
      runnable parallel threads, runnable MPI processes, and other runnable
      processes exceeds the number of processors in the system.

      Gang scheduling also permits low-latency interactions among threads in
      shared-memory parallel applications.

      Only applications using the HP-UX V11.0 MPI or pthread libraries can
      be gang scheduled. Because HP compiler parallelism is primarily built
      on the pthread library, programs compiled with HP compilers can
      benefit from gang scheduling.

 INTERFACE    [Toc]    [Back]
      The HP-UX gang scheduler is enabled and disabled using an environment
      variable. The variable is defined as:

           MP_GANG [ON | OFF]

      Setting MP_GANG to ON enables gang scheduling and setting it to OFF
      disables it.  If MP_GANG is not set, or if it is set to an undefined
      value, no action is taken.

      Gang scheduling is a process attribute that is inherited by child
      processes created by fork (see fork(2)).  The state of gang scheduling
      for a process can change only following a call to exec (see exec(2)).

 BEHAVIOR    [Toc]    [Back]
      After the MP_GANG environment variable is set to ON, any MPI or
      pthread application to execute and find this variable will enable gang
      scheduling for that process.

      Only the pthread and MPI libraries query the MP_GANG variable--the
      operating system does not.

      Gang scheduling is an inherited process attribute. When a process with
      gang scheduling enabled creates a child process, the following occurs:

 Hewlett-Packard Company            - 1 -   HP-UX 11i Version 2: August 2003

 gang_sched(7)                                                 gang_sched(7)

           +  The child process inherits the gang scheduling attribute.

           +  A new gang is formed for the child process. The child does not
              become part of its parent's gang.

      The gang scheduler is engaged only when a gang consists of multiple
      threads.  For a pthread application, this is when a second thread is
      created. For an MPI application, it is when a second process is added.

      As a process creates threads, the new threads are added to the
      process's gang if gang scheduling is enabled for the process. However,
      once the size of a gang equals the number of processors in the system,
      the following occurs:

           +  New threads or processes are not added to the gang.

           +  The gang remains intact and continues to be gang scheduled.

           +  The spill-over threads are scheduled with the regular
              timeshare policies.

           +  If threads in the gang exit (thus making room available), the
              spill-over threads are not added into the gang.  However,
              newly created threads are added into the gang when room is

      MPI processes are allocated statically at the beginning of execution.
      When MP_GANG is set to ON, all processes in an MPI application are
      made part of the same gang.

      Thread and process priorities for gangs are managed identically to
      timeshare policy. The timeshare priority scheduler determines when to
      schedule a gang and adheres to the timeshare policies.

      Although it is likely that scheduling a gang will preempt one or more
      higher priority timeshare threads, over the long run the gang
      scheduler policy is generally fair. All threads in a gang will have
      been highest priority by the time a gang is scheduled. Because all
      threads in a gang must execute concurrently, some threads do not
      execute when they are highest priority (the threads must wait until
      all other threads have also been selected, allowing other processes to
      run first).

      Gangs are scheduled for a single time-slice. The time-slice is the
      same for all threads in the system, whether gang-scheduled or not.

      When a single gang executes on a system, the gang's threads are
      assigned to processors in the system and are not migrated to different

 Hewlett-Packard Company            - 2 -   HP-UX 11i Version 2: August 2003

 gang_sched(7)                                                 gang_sched(7)

      In an oversubscribed system with multiple gangs, all gangs are
      periodically moved in order to give an equalized percentage of CPU
      time to each of the different threads. This rebalancing occurs every
      few seconds.

    Environment Variables
      The following environment variables affect gang scheduling of

           +  MP_GANG enables (when set to ON) and disables (when set to
              OFF) gang scheduling of processes. For details see the
              INTERFACE section of this man page.

           +  MP_NUMBER_OF_THREADS specifies the number of processors
              available to execute programs compiled for parallel execution.
              If not set, the default is the number of processors in the

 PERFORMANCE    [Toc]    [Back]
      Gang scheduling ensures that all runnable threads and processes in a
      gang are scheduled simultaneously. This improves the synchronization
      latency in parallel applications. For instance, threads waiting at a
      barrier do not have to wait for currently unscheduled threads.

      However, applications with lengthy parallel regions and infrequent
      synchronization may perform best when not gang scheduled. For those
      applications, some threads can be scheduled even if all threads are
      not scheduled at once.

      A gang-scheduled application's performance can be affected by the
      number of gang-scheduled applications on a system, and by the number
      of threads in each.  The gang scheduler assigns parallel applications
      to CPUs using a "best fit" algorithm that attempts to minimize CPU
      overlap among applications.

      On systems with complex workloads including gangs of varying sizes, or
      odd combinations of sizes, the workload may not optimally match the
      number of CPUs available. In this situation an application may perform
      better when not gang scheduled, thus enabling some threads to be
      scheduled rather than waiting for all threads to be scheduled as a

    Scheduling Overhead    [Toc]    [Back]
      Gang scheduling incurs overhead when the scheduler collects a set of
      threads, assigns a set of processors to the threads, and rendezvous
      the set of threads and processors to achieve concurrent execution.

      On an idle system, the gang scheduling overhead can be seen in the
      execution time of a single parallel application.

 Hewlett-Packard Company            - 3 -   HP-UX 11i Version 2: August 2003

 gang_sched(7)                                                 gang_sched(7)

    Kernel Blocking of Threads    [Toc]    [Back]
      If a thread from a gang blocks in the kernel, the thread's processor
      is available to run other non-gang-scheduled threads. When the blocked
      thread resumes and its gang is currently running, the thread can join
      the other ganged threads without having to rendezvous again.

      In a multi-gang environment, thread blocking can result in lower
      throughput.  This occurs if an application's threads block often in
      the kernel for long periods of time.

    Preempting by Realtime Threads    [Toc]    [Back]
      Gang-scheduled threads can be preempted from execution by realtime
      threads. This affects only the gang-scheduled thread running on the
      processor being preempted by a realtime thread. The remaining threads
      of the gang continue to run through the end of their time-slice.

 RESTRICTIONS    [Toc]    [Back]
      For this implementation of gang scheduling, the following restrictions
      exist. Some of these may be removed in future releases.

           +  Gang scheduling of processes being debugged is not supported.
              When a debugger attaches to a process, gang scheduling for the
              process is disabled. This avoids gang scheduling processes
              with one or more threads stopped by a debugger.

           +  Gang scheduling is completely shut down when Process Resource
              Manager (PRM) is enabled.

           +  If a gang-scheduled process is selected to be swapped out, the
              process will not be gang-scheduled when it is swapped back in.

           +  Realtime processes are not gang-scheduled.

           +  Gang scheduling is only supported for processes with timeshare
              scheduling policies.

           +  When a gang-scheduled process contains the maximum number of
              threads (or the maximum number of processes, for MPI
              applications),  threads or processes created after this point
              are not scheduled as part of the gang. For details see the
              BEHAVIOR section of this man page.

           +  Multiprocess applications that do not use MPI are not
              supported by the gang scheduler.

           +  Gang scheduling is not supported for PTHREAD_SCOPE_PROCESS
              threads.  From release 11i Version 1.6 of HP-UX, the default
              scheduling contention scope for threads is
              are created by an application, the initial thread will be
              treated as a PTHREAD_SCOPE_PROCESS.

 Hewlett-Packard Company            - 4 -   HP-UX 11i Version 2: August 2003

 gang_sched(7)                                                 gang_sched(7)

 FILES    [Toc]    [Back]
      The following are libraries used in providing gang scheduling:

      /usr/lib/libpthread.1              The pthread library.

      /opt/mpi       The directory containing MPI libraries and MPI
                     software. HP MPI is an optional product.

 SEE ALSO    [Toc]    [Back]
      fork(2), exec(2).

 Hewlett-Packard Company            - 5 -   HP-UX 11i Version 2: August 2003
[ Back ]
 Similar pages
Name OS Title
lpsched HP-UX start the LP request scheduler,
schedctl IRIX scheduler control call
crontab HP-UX user job file scheduler
lpfence HP-UX start the LP request scheduler,
lpshut HP-UX start the LP request scheduler,
lpmove HP-UX start the LP request scheduler,
sched_setscheduler FreeBSD set/get scheduling policy and scheduler parameters
NSTRSCHED HP-UX number of STREAMS scheduler daemons to run
nstrsched HP-UX number of STREAMS scheduler daemons to run
sched_getscheduler FreeBSD set/get scheduling policy and scheduler parameters
Copyright © 2004-2005 DeniX Solutions SRL
newsletter delivery service