\label{sec:ga}

%\sloppy
%For more detailed information look in 
%{\tt \$NWCHEM\_TOP/src/global/doc/global.doc}     %$ for emacs
%
%\fussy

Globally addressable arrays have been developed to simplify writing
portable scientific software for both shared and distributed memory
computers.  Programming convenience, code extensibility and
maintainability are gained by adopting the shared memory programming
model.  The Global Array (GA) toolkit provides an efficient and portable 
"shared memory" programming interface for distributed memory computers.
Each process in a MIMD parallel program can asynchronously access
logical blocks of physically distributed matrices without need for
explicit cooperation by other processes. 
The trade-off with this approach is that
access to shared data will be slower than access
to local data, and the programmer must be aware of this in designing modules.

From the user perspective, a global array can be used as if it was stored
in the shared memory. Details of the data distribution, addressing and
communication are encapsulated in the global array objects. However,
the information on the actual data distribution can be obtained and
taken advantage of whenever data locality is important.

The Global Arrays tool has been designed to complement the message-passing 
programming model.  The developer can use both shared memory and message
passing paradigms in the same program, to take advantage of existing
message-passing software libraries such as TCGMSG.  This tool is also 
compatible with the Message Passing Interface (MPI).  The Global Arrays toolkit
has been in the public domain since 1994 and is actively supported.  Additional
documentation and information on performance and applications is available
on the web site http://www.emsl.pnl.gov:2080/docs/global/.

Currently support is limited to 2-D double precision or integer arrays
with block distribution, at most one block per array per processor.

\subsubsection{Interaction Between GA and MA}

Available global (GA)
and local (MA) memory can interact within NWChem in only two ways,

\begin{enumerate}
\item GA is allocated within MA, and GA is limited only by the available
    space in MA.

\item GA is not allocated within MA, and GA is limited at initialization
    (within NWChem input this is controlled by the MEMORY directive)
\end{enumerate}

If GA is allocated within MA, then 
the available GA space is limited to the currently available MA space.  This
also means that the total allocatable memory for GA {\em and} MA must be
no more than the available MA space.
If GA is not allocated within MA, then local and global arrays occupy essentially
independent space.  The allocatable memory for GA is limited only by the available
space for GA, and similarly, the allocatable memory for MA is limited only
by the available local memory.

When allocating space for GA,
some care must be exercised in the treatment of the information returned by
the routine \verb+ga_memory_avail()+, whether or not
the allocation is done in MA.  The routine \verb+ga_memory_avail()+
returns the amount of memory (in bytes)
available for use by GA in the calling process.
This returned value must be converted to double precision words when
using double precision.
If a uniformly distributed GA is desired, it is also necessary to find
the minimum of this value across all nodes.  This value will in general be
a rather large number.
When running on a platform with many nodes and having a large memory, 
the agreggate GA memory, even in double precision words, could be a large enough
value to overflow a
32-bit integer.  Therefore, for calculations that require knowing the size of
total memory, it is advisable to first store the size of memory on each node 
in a double precision
number and then sum these values across all the nodes.

The following pseudo-code illustrates this process for an application.

\begin{verbatim}
#include "global.fh"
#include "mafdecls.fh"

    integer avail_ma, avail_ga

    avail_ma = ma_inquire_avail(mt_dbl)
    avail_ga = ga_memory_avail()/ma_sizeof(mt_dbl,1,mt_byte)

    if (ga_uses_ma()) then
c
c  available GA space is limited to currently available MA space,
c  and GA and MA share the same space
c
      allocatable_ga + allocable_ma <= avail_ma = avail_ga

    else
c
c  GA and MA are independent
c
         allocatable_ga <= avail_ga
         allocatable_ma <= avail_ma

    endif
c
c find the minimum value of available GA space over all nodes
c
    call ga_igop(msgtype,avail_ga,1,'min')
c
c determine the total available GA space
c
    double precision davail_ga
    davail_ga = ga_memory_avail()/ma_sizeof(mt_dbl,1,mt_byte)
    call ga_dgop(msgtype,davail_ga,1,'+')

\end{verbatim}


\subsubsection{List of GA Routines}

The following routines are invoked for operations that are globally collective.
That is, they must be
simultaneously invoked by all processes as if in SIMD mode.

\begin{itemize}
\item {\tt ga\_initialize\_()} --- initialize global array internal
  structures
\item {\tt ga\_initialize\_ltd(mem\_limit)} --- initialize global arrays and set
  memory usage limits (note: if \verb+mem_limit+ is less than zero specifies
  unlimited memory usage.)
\begin{itemize}
\item      integer {\tt mem\_limit}       --- [input] GA total memory ( specifying less than 0
means "unlimited memory")
\end{itemize}

\item {\tt ga\_create(type,dim1,dim2,array\_name,chunk1,chunk2,g\_a)} --- create an array
\begin{itemize}
\item     integer {\tt type}         --- [input] MA type
\item     integer {\tt dim1, dim2}   --- [input] array dimensions (dim1,dim2) as in FORTRAN
\item     character {\tt array\_name}--- [input] unique character string identifying the array
\item     integer {\tt chunk1, chunk2} --- [input] minimum size that dimensions should
                                         be chunked up into;
                                         setting chunk1=dim1 gives distribution by rows
                                         setting chunk2=dim2 gives distribution by columns 
                                         Actual chunk sizes are modified so that they are
                                         at least the min size and each process has either
                                         zero or one chunk. 
                                         (Specifying both as less than or equal to 1 
                                         yields an even distribution)
\item     integer {\tt g\_a}             [output] integer handle for future references
\end{itemize}

\item {\tt ga\_create\_irreg(type, dim1, dim2, array\_name, map1, nblock1,
map2, nblock2, g\_a)} --- create an array with irregular
  distribution
\begin{itemize}
\item     integer {\tt type}        --- [input] MA type
\item     integer {\tt dim1, dim2}  --- [input] array dimensions (dim1,dim2) as in FORTRAN
\item     character {\tt array\_name}--- [input] unique character string identifying the array
\item     integer {\tt map1}        --- [input] number ilo in each block
\item     integer {\tt nblock1}     --- [input] number of blocks dim1 is divided into
\item     integer {\tt map2}        --- [input] number jlo in each block
\item     integer {\tt nblock2}     --- [input] number of blocks dim2 is divided into
\item     integer {\tt g\_a}        --- [output] integer handle for future references
\end{itemize}

\item {\tt ga\_duplicate(g\_a, g\_b, array\_name)} --- create an array with same properties as reference
  array
\begin{itemize}
\item     character {\tt array\_name} --- [input] unique character string identifying the array
\item     integer {\tt g\_a}         --- [output] integer handle for reference array
\item     integer {\tt g\_b}         --- [output] integer handle for new array
\end{itemize}

\item {\tt ga\_destroy\_(g\_a)} --- destroy an array
\begin{itemize}
\item     integer {\tt g\_a}         --- [input] integer handle of array to be destroyed
\end{itemize}

\item {\tt ga\_terminate\_()} --- destroys all existing global arrays 
and de-allocates shared memory
\item {\tt ga\_sync\_()} --- synchronizes all processes (a barrier)
\item {\tt ga\_zero\_(g\_a)} --- zero an array
\begin{itemize}
\item     integer {\tt g\_a}         --- [input] integer handle of array to be zeroed
\end{itemize}

\item {\tt ga\_ddot\_(g\_a, g\_b)} --- dot product of two arrays (double precision only)
\begin{itemize}
\item     integer {\tt g\_a}         --- [input] integer handle of first array in dot product
\item     integer {\tt g\_b}         --- [input] integer handle of second array in dot product
\end{itemize}

\item {\tt ga\_dscal} --- scale the elements in an array by a constant
  (double precision data only)
\item {\tt ga\_dadd} --- scale and add two arrays to put result in a
  third (may overwrite one of the other two, doubles only)
\item {\tt ga\_copy(g\_a, g\_b)} --- copy one array into another
\begin{itemize}
\item     integer {\tt g\_a}          --- [input] integer handle of array to be copied
\item     integer {\tt g\_b}         --- [input] integer handle of array g\_a is copied into
\end{itemize}

\item {\tt ga\_dgemm(transa, transb, m, n, k, alpha, g\_a, g\_b, beta, g\_c} --- 
BLAS-like matrix multiply
\begin{itemize}
\item      character*1        {\tt transa, transb}
\item      integer            {\tt m, n, k}
\item      double precision   {\tt alpha, beta}
\item      integer            {\tt g\_a, g\_b, g\_c}
\end{itemize}

\item {\tt ga\_ddot\_patch(g\_a, t\_a, ailo, aihi, ajlo, ajhi,
         g\_b, t\_b, bilo, bihi, bjlo, bjhi)} --- dot product of two arrays (double precision
  only; patch version) (Note: patches of different shapes and distrubutions
are allowed, but not recommended, and both patches must have the same number
of elements)
\begin{itemize}
\item      integer {\tt g\_a}            --- [input] integer identifier of first array containing
                                               patch for dot product
\item     integer {\tt t\_a}             --- [input] transpose of first array
\item     integer {\tt ailo, aihi}       --- [input] high and low indices for i dimension of
                                               patch of array  for dot product
\item     integer {\tt ajlo, ajhi}       --- [input] high and low indices for j dimension of
                                               patch of array for dot product
\item      integer {\tt g\_b}            --- [output]integer identifier of second array contianing
                                               patch for dot product
\item     integer {\tt t\_b}             --- [input] transpose of second array
\item     integer {\tt bilo, bihi}       --- [input] high and low indices for i dimension of
                                               patch of array  for dot product
\item     integer {\tt bjlo, bjhi}       --- [input] high and low indices for j dimension of
                                               patch of array  for dot product
\end{itemize}

\item {\tt ga\_dscal\_patch} --- scale the elements in an array by a
  constant (patch version)
\item {\tt ga\_dadd\_patch} --- scale and add two arrays to put result
  in a third (patch version)
\item {\tt ga\_ifill\_patch} --- fill a patch of array with value
  (integer version)
\item {\tt ga\_dfill\_patch} --- fill a patch of array with value
  (double version)
\item {\tt ga\_matmul\_patch(transa, transb, alpha, beta, g\_a, ailo, aihi,
ajlo, ajhi, g\_b, bilo, bihi, bjlo, bjhi, g\_c, cilo, cihi,
cjlo, cjhi)} --- matrix multiply (patch version)
\begin{itemize}
\item      character {\tt transa}        --- [input] transpose of first array for matrix multiply
\item      character {\tt transb}        --- [input] transpose of second array for matrix multiply
\item      double precision {\tt alpha}  ---  ??
\item      double precision {\tt beta}   ---  ??
\item      integer {\tt g\_a}            --- [input] integer identifier of first array for matrix multiply
\item     integer {\tt ailo, aihi}       --- [input] high and low indices for i dimension of
                                               patch of first array for matrix multiply
\item     integer {\tt ajlo, ajhi}       --- [input] high and low indices for j dimension of
                                               patch of first array for matrix multiply
\item      integer {\tt g\_b}            --- [input] integer identifier of second array for matrix multiply
\item     integer {\tt bilo, bihi}       --- [input] high and low indices for i dimension of
                                               patch of second array for matrix multiply
\item     integer {\tt bjlo, bjhi}       --- [input] high and low indices for j dimension of
                                               patch of second array for matrix multiply
\item      integer {\tt g\_c}            --- [input] integer identifier of resultant array for matrix multiply
\item     integer {\tt cilo, cihi}       --- [input] high and low indices for i dimension of
                                               patch of resultant array for matrix multiply
\item     integer {\tt cjlo, cjhi}       --- [input] high and low indices for j dimension of
                                               patch of resultant array for matrix multiply
\end{itemize}


\item {\tt ga\_diag(g\_a, g\_s, g\_v, eval)} --- real symmetric generalized eigensolver
  (sequential version \verb+ga_diag_seq+ also exists)
\begin{itemize}
\item      integer {\tt g\_a}           ---  matrix to diagonalize
\item      integer {\tt g\_s}           ---  metric
\item      integer {\tt g\_v}           ---  global matrix to return evecs
\item      double precision {\tt eval(*)} --- local array to return evals
\end{itemize}

\item {\tt ga\_diag\_reuse(reuse,g\_a,g\_s,g\_v,eval)} --- a 
version of {\tt ga\_diag} for repeated use
\begin{itemize}
\item      integer {\tt reuse} --- allows reuse of factorized g\_s: flag is
                                     0  first time, greater than 0 for
                                        subsequent calls, less than 0 
                                     deletes factorized g\_s 
\item      integer {\tt g\_a}           ---  matrix to diagonalize
\item      integer {\tt g\_s}           ---  metric
\item      integer {\tt g\_v}           ---  global matrix to return evecs
\item      double precision {\tt eval(*)} --- local array to return evals
\end{itemize}

\item {\tt ga\_diag\_std(g\_a, g\_v, eval)} --- standard real symmetric eigensolver
  (sequential version also exists)
\begin{itemize}
\item      integer {\tt g\_a}            --- [input] matrix to diagonalize
\item      integer {\tt g\_v}            --- [output] global matrix to return evecs
\item      double precision {\tt eval(*)} --- [output] local array to return evals
\end{itemize}

\item {\tt ga\_symmetrize(g\_a)} --- symmetrizes matrix A into 0.5(A+A') (NOTE: diag(A)
remains unchanged.)
\begin{itemize}
\item      integer {\tt g\_a}           --- [input] matrix to symmetrize
\end{itemize}

\item {\tt ga\_transpose(g\_a)} --- transpose a matrix
\begin{itemize}
\item      integer {\tt g\_a}            --- [input] matrix to transpose 
\end{itemize}

\item {\tt ga\_lu\_solve(trans, g\_a, g\_b)} --- solves system of linear equations based
  on LU factorization (sequential version \verb+ga_lu_solve_seq+ also exists)
\begin{itemize}
\item      character*1 {\tt trans}       --- [input] transpose or not
\item      integer {\tt g\_a}            --- [input] matrix to diagonalize (coefficient matrix A)
\item      integer {\tt g\_b}            --- [output] rhs matrix B, overwritten on exit
                                        by the solution vector, X of AX = B
\end{itemize}

\item {\tt ga\_print\_patch(g\_a, ilo, ihi, jlo, jhi, pretty)} --- print a patch of an array to the
  screen
\begin{itemize}
\item      integer {\tt g\_a}           --- [input] integer identifier of array to be printed
\item      integer {\tt ilo, ihi}       --- [input] high and low indices for i dimension of patch
                                               of array to be printed
\item      integer {\tt jlo, jhi}       --- [input] high and low indices for j dimension of patch
                                               of array to be printed
\item      integer {\tt pretty}         --- [input] flag for format of output to screen;
\begin{itemize}
\item pretty = 0, spew output out with no formatting
\item pretty = 1, format output so that it is readable
\end{itemize}
\end{itemize}

\item {\tt ga\_print(g\_a)} --- print an entire array to the screen
\begin{itemize}
\item      integer {\tt g\_a}           --- [input] integer identifier of array to be printed
\end{itemize}

\item {\tt ga\_copy\_patch(trans, g\_a, ailo, aihi, ajlo, ajhi, g\_b, bilo,
bihi, bjlo, bjhi)} --- copy data from a patch of one global
  array into another array, (Note: patch can change shape, but total numer of elements
  must be the same between the two arrays)
\begin{itemize}
\item      character*1 {\tt trans}      --- [input] transpose or not
\item      integer {\tt g\_a}           --- [input] integer identifier of array to be copied
\item     integer {\tt ailo, aihi}      --- [input] high and low indices for i dimension of
                                               patch of array 
                                               to be copied
\item     integer {\tt ajlo, ajhi}      --- [input] high and low indices for j dimension of
                                               patch of array 
                                               to be copied
\item      integer {\tt g\_b}           --- [output]integer identifier of array data is to be
                                       copied into
\item     integer {\tt bilo, bihi}      --- [input] high and low indices for i dimension of patch
                                               of array being copied into
\item     integer {\tt bjlo, bjhi}      --- [input] high and low indices for j dimension of patch
                                               of array being copied into
\end{itemize}

\item {\tt ga\_compare\_distr\_(g\_a, g\_b)} --- compare distributions of two global
  arrays
\begin{itemize}
\item      integer {\tt g\_a}           --- [input] integer identifier of first array
\item      integer {\tt g\_b}           --- [output]integer identifier of second array
\end{itemize}

\end{itemize}

Operations that may be invoked by any process in true MIMD style:
\begin{itemize}
\item {\tt ga\_get\_(g\_a, ilo, ihi, jlo, jhi, buf, Id)} --- read from a patch of an array
\begin{itemize}
\item     integer {\tt g\_a}         --- [input] integer handle of array
\item     integer {\tt ilo, ihi}    --- [input] high and low indices for i dimension of region
\item     integer {\tt jlo, jhi}    --- [input] high and low indices for j dimension of region
\item     integer {\tt buf}         --- [output] ???
\item     integer {\tt Id}          --- [output] ???
\end{itemize}

\item {\tt ga\_put\_(g\_a, ilo, ihi, jlo, jhi, buf, Id)} --- write from a patch of an array
\begin{itemize}
\item     integer {\tt g\_a}         --- [input] integer handle of array
\item     integer {\tt ilo, ihi}    --- [input] high and low indices for i dimension of region
\item     integer {\tt jlo, jhi}    --- [input] high and low indices for j dimension of region
\item     integer {\tt buf}         --- [output] ???
\item     integer {\tt Id}          --- [output] ???
\end{itemize}

\item {\tt ga\_acc\_(g\_a, ilo, ihi, jlo, jhi, buf, Id, alpha)} --- accumulate into a patch of an array (double
  precision only)
\begin{itemize}
\item     integer {\tt g\_a}         --- [input] integer handle of array
\item     integer {\tt ilo, ihi}    --- [input] high and low indices for i dimension of region
\item     integer {\tt jlo, jhi}    --- [input] high and low indices for j dimension of region
\item     integer {\tt buf}         --- [output] ???
\item     integer {\tt Id}          --- [output] ???
\item     integer {\tt alpha}       ---  ????
\end{itemize}

\item {\tt ga\_scatter\_(g\_a, v, i, j, nv)} --- scatter elements of v into an array
\begin{itemize}
\item     integer {\tt g\_a}         --- [input] integer handle of array that elements of v are to be scattered into
\item     ????    {\tt v}           --- [input] array from which elements are to be scattered
\item     integer {\tt i, j}        --- [input] array element indices (i,j) as in FORTRAN
\item     integer {\tt nv}          ---  ????
\end{itemize}

\item {\tt ga\_gather\_g\_a, v, i, j, nv} --- gather elements from an array v into array g\_a
\begin{itemize}
\item     integer {\tt g\_a}         --- [input] integer handle of array that elements of v are to be gathered into
\item     ????    {\tt v}           --- [input] array from which elements are to be gathered 
\item     integer {\tt i, j}        --- [input] array element indices (i,j) as in FORTRAN
\item     integer {\tt nv}          ---  ????
\end{itemize}

\item {\tt ga\_read\_inc\_(g\_a, i, j, inc)} --- atomically read and increment the value
  of a single array element (integers only)
\begin{itemize}
\item     integer {\tt g\_a}         --- [input] integer handle of array 
\item     integer {\tt i, j}        --- [input] array element indices (i,j) as in FORTRAN
\item     integer {\tt inc}         --- [input] amount to increment array element value
\end{itemize}

\item {\tt ga\_locate(g\_a,i,j,owner)} --- determine which process `holds' an array
  element (i,j)
\begin{itemize}
\item     integer {\tt g\_a}         --- [input] integer handle of array
\item     integer {\tt i, j}        --- [input] array element indices (i,j) as in FORTRAN
\item     integer {\tt owner}       --- [output] index number of processor holding the element
\end{itemize}

\item {\tt ga\_locate\_region\_(g\_a, ilo, ihi, jlo, jhi, map, np)} --- 
determine which process `holds' an
  array section
\begin{itemize}
\item     integer {\tt g\_a}         --- [input] integer handle of array
\item     integer {\tt ilo, ihi}    --- [input] high and low indices for i dimension of region
\item     integer {\tt jlo, jhi}    --- [input] high and low indices for j dimension of region
\item     ??????  {\tt map}                 --- [output] ???
\item     integer {\tt np}          --- [output] index number of processor holding the region 
\end{itemize}

\item {\tt ga\_error(string, icode)} --- print error message and terminate the
  program
\begin{itemize}
\item      character {\tt string}     --- [input] ????
\item      integer {\tt icode}        --- [input] integer flag for error code
\end{itemize}

\item {\tt ga\_summarize(verbose)} --- print information about all
  allocated arrays (note: assumes no more than 100 arrays are allocated and
are numbered -1000, -999, etc.)
\begin{itemize}
\item      integer {\tt verbose}      --- [input] if non-zero, print distribution information
\end{itemize}

\end{itemize}

Operations that may be invoked by any process in true MIMD style and
are intended to support writing of new functions:
\begin{itemize}
\item {\tt ga\_distribution\_(g\_a, me, ilo, ihi, jlo, jhi)} --- find coordinates of the array patch
  that is `held' by a processor
\begin{itemize}
\item     integer {\tt g\_a}         --- [input] integer handle of array
\item     integer {\tt me}          --- [input] index number of processor holding the patch
\item     integer {\tt ilo, ihi}    --- [output] high and low indices for i dimension of region
\item     integer {\tt jlo, jhi}    --- [output] high and low indices for j dimension of region
\end{itemize}

\item {\tt ga\_access(g\_a, ilo, ihi, jlo,jhi, index, Id)} --- provides access to a patch of a global array
\begin{itemize}
\item     integer {\tt g\_a}         --- [input] integer handle of array to be accessed
\item     integer {\tt ilo, ihi}    --- [output] high and low indices for i dimension of region
\item     integer {\tt jlo, jhi}    --- [output] high and low indices for j dimension of region
\item     integer {\tt index}       ---  ????
\item     integer {\tt Id}          ---  ????
\end{itemize}

\item {\tt ga\_release(g\_a, ilo, ihi, jlo, jhi)} --- relinquish access to internal data
\begin{itemize}
\item     integer {\tt g\_a}         --- [input] integer handle of array to be released
\item     integer {\tt ilo, ihi}    --- [output] high and low indices for i dimension of region
\item     integer {\tt jlo, jhi}    --- [output] high and low indices for j dimension of region
\end{itemize}

\item {\tt ga\_release\_update\_(g\_a, ilo, ihi, jlo, jhi)} --- relinquish access after data were
  updated
\begin{itemize}
\item     integer {\tt g\_a}         --- [input] integer handle of array to be updated and released
\item     integer {\tt ilo, ihi}    --- [output] high and low indices for i dimension of region
\item     integer {\tt jlo, jhi}    --- [output] high and low indices for j dimension of region
\end{itemize}

\item {\tt ga\_check\_handle(g\_a, fstring)} --- verify that a GA handle is valid
\begin{itemize}
\item     integer {\tt g\_a}         --- [input] integer handle of array
\item     character* {\tt fstring}  --- [input] name of routine originating the check
\end{itemize}

\end{itemize}

Operations to support portability between implementations:
\begin{itemize}
\item {\tt ga\_nodeid\_()} --- find requesting compute process message id
\item {\tt ga\_nnodes\_()} --- find number of compute processes
\item {\tt ga\_dgop(type, x, n, op)} --- equivalent to TCGMSG dgop, for use in data-server
mode where only compute processes participate
\begin{itemize}
\item     integer {\tt type}        --- [input] integer handle of array
\item     integer {\tt n}           --- [input]
\item     double precision {\tt x}  --- [input] 
\item     character {\tt op}        --- [input] 
\end{itemize}

\item {\tt ga\_igop(type, x, n, op)} --- equivalent to TCGMSG {\tt igop}, for use in data-server mode
where only compute processes participate; performs the operation specified by the input variable
{\tt op} (supported operations include addition, multiplication, maximum, minimum, 
and maximum or minimum of the absolute
value), and returns the value in {\tt x}.
\begin{itemize}
\item     integer {\tt type}        --- [input] integer handle of array
\item     integer {\tt n}           --- [input]
\item     double precision {\tt x}  --- [input/output]
\item     character {\tt op}        --- [input] 
\end{itemize}

\item {\tt ga\_brdcst(type, buf, len, originator)} --- equivalent to TCGMSG brdcst, for use in data server mode
with predefined communicators
\begin{itemize}
\item     integer {\tt type}        --- [input] integer handle of array
\item     ????    {\tt buf}         --- [input]
\item     integer {\tt len}         --- [input]
\item     integer {\tt originator}  --- [input] number of originating processor
\end{itemize}

\end{itemize}

Other utility operations:
\begin{itemize}
\item {\tt ga\_inquire\_(g\_a, atype, adim1, adim2)} --- find the type and 
                    dimensions of the array
\begin{itemize}
\item     integer {\tt g\_a}        --- [input] integer identifier of array
\item     integer {\tt atype}       --- [output] MA type
\item     integer {\tt adim1, adim2} --- [output] array dimensions (adim1,adim2) as in FORTRAN
\end{itemize}

\item {\tt ga\_inquire\_name\_(g\_a, array\_name)} --- find the name of the array
\begin{itemize}
\item     integer {\tt g\_a}         --- [input] integer identifier of array
\item     character* {\tt array\_name}  --- [output] string containing name of the array
\end{itemize}

\item {\tt ga\_inquire\_memory\_()} --- find the amount of memory in
  active arrays
\item {\tt ga\_memory\_avail\_()} --- find the amount of memory (in bytes) left for
  GA
\item {\tt ga\_summarize(verbose)} --- prints summary info about allocated
  arrays
\begin{itemize}
\item     integer {\tt verbose}     --- [input] if non-zero, print distribution information
\end{itemize}

\item {\tt ga\_uses\_ma\_()} --- finds if memory in arrays comes from MA
  (memory allocator)
\item {\tt ga\_memory\_limited\_()} --- finds if limits were set for
  memory usage in arrays
\end{itemize}

Note that consistency is only guaranteed for
\begin{enumerate}
\item Multiple read operations (as the data does not change)
\item Multiple accumulate operations (as addition is commutative)
\item Multiple disjoint put operations (as there is only one writer
  for each element)
\end{enumerate}
The application has to worry about everything else (usually by
appropriate insertion of {\tt ga\_sync} calls).

\subsubsection{New(?) Stuff}

Subroutines that appear in the files of directory .../src/global/src, but
are not in the (ga.tex) document;

\begin{itemize}
\item {\tt ga\_get\_local(g\_a, ilo, ihi, jlo, jhi, buf, offset, Id, proc)} --- local
read of a 2-dimensional patch of data into a global array
\item {\tt ga\_get\_remote(g\_a, ilo, ihi, jlo, jhi, buf, offset, Id, proc)} --- read an
array patch from a remote processor
\item {\tt ga\_put\_local(g\_a, ilo, ihi, jlo, jhi, buf, offset, Id, proc)} --- local
write of a 2-dimensional patch of data into a global array
\item {\tt ga\_put\_remote(g\_a, ilo, ihi, jlo, jhi, buf, offset, Id, proc)} --- write an
array patch from a remote processor
\item {\tt ga\_acc\_local(g\_a, ilo, ihi, jlo, jhi, buf, offset, Id, proc, alpha)} --- local
accumulate of a 2-dimensional patch of data into a global array
\item {\tt ga\_acc\_remote(g\_a, ilo, ihi, jlo, jhi, buf, offset, Id, proc, alpha)} --- accumulate an
array patch from a remote processor
\item {\tt ga\_scatter\_local(g\_a, v, i, j, nv, proc)} --- local
scatter of v into a global array
\item {\tt ga\_scatter\_remote(g\_a, v, i, j, nv, proc)} --- scatter of v into an
array patch from a remote processor
\item {\tt ga\_gather\_local(g\_a, v, i, j, nv, proc)} --- local
gather of v into a global array
\item {\tt ga\_gather\_remote(g\_a, v, i, j, nv, proc)} --- gather of v into an
array patch from a remote processor
\item {\tt ga\_dgop\_clust(type, x, n, op, group)} --- equivalent to TCGMSG dgop, for use in data-server
mode where only compute processes participate
\item {\tt ga\_igop\_clust(type, x, n, op, group)} --- equivalent to TCGMSG igop, for use in data-server
mode where only compute processes participate
\item {\tt ga\_brdcst\_clust(type, buf, len, originator, group)} --- internal GA routine
that is used in data server mode with predefined communicators
\item {\tt ga\_debug\_suspend()} --- ??? option to suspend debugging for a particular process
\item {\tt ga\_copy\_patch\_dp(t\_a, g\_a, ailo, aihi, ajlo, ajhi,
                   g\_b, bilo, bihi, bjlo, bjhi)} --- copy a patch by column order (Fortran convention)
\item {\tt ga\_print\_stats\_()} --- print GA statistics for each process
\item {\tt ga\_zeroUL(uplo, g\_A)} --- set to zero the L/U tirangle part of an NxN
double precision global array A
\item {\tt ga\_symUL(uplo, g\_A)} --- make a symmetric square matrix from a
double precision global array A in L/U triangle format
\item {\tt ga\_llt\_s(uplo, g\_A, g\_B, hsA)} --- solves a system of linear equations [A]X = [B],
\item {\tt ga\_cholesky(uplo, g\_a)} --- computes the Cholesky factorization of an NxN
double precision symmetric positive definite matrix to obtain the L/U factor
on the lower/upper triangular part of the matrix
\item {\tt ga\_llt\_f(uplo, g\_A, hsA)} --- computes the Cholesky factorization of an NxN 
double precision symmetric positive definite global array A
\item {\tt ga\_llt\_i(uplo, g\_A, hsA)} --- computes the inverse of a global array that is
the lower triangle L or the upper triagular Cholesky factor U of an NxN 
double precision symmetric positive definite global array (LL' or U'U)
\item {\tt ga\_llt\_solve(g\_A, g\_B)} --- solves a system of linear equations [A]X = [B]
using the cholesky factorization of an NxN
double precision symmetric positive definite global array A
\item {\tt ga\_spd\_invert(g\_A)} --- computes the inverse of a double precision array
using the cholesky factorization of an NxN
double precision symmetric positive definite global array A
\item {\tt ga\_solve(g\_A, g\_B)} --- solves a system of linear equations [A]X = [B], trying
first to use
the Cholesky factorization routine; if not successful, calls the LU
factorization routine \verb+ga_llt_solve+, and solves the system with forward/backward
substitution
\item {\tt ga\_ma\_base\_address(type, address)} --- auxiliary routine to provide MA
base addresses of the data  (calls C routines {\tt ga\_ma\_get\_ptr()})
\item {\tt ga\_ma\_sizeof(type)} --- auxiliary routine to provide MA
sizes of the arrays (calls C routines {\tt ga\_ma\_diff()})

\end{itemize}

\subsubsection{Use of TCGMSG global operation routines}

In some cases (notably workstation clusters) the global array tools
use a ``data-server'' process on each node in addition to the compute
processes.  Data-server processes don't follow the same flow of
execution of compute processes, so TCGMSG global operations
(\verb+brdcst+, \verb+igop+, and \verb+dgop+) will hang when invoked.
The global array toolkit provides ``wrapper'' functions
(\verb+ga_brdcst+, \verb+ga_igop+, and \verb+ga_dgop+) which properly
exclude data server processes from the global communication and must
be used instead of the corresponding TCGMSG functions.

\subsubsection{Interaction between GA and message-passing}

The limited buffering available on the IBM SP-1/2 means that GA and
message-passing operations cannot interleave as readily as they do on
other machines.  Basically, in transitioning from GA to message
passing or vice versa the application must call {\tt ga\_sync()}.
