### Disk Resident Arrays

Disk Resident Arrays (DRA) extend the Global Arrays (GA) programming model to disk. The library encapsulates the details of data layout, addressing and I/O transfer in disk arrays objects. Disk resident arrays resemble global arrays except that they reside on the disk instead of the main memory. The main features of this model are:

- Data can be transfered (copied) between disk and global memory.
- I/O operations have nonblocking interface to allow overlapping of I/O with computations.
- All I/O operations are collective.
- Either whole or sections of global arrays can be transferred between GA memory and the disk.
- Reshaping and transpose operation are allowed during the transfer.
- Disk resident arrays might be temporary or persistent (saved after program termination and available in next runs).
- Persistent disk arrays can be accessed by any program executing on arbitrary number of processors.
- Distribution and internal data layout on the disk is optimized for large data transfer.
- Hints provided by the user are utilized to guide optimization of the library performance for the specific I/O patterns.

DRA can take advantage of a shared file system of a collection of independent filesystems accessible from individual computing nodes.

## dra_init

    status = dra_init(max_arrays, max_array_size, total_disk_space, max_memory)
             integer max_arrays [input]
             double precision max_array_size [input]
             double precision total_disk_space [input]
             double precision max_memory [input]

Initializes Disk Resident Array I/O subsystem.

max_array_size , total_disk_space and max_memory are given in bytes.

max_memory specifies how much local memory per processor the application is willing to provide to the DRA I/O subsystem for buffering.

The value of -1 for any of input arguments means: "don't care ", "don't know" , or "use defaults ".

## dra_terminate

      status=dra_terminate()

Close all open disk resident arrays and shut down DRA I/O subsystem.

## dra_create

         status = dra_create(type, dim1, dim2, name, filename, mode,rdim1,rdim2,d_a)
                  integer type                      [input]   ! MA type identifier
                  integer dim1                      [input]
                  integer dim2                      [input]
                  character*(*) name                [input]
                  character*(*) filename            [input]
                  integer mode                      [input]
                  integer rdim1                     [input]
                  integer rdim2                     [input]
                  integer d_a                       [output]  ! DRA handle

Creates a new disk resident array with specified dimensions and data type (represented by MA type handle).

The string filename specifies name of an abstract meta-file that will store the data on the disk. The meta-file might be implemented as multiple files that will contain parts of the disk resident array. The component files will have names derived from the string filename according to some established scheme(s). Only one DRA object can be stored in DRA meta-file identified by filename .

DRA objects persist on the disk after calling dra_close. dra_delete should be used instead of dra_close to delete disk array and associated meta-file on the disk.

String name can be used for more informative (longer)names.

The data in disk resident array is implicitly initialized to 0 .

Access permissions (read, write, read& write) are set in mode . These are set using defined in the header files DRA.fh (Fortran) and DRA.h (C) preprocessor constants: DRA_R, DRA_W, DRA_RW.

The pair [rdim1, rdim2] specifies dimensions of a typical request. The value of -1 for either of them means "unspecified". The layout of the data on the disk(s) is determined based on the values of these arguments. Performance of the DRA operations depends on the dimensions (section shape) of the requests. If data layout is optimized for column-like sections, performance of DRA operations for row-like sections might be seriously degraded. This is analogous to the effect of wrong loop ordering iyielding frequent cache misses in the following example .

                  double precision a(1000, 1000)
                  do i = 1, 1000
                     do j = 1, 1000
                        a(i,j) = dfloat(i+j)
                     enddo
                  enddo

instead of

                  do j = 1, 1000
                     do i = 1, 1000
                        a(i,j) = dfloat(i+j)
                     enddo
                  enddo

## dra_open

        status = dra_open(filename, mode, d_a)
                  character*(*) filename            [input]
                  integer mode                      [input]
                  integer d_a                       [output]  ! DRA handle

Open and assign DRA handle to disk resident array stored in DRA meta-file filename. Disk resident arrays that are created with dra_create and saved by calling dra_close can be later opened and accessed by the same or different application.

Attributes of the disk resident array can be found by calling dra_inquire.

## dra_write

         status = dra_write(g_a, d_a, request)
                  integer g_a                       [input]  ! GA handle
                  integer d_a                       [input]  ! DRA handle
                  integer request                   [output] ! request id

Write asynchronously specified global array to specified disk resident array.

The dimensions and type of arrays represented by handles g_a and d_a must match. If dimensions don't match, dra_write_section should be used instead.

The operation is by definition asynchronous but it might be implemented as synchronous i.e., it would return only when the I/O is completed.

request can be used to dra_probe or dra_wait for completion of the associated operation.

## dra_write_section

         status = dra_write_section(transp, g_a, gilo, gihi, gjlo, gjhi,
                                            d_a, dilo, dihi, djlo, djhi, request)
                  logical transp                    [input] ! transpose operator
                  integer g_a                       [input] ! GA handle
                  integer d_a                       [input] ! DRA handle
                  integer gilo                      [input]
                  integer gihi                      [input]
                  integer gjlo                      [input]
                  integer gjhi                      [input]
                  integer dilo                      [input]
                  integer dihi                      [input]
                  integer djlo                      [input]
                  integer djhi                      [input]
                  integer request                   [output] ! request id

Write asynchronously specified global array section to specified disk resident array section:

                    OP(g_a[ gilo:gihi, gjlo:gjhi]) -->  d_a[ dilo:dihi, djlo:djhi]

where OP is the transpose operator (.true./.false.). Return error if the two section's types or sizes mismatch. See dra_write specs for discussion of request .

## dra_read

         status = dra_read(g_a, d_a, request)
                  integer g_a                       [input]  ! GA handle
                  integer d_a                       [input]  ! DRA handle
                  integer request                   [output] ! request id

Read asynchronously the specified global array from the specified disk resident array.

Dimensions and type of arrays referred to by handles g_a and d_a must match. If dimensions don't match, dra_read_section could be used instead.

See dra_write specs for discussion of request .

## dra_read_section

         status = dra_read_section(transp, g_a, gilo, gihi, gjlo, gjhi,
                                           d_a, dilo, dihi, djlo, djhi, request)
                  logical transp                    [input] ! transpose operator
                  integer g_a                       [input] ! GA handle
                  integer d_a                       [input] ! DRA handle
                  integer gilo                      [input]
                  integer gihi                      [input]
                  integer gjlo                      [input]
                  integer gjhi                      [input]
                  integer dilo                      [input]
                  integer dihi                      [input]
                  integer djlo                      [input]
                  integer djhi                      [input]
                  integer request                   [output] ! request id

Read asynchronously specified global array section from specified disk resident array section:

                    OP(d_a[ dilo:dihi, djlo:djhi]) -->  g_a[ gilo:gihi, gjlo:gjhi]

where OP is the transpose operator (.true./.false.).

See dra_write specs for discussion of request .

## dra_probe

         status = dra_probe(request, compl_status)
                  integer request                   [input]  ! request id
                  integer compl_status              [output] ! completion status

Tests for completion of dra_write/read or dra_write/read_section operation which set the value passed in request argument.

compl_status .eq. 0 means the operation has been completed.

compl_status .ne. 0 means "not done yet ".

## dra_wait

         status = dra_wait(request)
                  integer request                   [input]  ! request id

Blocks until completion of dra_write/read or dra_write/read_section operation which set the value passed in request argument.

## dra_inquire

         status = dra_inquire(d_a, type, dim1, dim2, name, filename)
                  integer d_a                       [input]  ! DRA handle
                  integer type                      [output]
                  integer dim1                      [output]
                  integer dim2                      [output]
                  character*(*) name                [output]
                  character*(*) filename            [output]

Return dimensions, type , name of disk resident array, and filename of DRA meta-file associated with d_a handle.

## dra_delete

         status = dra_delete(d_a)
                  integer d_a                       [input]  ! DRA handle

Delete a disk resident array associated with d_a handle. Invalidate handle. The corresponding DRA meta-file is destroyed.

## dra_close

         status = dra_close(d_a)
                  integer d_a                       [input]  ! DRA handle

Close DRA meta-file associated with d_a handle and deallocate data structures corresponding to this disk array. Invalidate d_a handle. The array on the disk is persistent.

## dra_flick

         subroutine dra_flick()

Returns control to DRA for a VERY short time to improve progress of pending asynchronous operations.