# Notes on running NWChem on the ANL IBM SP1

Much of the information below has been exracted from information available through the web at ANL or IBM SP man pages by Robert Harrison (rj_harrison@pnl.gov).

For NWChem support mail nwchem-support@emsl.pnl.gov or visit the NWChem homepage.

For support on SP specific issues contact spsupport@mcs.anl.gov.

## General SP information

ANL use their own environment (EASY) for scheduling interactive and batch use of the machine. You need to know about just a few facts and commands to get going.

You will log into either `bonnie` or `clyde.mcs.anl.gov` which are not really part of the SP2 - they are 'front-ends'. Information on the SP, examples etc. can be found on the MCS SP Web pages.

The command `sphelp` gives a useful summary of SP specific commands and is a good starting point ... here is the sp command summary

`cacReport`
  Prints report on current state of CAC's allocation.
`spfree`
  Prints the number of nodes that are currently free.
`spq`
  Prints out who's running what on the machine and the queue, how much time they requested or when they'll finish.
`sprelease`
  Is used to remove jobs, from either the queue, or the machine, if the job has already started. This returns node(s) back to the system, and stops the charging against the Charge Allocation Category (CAC) on the ANL system.
`spsubmit`
  Used to submit jobs to the queue.
`spstatus`
  Prints out the current status of the SP2.
`spusage`
  Shows who is using what nodes. The format is:

        node# 0|1 username JID JobTerminationTime (MMDDhhmm)
        0 = node is available, 1 = in use.

`xspusage`
  Same as above except for x windows.

To run either interactively or with batch you have to schedule the resources using spsubmit. By mail you'll be told when the resources become available. If running interactively, you can then login into the nodes you've been assigned (details below) or if running in batch you provided spsubmit a script to run (provided for NWChem). From prior experiences, you may experience a significant delay in receiving mail that you have the nodes. The use of `xspusage` is best (see below). If you don't have X, use `spq`.

If a node is not currently assigned to you then you will not be able to log into it.

Each node of the SP is a Power 1 cpu with 128 MB of physical memory and a local scratch disk named `/tmp` of about 0.25 to 0.4 GB. The O/S and I/O buffers consume about 10 MB (estimate), and the NWChem executable is 5 MB, leaving about 113 MB to be allocated. The local scratch disk is wiped clean at the end of each job. Thus, in order for useful files to be saved by default and to make sure that scratch files with high bandwidth requirements are in `/tmp`, you should always

- specify full path names for files, and
- ensure that any directory names are accessible from all nodes (i.e., either just `/tmp` for the local disk on each node, or a subdirectory of your home directory).

## Running NWChem in batch

Do this first - it's a lot less confusing than running interactively.

In the directory `/sphome/harrison/hpcci` are several files

`ibm_sp@anl.html`
  this file
`nwchem`
  the SP executable
`library`
  the basis set library
`job`
  a script for running spsubmit
`runnwchem`
  a script for running nwchem in batch
`interactive`
  a script for running nwchem interactively
`examples`
  a directory with example inputs and outputs for NWChem
`user.ps.Z`
  NWChem documentation (compressed postscript)

Copy one of the example input files into your login directory (e.g., `/sphome/harrison/hpcci/examples/scf_h2o.nw` - input for a conventional SCF calculation on water) into your login directory. (This input file is also commented and tailored to the SP).

To run the calculation in batch you must use `spsubmit` to

- allocate the nodes
- specify the script to run
- provide the arguments (work directory, to the script.

The standard script `/sphome/harrison/hpcci/runnwchem` takes 3 arguments

1. the work directory (made if necessary)
2. the full pathname of input file
3. the full pathname of output file which defaults to the name of input file with .nw replaced by .out

e.g.

         runnwchem /tmp /sphome/harrison/test.nw

or

         runnwchem /tmp /sphome/harrison/test.nw /sphome/harrison/test.out

`Spsubmit` prompts for the parameters it wants ... here is a sample dialog that runs a five minute nwchem job on eight nodes in batch (when you run this use your own username instead of harrison).

        /sphome/harrison % spsubmit
        *********************************************************************
        112 nodes are available
        16 nodes are allocated for I-Way use

        Please report any problems to spsupport@mcs.anl.gov
        *********************************************************************
        Charge Allocation Category: [default harrison]
        Maximum Wall-clock Run-Time (minutes): (1-???) 5
        Number of Nodes Required: (1 - 128) 8
        CAC:  "harrison"  *now* has 4397 RUs available, after committing.
        (I)nteractive (B)atch: B
        Job Classification (M)PL, (T)ask Farm, (P)4: M
        IP over the switch [y/n]: Y
        Full path to Shell Script/Program: /sphome/harrison/hpcci/runnwchem
        Command Line arguments for your job: /sphome/harrison \
                                             /sphome/harrison/scf_h2o.nw
        (C)ommit or (A)bort: C

In order to be crystal clear, I typed (comments in parentheses)

        spsubmit (the command)
        carriage return (to accept charging to the default account)
        5 (for a 5 minute job)
        8 (for 8 processors)
        B (for batch)
        M (for MPL)
        Y (to let IP also run over the switch)
        /sphome/harrison/hpcci/runnwchem (name of script)
        /sphome/harrison /sphome/harrison/scf_h2o.nw (runnwchem arguments)
        C (to commit)

The script `/sphome/harrison/hpcci/job` runs `spsubmit` from input in a file - take a copy and edit it to automate use of `spsubmit`.

The command line arguments given to the `runnwchem` script specify the work directory (`/sphome/harrison`) and the full path to the input file (`/sphome/harrison/scf_h2o.nw`). `Runnwchem` will leave the output in `/sphome/harrison/scf_h2o.out`.

Since it can take a large elapsed time to get a job started five minutes is the practical lower limit that should be used for the job time.

The command `spq` will indicate the position of the job in the queue and its status. When the job is started you'll receive mail indicating the nodes actually allocated. At the end of the job you'll receive another mail message providing the output of the shell script. Only the NWChem output goes into the output file. The advantage of doing this is that you can examine the output while the job is executing. The unix command '`tail -f filename`' is useful to watch a job run.

Note that only one node (the master) executes the script and that only the commands run with POE are actually executed in parallel. This causes seemingly simple things to fail. For instance, if you request to run in a subdirectory of `/tmp` (e.g., `/tmp/rjh`) then `runnwchem` will make this directory and `cd` into it. Subsequent POE commands will fail since this directory will not be present except on the master node.

If you want to run multiple NWChem calculations in the same job you'll have to modify the `runnwchem` script. Basically, this just requires replicating the line running `poe` and specifying the correct input and output file on each line. Make sure that you leave the `sprelease` at the end of the file since it ensures that you won't be charged for more time than you use.

## Running NWChem interactively

Again, you use `spsubmit` to reserve nodes for your use, but this time specifying interactive access. In this instance it will not prompt you for a script or arguments. Mail will notify you when your nodes are available. Alternatively, you can keep looking with `spq` to see when your job starts. There is also a graphical program (`xspusage`) that names nodes according to who 'owns' them.

The mail message, or the file `SPnodes.$JOBID` in your login directory (you can get the jobid from the output of spsubmit), or `xspusage` tell you which nodes you have access to. Simply telnet to one of these and login with your usual id and password, or with `xspusage` you just click on a node you own to open an xterm on that node.

Now that you're logged into the partition you can run NWChem using the `/sphome/harrison/hpcci/interactive` shell script. This differs from the `runnwchem` script only in that output comes to the terminal and the `sprelease` has been removed. Thus, you can run multiple calculations in one interactive session.

Don't forget to do an `sprelease` with the correct jobid after completing the calculations otherwise you will be charged for all the time that you booked, rather than just the time that you used. Once `sprelease` is issued you no longer own the nodes and you will be unable to log into them.

E.g.,

      (... log into spnode ...)
      interactive /tmp/ /sphome/harrison/hpcci/examples/scf_h2o.nw
      interactive /tmp/ /sphome/harrison/hpcci/examples/mp2_h2o.nw
      interactive /tmp/ /sphome/harrison/hpcci/examples/dft_h2o.nw
      sprelease 070714025108
      exit

`Xspusage` is the easiest way to run interactively. Before using `xspusage` make sure that your `DISPLAY` environment variable is correctly set (`hostname:0.0`) and that your local X server permits remote window creation (use the command '`xhost +`' in a local shell).

## Troubleshooting

Keep the email with the `runnwchem` shell script output since this can be used to identify nodes with problems.

#### Job dies with a strange error code and an unknown error message

I've had this happen when I've specified a work directory that is not accessible to all processes (e.g., a subdirectory of `/tmp`). This often works for a single process, but fails for multiple processes since only the master process has access to this directory. Rerun the job specifying either just `/tmp` or a subdirectory of your login directory.

#### Job dies complaining of undefined symbols

This was a problem with incomplete instalation of a system Fortran upgrade. It should be fixed, but if the problem appears contact `spsupport@mcs.anl.gov` sending them the shell script output and the output in your logfile. They may ask you to rerun it with the `MP_INFO` variable increased to 3 or 4 in order to generate debug information. To do this take a copy of the `runnwchem` script and edit it.

#### Timeout error messages

I haven't seen this problem but apparently it can arise from excessive load from others on the Ethernet. Just resubmit the job.

#### Running out of disk space

Some of the lower numbered nodes apparently have some system junk sitting in the `/tmp` filesystem that is not automatically deleted. You can rely on only about 0.25 GB per node, though some will have up to 0.4 GB.

#### Accounting

Development accounts have just 125 node hours per quarter - I blew this in two days. The command '`cacReport -u userid`' reports the time available to you. The accounting is not rigorously enforced and a request to `spsupport` to increase your time will usually be granted.

#### Expansion of ~ (tilde) on SP nodes

Only the user that currently has allocated an SP node can log into it. This is done by restricting entries in the node's `/etc/passwd` file. A side effect of this is that an SP node has no knowledge of other users. Hence, the usual shell expansion mechanism for specifying a login directory by prefacing a user name with a tilde will not work, unless you are referring to yourself. The rule of thumb is always to use fully qualified filenames.

## NWChem support

Report NWChem problems, suggestions, feedback, etc., to `nwchem-support@emsl.pnl.gov` or via the NWChem home page.

There is a mailing list for NWChem users that can be used for announcements and discussion amoung users. To subscribe send email to majordomo@emsl.pnl.gov with the body '`subscribe nwchem-users`'. You can do this with the following command

         echo subscribe nwchem-users | mail majordomo@emsl.pnl.gov

To post a message to the mailing list send it to nwchem-users@emsl.pnl.gov.

## Acknowledgements

Thanks to Dave Bernholdt and Mike Minkoff for useful input to this document and to Jeff Tilson for an initial version of the execution scripts.