OT EOCOM Data Archive - Processing
JWH 8 Aug 2000

Introduction

This describes the scripts / codes / format used to retrieve EOCOM data stored on the mass store and save it to be used for SFIT or what have you. This is currently setup and running on sanjuan at /sj/jamesw/OTA. This process employs the INDEX file to find the filter region of a particular spectrum defined by a volume number and a sequence number range. The spectra (sequences) are coadded in groups of ten contiguous sequence numbers and stored in a compact format readable by SFIT preprocessing FORTRAN code on sanjuan (bnr format). The wavenumber range of the bnr file spans the optical filter bandwidth. During the creation of the bnr file two other files are created 1. '.rcd' file is a list of parameters from the header of each sequence in the average, and 2. '.lst' is the bnr header containing parameters for the averaged file. A header is 80 characters long and looks like:

V81300 14Mar92 17:15:27 82.28LT 14.79LN 88.58ZA 11.32AL NBM 4F 0001-0010

Each averaged spectrum is stored in its own file whose name is composed of the volume number and sequence range. The preceding header is for file V81300.0001-0010. The format of the header is also critical to the SFIT processing scripts.

Currently the scripts and code run on ouray. The code to open and read the mass store volumes is based on avgplt.f.

Processing

The rough outline is that you use the index to locate the sequence numbers and associated optical filters of the 'T'ransform volumes you are interested in. Then use this list to create an input file and script that is sent to ouray for processing. The four output files are returned in your rje.

More specifically copy /sj/jamesw/OTA to your logon. It should look more or less like:

sanjuan%ls -l /sj/jamesw/OTA
5688 -rw-r--r-- 1 jamesw ot 5822481 Jun 16 11:08 INDEX
3 -rwxr-xr-x 1 jamesw ot 2843 Aug 08 17:00 arch.k*
1 -rw-r--r-- 1 jamesw ot 1030 Aug 08 15:37 archive.seq
18 -rw-r--r-- 1 jamesw ot 18835 Aug 03 10:38 avgpltbat.f.arch
1 -rwxr--r-- 1 jamesw ot 911 Aug 08 16:02 avp.script.template.arch*
10 drwxr-sr-x 2 jamesw ot 10752 Aug 08 17:06 done/
3 -rw-r--r-- 1 jamesw ot 3063 Aug 07 12:10 format.bnr3
13 -rwxr-xr-x 1 jamesw ot 12888 Jun 16 11:03 setup*
0 -rw-r--r-- 1 jamesw ot 189 Aug 08 15:37 setup.inp
1 drwxr-sr-x 2 jamesw ot 1024 Jun 16 11:11 src/
Now take a look at setup.inp:

81303 81304
1 1900. 4000.
2 1400. 1810.
3 660. 1300.
4 2850. 3030.
5 1990. 2270.
6 1490. 1650.
7 750. 1050.
8 3970. 4880.
9 1760. 2000.
10 1210. 1380.
11 3030. 3170.

The program 'setup' takes the file INDEX and setup.inp and will sort all sequences of volumes 81303 and 81304 by filter number into groups of 10. Each line below the first is a filter number and the wavenumber range expected for that filter. These are then the wavenumber limits that will be returned in the binary file. Edit this to your liking. The limit on the volume number range is that all volumes should be in the same directory on the mass store ie. /COFFEY/AASE2. If you only want one volume enter it twice on the first line. When your happy with setup.inp run 'setup'. Its output will be the file archive.seq and is the input for the next step.

Now, run arch.k. It takes an optional argument 'step' to step through each entry in archive.seq otherwise it runs quickly to the end of the file. For each entry in archive.seq the script arch.k makes an input file and a script which it sends to ouray for processing. Note that there are a few parameters that may need to be set here at the beginning of a run and the script will pause so they can be checked. You'll see something like:

present working directory: /sj/jamesw/OTA
source file name: avgpltbat.f.arch
mss input path for T volume: /COFFEY/AASE2
transform volume prefix: T
input path on local machine: sanjuan.acd.ucar.edu:/sj/jamesw/OTA
output path on local machine: sanjuan.acd.ucar.edu:/sj/jamesw/rje

Batch mode production of binary spectra files for SFIT from OT archive.
If all of the above parameters are correct enter y to continue.

If you need to change anything stop here and edit the script. Everything you need (input files, scripts, templates) should be in the present working directory ($PWD).



Output

Output is returned to the rje directory given as output path in the script. The four files returned as output for the first entry in the above example are:
V81303.0001-0010 - bnr formatted binary file
V81303.0001-0010.lst - the header of the bnr file
V81303.0001-0010.rcd - list of 10 headers for each sequence
V81303.0001-0010.act - job accounting file

See the following attachment for the bnr file format. Note that this format is slightly different and more compact then the original bnr format. It is sometimnes referred to as bnr3 format.

-end-