Processing FTIR spectra:
running FASTCODE and SFIT on sanjuan
August 2000 JWH

Introduction

The purpose of the code and scripts described here is to facilitate fitting multiple regions in multiple spectra from differing spectral data sets easily, efficiently and more importantly in an homogeneous way. Ultimately we want the most consistent analysis across data sets. We also want a system and structure that is reasonable, thourough and doesen't need to be changed frequently. We need the ability to process many spectra in a batch mode with as few dropouts as possible but on the other hand we need to fit a single spectra many times, largely during setup and be assured that this process is identical we the batch processing to follow.

There are many ways this can be accomplished. What has been developed is an hierarchical group of scripts, supporting pre-processing code, a directory structure and post-processing code including plotting programs which achieves our goals. The scripts are korn shell scripts and so can be run easily on any UNIX machine. Pre- and post-processing code is written in C or FORTRAN. SFIT and FASTCODE are FORTRAN. Plotting code is in IDL. The compiled code now only runs on sanjuan, eventually it will be compiled on the computer which replaces sanjuan or acd.



Design

The scripts use a specific directory naming convention and structure to simplify acquiring input files and storing output files. We'll start by looking at the structure and giving a description of what it is for and perhaps why it is setup this way. The directory tree looks like:

<your-directory>:
<molecule_name.wave#>
HCl.2926/
binput
HF.4039/
binput
...
spectra.1/
a.nrr
b.bnr
...
fasc41.prf
spectra.2/
q.bnr
r.bnr
...
fasc41.prf
auto/
fast/
fitbatch*
fasc
prebnr*
slist
rlist
plot/
save/
sfit/
test/
setup/
xtra/

Starting at the top <your-directory> can be anywhere there is enough diskspace. In <your-directory> setup as many <molecule.wave#> region directories as you have regions to fit ie. unique binput files for SFIT. There are too many parameters unique to a specific region/fit that it would be too much effort to write each binput from some other set of inputs everytime we have a spectrum to fit. On the other hand the binput file in each of these region directories has a couple or seceral unique patterns for sed to recognise that must be correct and it would be wwise to save a copy of a working binput before editing it in the course of optimising the fit. Currently no other region-specific files necessary to the process are stored in these directories.

Listed next are spectra.? directories which hold the bnr files and a reference profile that defines the atmospheric model applicable to all the spectra stored in the directory. Although the spectra.? directories are listed here, in general they can be anywhere on the system as long as they can be copied to the auto directory. Since the fasc41.prf file in these directories defines the atmosphere model used in the forward calculation for these spectra, you may need several directories to analyze a complete data set. For example, one for inside the vortex in mid-winter, one outside, one inside early spring etc.. Although the data in the fasc41.prf varies with the model the format is unique and the name must remain the same in each directory.

The auto directory is where the scripts reside and is basically the working directory from which you run the batch processing. Here is the main script fitbatch and its input files slist and rlist which are described later. The executable prebnr is called by fitbatch and obtains the header data from the bnr file. The file fasc is the sed-able input file for FASTCODE. The sed patterns must be left alone but the layers may need to be edited to suit the spectra being fitted. Finally are the subdirectories fast, sfit and plot. Fast and sfit are where FASTCODE and SFIT are run respectively just to keep the working files separate. Plot is not used right now. Save and test contain extra code and input files used in developing the scripts.

Next is setup in the main directory which contains input files and the sfitinput executable that can aid in creating a new binput from a menu driven program. This is not too useful anymore. The xtra directory contains spare code for dealing with the bnr files and .prf files which should save time in preparing data files.









Here the few steps necessary to fit the NPL data on sanjuan. Steps 1 & 2 are specific to the NPL data because of their format. Step 3, using the script fitone is the standard route to take when processing a spectra when a FASTCODE run must also be performed. fitone calls a script fasc.ksh to run FASTCODE. When SFIT only needs to be run use the script sfitnx which is also called by fitone. When many spectra need to be processed use fitbatch which in turn calls fitone. Note that all the filenames are hard coded in all the scripts and some programs.

To get started copy the data Hfnnn.sf to a working directory. Make sub directories fastcode, sfit, plot. Copy to the working directory:
a. fasc0
b. fasc1
c. fasc0.prf
d. binput
e. sfit.dir
f. fitone.inp

Examples can be found in /sj/tools/sfit/input.files. Then perform the following steps. If you need to adjust the layers in the fasc0 & fasc1 files. fasc0.prf is a 41 layer input profile for FASTCODE. The profiles can be edited but we must maintain the 41 layers. Be careful not to change the sed patterns in fasc0, fasc1 and binput. sfit.dir points to the cfgl files used in SFIT v1.09eep. fitone.inp is edited for each run.

Output from SFIT are:
a. bdetail - standard verbose SFIT output
b. short - pertinent information used in Result and the plot header
c. Result - continually appended with data from short
d. pbpfile - the input, fit and difference spectra in ascii format

These files are left in the sfit sub-directory but Result is also copied back to the working directory. These (excluding Result) get overwritten with each call to fitone...

1. Make the correct binary spectrum file.
The input file Hfnnn.sf are given by NPL. These are written in C on a pc. We need to change it to a FORTRAN readable file and swap the bytes for sanjuan. Run:

c2bnr3 Hfnnn.sf

This creates a file temp.bnr3. It is readable by the SFIT spectrum processing file specasc3.

2. Compute the solar zenith angle for the spectrum.
It appears that the time and date are the first two strings in the bnr3 header so run:

solela

This reads temp.bnr3 and computes the zenith angle needed for the FASTCODE calculation. See that it prints out the bnr header. Also check that the time and date used in the calculation are the same as in the header.

3. Run script fitone which in turn runs FASTCODE and SFIT.

To do this you need the following input files:
a. fasc0 - input file for FASTCODE it is copied by a sed command to fasc0.inp in sub-directory fastcode
b. fasc1 - ditto
c. fasc0.prf - the reference profiles made from reftmo96.41 & zpttmo41.nov
d. fitone.inp - explained below
e. binput - input file for SFIT it is copied by a sed to the sub-directory sfit

fasc0 & fasc1 are sed-able and if you have the right ones they do not need to be edited. Also the .prf file is the same for all fits. fitone.inp must be edited for each run. It looks like:

/sj/thule/NPL-TMF the full path to the working directory
52.8860000 zenith angle form solela calculation, sed needs all 10 digits
2.58000000 altitude [km] at TMF
128.600000 maximum path difference [cm] (0.9/res) res given in bnr header
2.954 field of view [mr] given in bnr header
0 SFITapodization index 0=boxcar, OPUS designation 'BX' given in bnr header
1 yes do a FASTCODE calculation
Result filename in which to store the abbreviated results, suitable for making a list

binput is also sed-able so if you have the right one don't change the sed patterns in the file as you change how SFIT fits the spectrum.

4. Plotting the spectra and results.
There are (at least) two plotting programs sfitplt and sfitplt1.pro. sfitplt is a FORTRAN program which makes a gmeta file. For now use sfitplt1.pro in idl. The source for this IDL routine is in /sj/tools/bin and when you are in IDL it should see it if not put the following into your .profile (and source it) :

export IDL_PATH=/sj/tools/bin:$IDL_PATH

Go to the sfit sub-directory startup idl and enter

.run sfitplt1

% Compiled module: SFITPLT.
% Compiled module: USAGE.

Then enter:

sfitplt, "binput", "lowres", "portrait"

Substitute hires for lowres on a high resolution monitor and landscape for portrait to change the postscript file. A plot should appear in an X-window and a file pbpfile.ps should be left in the sfit sub-directory


Converting OPUS data files to bnr format.

To fit a spectra made on the Bruker using SFIT on sanjuan we first convert the 'Single Channel Data Block' portion of the OPUS data file to a bnr3 file. This requires that when the OPUS file is made there is a 'Single Channel Data Block' in the file. The current program was made to run from a script so it uses an input file and no user interaction. Find the sample input file /sj/tools/sfit/input.files/o2bqx.inp, copy it to the working directory. It lools like:

28aug97.0 - OPUS data file name
0 - time offset between spectrum timestamp and UTC (bruker pc is on UTC)
39.0 - N latitude of observation
105.0 - W longitude of observation

After the input is set up run

o2b3q

It creates three files:
a. temp.bnr3 - bnr3 file of the spectrum
b. o2bqx.out - contains information needed to run SFIT, it looks like:

00047.0919 - zenith angle, note sed likes all 10 digits
00257.1429 - maximum path difference [cm]
2.955 - field of view [mr]
BX - apodization code from OPUS, SFIT apodization index 0
2390 - beginning wavenumber in bnr file
2730 - ending wavenumber
5/3/1999 15:49:6UT Z:47.09 DUR:6504.89 RES:0.003500 APD:BX FOV:2.9545mr
- header in bnr file

c. o2bqx.err - more info mainly about the progress of the program

With the temp.bnr3 and o2bqx.out files you can setup binput and the fastcode input files and run SFIT.