0. About this recipe...

• A list of checks to be done and procedures to be followed during reduction & analysis of your LWS02 observation.
• A place to look for tips about how to deal with the aspects of the data reduction which are critical for a faint source.
• .....

• A tutorial about the use of ISAP and LIA
• An exhaustive description of ISAP and LIA functionality (refer to the tutorial documents).
• A detailed description of the ISAP and LIA Graphical  User Interfaces
• .....

1. Definitions & Requirements

• A faint source, for the purpose of this document, has a flux lower than 100 Jy at all wavelengths in the LWS range. Certain steep-spectrum sources will fall into this category only for a limited portion of their spectrum; this document is valid for that portion of the spectrum only.
• This recipe assumes you have a basic knowledge of the LWS instrument and its main sub-systems: detectors, grating, illuminators.
• This recipe assumes you are already familiar with ISAP's and LIA's GUIs, as well as with the main functionality available in those packages.
• This recipe assumes the following packages are available to you:
• ISAP Version 1.6 or later. An earlier ISAP version does not have all the functionality needed to follow all the steps and procedures described in this recipe
• LIA Version 7.2 or later
• This recipe assumes the following files pertaining to your ISO LWS observation are available to you, and have been obtained with Version 7  or later  of the ESA automatic data reduction pipeline (OLP):

• LSPD - Raw science data at an intermediate level of reduction; this will be the starting point for our data reduction
• LIPD - Calibration data obtained towards internal calibration sources during your observation
• LIAC - File holding the Dark Current estimates performed by the ESA automatic data reduction pipeline (OLP)
• LGIF - File holding the absolute responsivity correction factors estimated by the ESA automatic data reduction pipeline (OLP)
• LSAN - Calibrated science data produced by the ESA automatic data reduction pipeline (OLP)
• (IIPH - Pointing history file, contains information about the instantaneous pointing of the satellite during your observation)

2. Inspecting your Data

• Change directory where the files containing the data of your observations are stored
• Enter ISAP by typing the command isap at the prompt
• Load the LSAN file into ISAP
• Select all Scans, Detectors,  Lines, and  scan directions, and plot the data. Likely, you will see something like this:

 

 
 
 
 
 
 
 


 

Fig. 1

This is your calibrated spectrum; not very informative at this stage, but we can note that the noise of the spectrum  is higher in the region with lambda < 93um, corresponding to the SW detectors. There is nothing wrong with this observation; in fact, this is common for the LWS  and it is due to the combined effect of  higher NEPs and narrower spectral element size for the SW detectors. Let's make a couple of checks for memory effects and detector intercalibration:
 

• Memory effects. There are two different types of memory effects:

• Effects caused by energetic particles hitting the detectors. This event causes an abrupt modification of the detector responsively resulting in a jump of the signal output of the detector ('glitch') followed by a more or less slow recovery to normal behaviour. The recovery time is unpredictable; we have seen cases of almost instantaneous recovery, and cases where the detector took hours to go back to normal behaviour. Note that severe glitches are zeroed,, as are the next three data points, and are flagged as bad  by the pipeline processing, and these flagged  data are identified by ISAP at read-in time and are removed from the data set. Here is an example of a glitch  which was small enough to remain undeleted by the pipeline processing in an LWS02  grating scan (here the grating is scanning towards higher wavelengths.

 

 
 
 
 
 
 
 
 
 

•  

 For remaining glitches, the user can zap these in ISAP,  including  the trailing edge of each glitch.
 
• Effects caused by the average amount of flux falling onto the detector  There is a certain flux range for each detector (e.g. between 300 and 1000 Jy for detector LW2) where the response time of the detectors is a strong function of the temporal gradient of the signal; in other words the observed spectrum will be different for the two grating scan directions. As anticipated, this effect mostly manifests itself in well defined flux range for each detector; in case of a faint source we do not  expect it.  To check for this effect, on averaged or unaveraged data,  in  the  ISAP widget, change the Plot Style to Line Style: Connected and Colour Coding Preference: Scan Direction and plot.  You should see something like this:

 

 
 
 
 
 
 
 

Fig. 3

The green and the blue lines represent the two scan directions; you can zoom and closely inspect different portions of your spectrum. For faint sources you should not see any systematic differences among the green and blue lines (for the same detectors); if you instead see a significant difference (well above the noise of the green and blue lines) consider that the effect has not been modelled, so that it is not possible to correct for it. In case a line is present it is strongly advisable to keep the two scan directions separate and estimate the line parameters separately; the two line fluxes can be averaged later, but the two determinations will give you the best estimate of the uncertainty to assign to the average line flux.
 

• Detector Intercalibration.  Now change the plot style to Colour Coding Preference: Detector number and plot; you will now see something like this:

 

 
 
 
 
 
 
 
 
 
 

Fig. 4

In principle,  spectral segments from adjacent detectors, and  segments from a single detector should line up nicely; the overlapping portions of the spectra should match both in shape and in absolute value, but this generally does not happens e.g., at 128 microns above. What are the possible reasons for a mismatch ?
 

• Incorrect Dark Current subtraction.
• Incorrect estimate of the Absolute Responsivity Correction factors.
• The LWS flux calibration (both absolute and relative) is based on a point-like source (a planet). If the source is extended, we may expect detector mismatches depending on the extension as a function of wavelength (e.g., the source may be more extended at short wavelengths than at long wavelengths). A tool to apply a correction for the source extension (assuming extension larger than the beam and uniform brightness distribution) will be available in ISAP Version 2.0.


Now you should  quantify a bit this mismatch. Generally speaking you might be happy with a 10-15% mismatch, but this of course depends on the source. If the source is pointlike, you should expect less mismatch because (see above) the primary calibrator for the LWS is also pointlike and the beam is stable at all wavelengths.   Examine your spectrum and take notes about which detectors are more critical in the aspects we have just discussed.
 

• Negative Fluxes. There is another aspect you should note, which is not present in the observation we are using as an example. When the source is faint the calibrated fluxes can be negative, generally due to an oversubtraction of dark current. Note that we are speaking of systematically lower fluxes; a noisy spectrum of a faint source will certainly have negative points. Note down all detectors with systematically negative fluxes, as you will pay closer attention at them when you will revise the dark currents (below).

3. Reprocessing your Data with LIA

There are 4 steps in the LIA reprocessing of an LWS02 AOT:

• Dark Current estimate and subtraction
• Correction for time-dependent Responsivity drifts
• Estimate of the Absolute Responsivity Correction factors
• Re-calibration of the spectra in wavelength and flux

3.1 Dark Currents

While at the ISAP> prompt, type: IA_DARK, tdt='TDT', where TDT is the eight digit number attached to the filename of all you data files. If you are running ISAP in a directory which is different from the one where your data files are stored, an additional parameter has to be given; the call would then be: IA_DARK,tdt='TDT',dir='your directory'. Once the widget is up, click on detector  such as LW1. You will see something like this:

Fig. 5

The red crosses are the DC measurements, while the white points are your (uncalibrated) data; everything is plotted as a function of time. If your observations was taken at a revolution number earlier than about 400 you may see DC measurements (red crosses) taken in between the observation; however only the first and the last (before and after your observation) DC measurements can actually be used.

Here the various scans of the grating can be recognized as an oscillating pattern on your data.  The discontinuities in the oscillation segments above represent the different "lines" scanned in the execution of the LWS02 AOT. Remember that this data is still uncalibrated at this stage and the transmission profiles (called RSRF, 1 with the appropriate RSRF segment or each line/detector combination) of the instrument are still to be divided out. To check where the different lines and cans start and end, go with the mouse on a point and click the middle button of the mouse: the text area at the bottom of the IA_DARK widget will give you all the information regarding the nearest point to the mouse actual position, e.g.:

Raster ID is 1 1 - Line # 0 - Scan # 0 - ITK = 228327855 - Phc = 8.64105e-16 Amps

Again, it is important to remember that at this stage the instrumental transmission is not yet divided out and, if there is sufficient flux from your source, it will manifest itself as a periodic pattern throughout your dataset.  It is important to notice that that glitches are present also during the dark current measurements; if you bring up the second  DC measurement you will see this:

Fig. 6

You will note the  gap between the first and second point here - a glitch occurred  and was removed  possibly leaving a trailing ramp.   The median-clipped averaged performed on the whole set of points above gives the DC estimated made by the OLP, which is visible in the box 'OLP Used...' in Fig. 6; it is clear however that this may be an overestimate because at least the point after  the gap is artificially high.  (This is a fairly mild example of this effect. )   It seems that the best estimate should be based on all but the second point, which we mask out and go ahead and make the DC estimate without it.

When you come to the subtraction of your estimated DCs from the data, a lot of common sense is needed. Take the example in Fig. 7.

Fig. 7

The problem here is that the source flux  (white)  is weaker than the first dark current measurement (first set of red points, with the mean shown as a red line). Yet the source flux includes dark current. This indicates that the first dark current measurement is measuring more dark current than is present during the observation and is not reliable. One possible choice is to rely solely on the second dark, which is weaker than the source flux (red points at the bottom) or else give up on making the correct dark subtraction and instead  remove all the dark current and continuum by  interpolating a "dark" function following the lower envelope of your data.  However, note here how the glitches  (SW1 has the worst) are altering the detector gain in time and causing steep falloffs. Compounding the problems here is the fact that their are  discontinuities caused by switching to  the different line wavelengths. For multi-line LWS02 observations, when dark measurements are unreliable,  systematic changes in dark current and gain are difficult to untangle from intrinsic line and continuum fluxes associated with the successively observed lines.

Likewise, with an LWS02  Raster Map ,  signal variation  is bound to occur during the observation due to the source's spatial structure, and between the starting and ending dark observations, it is practically impossible to isolate a systematic trend in the instrument behaviour .

Consider the ia_dark result for the LW1 detector for the same observation...
 

Fig. 7a
 

You should by now be convinced that the oscillations you see in the signal (white points) in Fig. 7a is due to spectral scanning back and forth over the line region interval and is dominated by the detector's relative spectral response  (which has not yet been divided-out) - and the oscillation is seen here in 5 different  segments, one for each line observed. The oscillations will disappear when we process our spectrum to the AAR stage. The best (and most conservative) thing you want to do at this stage is to assume that the DC is constant through out the observation. In general, if the DC measurements done before and after the observation are comparable (as with the example above), within the noise of the single measurements, we recommend you choose the linear interpolation option which uses the average of the two measurements. This is the option applied by the OLP, and unless the two measurements are significantly different or there is a clear DC trend in your data, you should stick with that option.

 In general, the advanced options of ia_dark; the option to add points and to fit with polynomial functions, are not advised for LWS02.... typically, there is insufficient data to be able to decide to take such actions. The exceptional case would be where a line was observed by scanning so many times that an instrumental trend was visible.  In such a case, the user is advised to distinguish between
dark variations, which elevate or lower the mean signal strength, leaving peak to peak variations constant, and responsively trends, which change the span of peak to peak variations range and cause line fluxes obtained in individual scans to change. If your data are of sufficient quality (i.e., you can see the effect described below) then here are the corrective actions:

• Responsivity trend. Do nothing at this stage; here we can only correct for DC trends. You will be able to correct for gain trends in IA_DRIFT
• DC trend. Add Points throughout your data and Interpolate Linear using All darks.

3.2 Responsivity Drifts

While there may be responsively drifts within the period of time that an LWS02 observation is taken, at this time there is no way to isolate and correct these drifts. However, absolute responsively changes  (a constant detector gain change from the nominal) may be measured and corrected as described below.

3.3 Absolute Responsivity Correction Factors
The absolute responsively correction factors can revised using the routine IA_ABSCORR, whose tutorial is available at http://www.ipac.caltech.edu/iso/lia/abs_corr.html. Since the estimate of these factors is based on the Illuminator Flash data, the characteristics of the source observed are not important; the tutorial has all the information you need to proceed.

3.4 Recalibration

4. Data Analysis with ISAP

• Zap the bad data. We have to manually edit the data for each detector  and line separately to identify and discard the clear outliers resulting from glitches. If you have a large enough data set (i.e. more than 10 scans) you might want to split apart your AAR according top the detector number. In fact after each zapping ISAP has to reload all the structure and it may take few seconds to do it; if you multiply these 10 seconds by the number of zappings that you will have to do this will result in an unacceptable overhead. If you find out that this is the case, it is better to SPLIT you AAR and work on smaller datasets. If in the preliminary analysis you identified detectors which showed memory effects, it is better to work on each scan direction separately. Be careful to only zap clear outliers; in the example below we can clearly recognize few glitch trails (the orange, green,  and blue scans): zap them out. Also zap the  outlier at the right hand of the spectrum.

 

 

Fig. 9

Do not start zapping in the noise; you should only zap what's really systematic otherwise you will modify the statistics of the measurement and the averaged spectrum will be perturbed.

Do this zapping for all detectors; remember to STORE the AAR after finishing zapping each detector. If you SPLIT apart your AAR, it  is time now to merge all the pieces back into a single AAR.

• Average. Select all the data and send it to Average. The default settings can be used to give it a try; if you already know that there are no memory effects in your data, e.g. no systematic difference between scan directions for the same detector, de-select the Scan Direction check box at the top left corner of the Average widget. If you have memory effects is better to average scans for each scan direction separately (i.e., leave the default selection for the check boxes at the top left of the widget); memory effects, if any, are not in general present in all detectors so the easiest thing is to do two averages with the two options.

 

 
 
 

Here are some guidelines for averaging. If you have a limited number of scans you may want to increase the bin size to gain a bit in S/N at the expense of the spectral resolution. Consider that unless you are observing, e.g., a supernova remnant where the line can be broader than 1500 km/s, your line will be unresolved and there is nothing band in increasing a bit the bin size to push down a bit the noise. Play also a bit with the sigma clipping threshold; a too high value will not discard enough outliers, while a too low value will start discarding too many points; there should be an optimum value which minimizes the noise level of the averaged spectrum. We cannot give more precise guidelines here because this is a function of the data. Also, the bin size and sigma values which give good results for one detector may be not optimized for another detector; if this is the case you can always send separate detectors to Average and merge the results back into a single AAR later (remember to STORE each averaged piece before you go to the next). The averaging algorithm is another parameter you can play with; the Mean and Median methods do not clip outliers and are not recommended. The Standard Clip & Mean does one single pass to discard outliers, while the Special Clip & Mean is more drastic and does iterative passes until there is nothing more to discard. Essentially play with the bin size, sigma and the averaging method until you are happy with your result; the precise combination of these three parameters will be a function of your data.

•  Extended Source Correction. A new functionality is available under ISAP v2.0 and later: the possibility to apply a correction to your spectrum to make the LWS calibration valid for extended sources. This is needed because the calibration files used to reduce your data were derived using a point-like source. By comparing the amount of source flux which is clipped by the focal plane aperture in the two extreme cases of point-like and infinitely extended and uniform source, the instrument team could determine a set of correction factors which, after interpolating, can be applied to your data. This is a gain correction which can reduce your LWS fluxes by as much as 40%. This correction should be applied for extended sources only.

 

 

Q. "This is all very nice. But my source is not infinitely extended and it not uniformly bright!! What should I do ?"
A. This is of course a most tricky and common situation, but unfortunately there is no definite indication; the truth will be in the middle. However, from tests done on real cases (extended galaxies, hence non uniform) it seems that the correction works well for sources more extended than 3-4 arcmin.

• And now ....? Now the fun starts!!

• Convert from flux to brightness
• Fit lines and derive integrated fluxes
• Fit combinations of blackbodies to have a first idea of the properties of the continuum spectrum
• Integrate the SED under the IRAS filters to compare with the PSC.
• Smooth the spectrum
• Rebin the spectrum to a different wavelength scale, or to the same scale of another observation
• Make arithmetic operations between two spectra
• ....