0. About this recipe...

• A list of checks to be done and procedures to be followed during reduction & analysis of your LWS01 observation.
• A place to look for tips about how to deal with the aspects of the data reduction which are critical for a medium-brightness source.
• .....

• A tutorial about the use of ISAP and LIA
• An exhaustive description of ISAP and LIA functionality (refer to the tutorial documents).
• A detailed description of the ISAP and LIA Graphical  User Interfaces
• .....

Recent changes: Sects 3.2 and 3.4.

1. Definitions & Requirements

• A medium-brightness source, for the purpose of this document, has a flux higher than 100 Jy but lower than 1000 Jy around 100um.
• This recipe assumes you have a basic knowledge of the LWS instrument and its main sub-systems: detectors, grating, illuminators.
• This recipe assumes you are already familiar with ISAP's and LIA's GUIs, as well as with the main functionality available in those packages.
• This recipe assumes the following packages are available to you:
• ISAP Version 1.6 or later. An earlier ISAP version does not have all the functionality needed to follow all the steps and procedures described in this recipe
• LIA Version 7.2 or later
• This recipe assumes the following files pertaining to your ISO LWS observation are available to you, and have been obtained with Version 7 of the ESA automatic data reduction pipeline (OLP):

• LSPD - Raw science data at an intermediate level of reduction; this will be the starting point for our data reduction
• LIPD - Calibration data obtained towards internal calibration sources during your observation
• LIAC - File holding the Dark Current estimates performed by the ESA automatic data reduction pipeline (OLP)
• LGIF - File holding the absolute responsivity correction factors estimated by the ESA automatic data reduction pipeline (OLP)
• LSAN - Calibrated science data produced by the ESA automatic data reduction pipeline (OLP)
• (IIPH - Pointing history file, contains information about the instantaneous pointing of the satellite during your observation)

2. Inspecting your Data

• Change directory where the files containing the data of your observations are stored
• Enter ISAP by typing the command isap at the prompt
• Load the LSAN file into ISAP
• Select all Scans, Detectors, and the two scan directions, and plot the data. Likely, you will see something like this:

 

 

Fig. 1

This is your calibrated spectrum; not very informative at this stage, but we can note that the noise of the spectrum  is higher in the region with lambda < 93um, corresponding to the SW detectors. There is nothing wrong with this observation; infact, this is common for the LWS  and it is due to the combined effect of  higher NEPs and narrower spectral element size for the SW detectors. Let's make a couple of checks for memory effects and detector intercalibration:
 

• Memory effects. There are two different types of memory effects:

• Effects caused by energetic particles hitting the detectors. This event causes an abrupt modification of the detector responsivity resulting in a jump of the signal output of the detector ('glitch') followed by a more or less slow recovery to normal behaviour. The recovery time is unpredictable; we have seen cases of almost instantaneous recovery, and cases where the detector took hours to go back to normal behaviour. Here is an example where 5 'glitches' follow one another during a grating scan (here the grating is scanning towards lower wavelengths):

Fig.2

We tried to model, among other things, the recovery time as a function of the signal jump, but without success; the first three points of each jump have actually been discarded by the 'deglitching' algorithm present in the first stage (ERD ===> SPD) of the automatic data reduction pipeline and they do not appear in Fig. 2. The only possibility is then to zap the crap; later, in ISAP, we will have to discard the trailing edge of each glitch.
 
• Effects caused by the average amount of flux falling onto the detector. There is a certain flux range for each detector (e.g. between 300 and 1000 Jy for detector LW2) where the response time of the detectors is a strong function of the temporal gradient of the signal; in other words the observed spectrum will be different for the two grating scan directions. For medium-brightness sources we expect this effect to be present, especially in detectors LW2 and LW3. To visualize this effect we have to average the data; for this particular purpose we can use the default averaging options and use Special Clip & Mean as the averaging method. When you are back in the main ISAP widget, change the Plot Style to Line Style: Connected and Colour Coding Preference: Scan Direction and plot.  You should see something like this:

 

 

Fig. 3

The green and the blue lines represent the two scan directions; you can zoom and closely inspect different portions of your spectrum. Without even zooming portions of the above plot, we can already note dramatic differences among scan directions for detectors SW1, SW2 and LW2; a closer inspection show we have similar problems on SW3 and LW3.
 

• Why these problems ? The response time of the detectors is different depending if the signal increases or decreases with time; in particular, detectors are faster to respond to decreasing signals. I have deliberately used the term signal and not the term flux because what the detector sees is the source flux convoluted with its Relative Spectral Response Function (RSRF). Since intrinsic source spectra are in general shallow (not considering lines and spectral features in general) over the wavelength range covered by each detector, the RSRF is the function that tells you when the detectors is experiencing an increasing or a decreasing signal. Now, let's examine in more detail the situation of detector LW2:

Fig. 4

N.B. the green is for Scan Direction =0, where the grating scans from higher to lower wavelengths.
And now let's see what is the RSRF for this detector:

Fig. 5

The turnover of the RSRF is around 120 microns. Since the green scans are done towards decreasing wavelengths, the part of a green scan from 133 to 120 microns will experience an increasing signal and the detector will be "slow" in responding; the recorder signal will therefore be lower than the real signal, because the detector has not enough time to adjust its response to the increased signal before the grating moves again. Below 120 microns the green scans will be ok. On the other hands the blue scans start from 102 microns towards higher wavelengths, and the detector will experience increasing incoming signals up to 120 microns; for this range the detector output will lead to an underestimate of the source flux. Summarizing: trust the green scans longward of 120 microns, and the blue scans shortward of 120 microns.
You should make similar considerations for all detectors for which you see this problem occuring in Fig. 3. The RSRF for all detectors can be found at http://www.ipac.caltech.edu/iso/lws/lia/glossary.html#Relative Spectral Response Function (RSRF).
 
• How do I proceed in the analysis ? When you will have completed you LIA reduction (see below) and you will start the analysis, remember not to average across scan direction for the detectors where the above problems are spotted.
• Detector Intercalibration.  Now change the plot style to Colour Coding Preference: Detector number and plot; you will now see something like this:

 

 

Fig. 6

The spectra of the 10 detectors should line up nicely; the overlapping portions of the spectra should match both in shape and in absolute value (once the correct scan direction is picked for that particular detector), but this generally does not happens. What are the possible reasons for a mismatch ?
 

• Incorrect Dark Current subtraction. This should not be an issue for medium brightness sources; it is anyway advisable to check the DCs.
• Incorrect estimate of the Absolute Responsivity Correction factors.
• The LWS flux calibration (both absolute and relative) is based on a point-like source (a planet). If the source is extended, we may expect detector mismatches depending on the extension as a function of wavelength (e.g., the source may be more extended at short wavelengths than at long wavelengths). A tool to apply a correction for the source extension (assuming extension larger than the beam and uniform brightness distribution) will be available in ISAP Version 2.0.


Now you should  quantify a bit this mismatch. Generally speaking you might be happy with a 10-15% mismatch, but this of course depends on the source. If the source is pointlike, you should expect less mismatch because (see above) the primary calibrator for the LWS is also pointlike and the beam is stable at all wavelengths. If your source is extended or there another source close enough (<80" for a 300 Jy source)  to induce fringing on your spectrum (which is not the case for the example in Fig. 6), then the situation is more critical because the LWS beam is modulated and its FWHM may vary of about +/- 10% within each detector band (the amplitude of the effect increases with wavelength). Examine your spectrum and take notes about which detectors are more critical in the aspects we have just discussed.

3. Reprocessing your Data with LIA

There are 4 steps in the LIA reprocessing of an LWS01 AOT:

• Dark Current estimate and subtraction
• Correction for time-dependent Responsivity drifts
• Estimate of the Absolute Responsivity Correction factors
• Re-calibration of the spectra in wavelength and flux

3.1 Dark Currents

While at the ISAP> prompt, type: IA_DARK, tdt='TDT', where TDT is the eight digit number attached to the filename of all you data files. If you are running ISAP in a directory which is different from the one where your data files are stored, an additional parameter has to be given; the call would then be: IA_DARK,tdt='TDT',dir='your directory'. Once the widget is up, click on detector SW3. You will see this:

Fig. 7

The red crosses are the DC measurements, while the white points are your (uncalibrated) data; everything is plotted as a function of time. If your observations was taken at a revolution number earlier than about 400 you may see DC measurements (red crosses) taken in between the observation; however only the first and the last (before and after your observation) DC measurements can actually be used.

Here the various scans of the grating can be recognized as an oscillating pattern on your data. Remember that this data is still uncalibrated at this stage and the transmission profiles (called RSRF, 1 per detector) of the instrument are still to be divided out. To check where the different scans start and end, go with the mouse on a point and click the middle button of the mouse: the text area at the bottom of the IA_DARK widget will give you all the information regarding the nearest point to the mouse actual position.  Again, it is important to remember that at this stage the instrumental transmission is not yet divided out and, if there is sufficient flux from your source, it will manifest itself as a periodic pattern throughout your dataset; it is not anything you want to model and subtract as a DC or Gain trend.

You will note that the average signal from the source is about a factor 4 higher than the DC values. In a case like this an incorrect DC estimate will not affect your data unless the DC is wrong by a factor 4 of course. It is always an advisable practice to check and possibily refine the DC
estimates also if the refinement is not going to affect your data, because the DCs are also subtracted from your illuminator flash data before estimating the Absolute Responsivity Correction Factors. An increasing trend is clearly visible in the data, but since the DCs are much lower than that, it must be a gain trend; we can correct for this in  IA_DRIFT or in ISAP.

Speaking of instrumental trends, If you have a Raster Map you will be in the uncomfortable situation to have a signal variation during the observation. In a single pointing observation, you are sure that the intrinsic signal from the source is not varying so that you can assign trends either to DC or Gain variations (and remove them). In a raster, unless you have an extended uniform brightness source, the intrinsic signal is varying; it is practically impossible to spot a trend in the instrument behaviour in this conditions.

3.2 Responsivity Drifts

The only plausible reason to still use IA_DRIFT is when there is evidence for a non-linear responsivity drift: please refer to the tutorial available at http://www.ipac.caltech.edu/iso/lws/lia/drift.html. However, if a memory effect is spotted for a detector, i.e. there is a  systematic difference between the two scan directions, it is important you remember the following:

• If you are removing the Drift with IA_DRIFT: once you have the key points and you have to choose the interpolation method to divide the drift out, always choose the 1^st or 2^nd order polynomial interpolation. Choosing the Linear Interpolation method will fit the systematic difference between scan directions as a drift with the result of corrupting the good scan direction.
• If you are removing the Drift by Shifting the Scans in ISAP: do the scan shifting for each scan direction separately.

3.3 Absolute Responsivity Correction Factors

3.4 Recalibration

4. Data Analysis with ISAP

• Zap the crap. We have to manually edit the data for each detector separately to identify and discard the clear outliers resulting from glitches. If you have a large enough data set (i.e. more than 10 scans) you might want to split apart your AAR according top the detector number. Infact after each zapping ISAP has to reload all the data structure and it may take few (say 10) seconds to do it; if you multiply these 10 seconds by the number of zappings that you will have to do this will result in an unacceptable overhead. If you find out that this is the case, it is better to SPLIT your AAR and work on smaller datasets. If in the preliminary analysis you identified detectors which showed memory effects, it is better to work on each scan direction separately. Be careful to only zap clear outliers; in the example below we can clearly recognize few glitch trails (the green, magenta and yellow scans): zap them out.

 

 

Fig, 8

Do not start zapping in the noise; you should only zap what's really systematic otherwise you will modify the statistics of the measurement and the averaged spectrum will be perturbed.

Do this zapping for all detectors; remember to STORE the AAR after finishing zapping each detector. At that moment, if you SPLITted apart your AAR, you will have to merge all the pieces back into a single AAR.

• Shift Scans. If you did not estimate and divide out the gain drift from your data with IA_DRIFT, it is time to do it now. This step has to be done for each scan direction separately for those detectors where you saw the systematic differences between scan directions. The easiest thing is to do as follows (let's assume that the zapped and merged data set is called "zapped".

• Select all data with the right mouse button and choose Shift in the toolbox; choose the Gain option and Store the resulting AAR(let's call it "all").
• Go back to the previous (zapped) data set (make it the prime data set) and select all data with Scan Direction 0 and make an AAR out of it; send it to Shift and choose the Gain option; Store the resulting AAR (let's call it "sdir0").
• Go back to the previous (zapped) data set (make it the prime data set) and select all data with Scan Direction 1 and make an AAR out of it; send it to Shift and choose the Gain option; Store the resulting AAR (let's call it "sdir1").
• Merge sdir0 with sdir1 (let's Store the result as "allsdir").

• Average. Averaging of data must be done separately for each scan direction if you have detectors with systematic dufferences between scan directions. Adopting the nomenclature of the example in previous item, you will have to use Average with data sets "all" and "allsdir". In the first run of Average you will leave the default selection for the check boxes at the top left of the widget; in the second run of Average you will de-select the Scan Direction check-box.

 

 

Here are some guidelines for averaging. If you have a limited number of scans you may want to increase the bin size to gain a bit in S/N at the expense of the spectral resolution. Consider that unless you are observing, e.g., a supernova remnant where the line can be broader than 1500 km/s, your line will be unresolved and there is nothing bad in increasing a bit the bin size to push down a bit the noise. Play also a bit with the sigma clipping threshold; a too high value will not discard enough outliers, while a too low value will start discarding too many points; there should be an optimum value which minimizes the noise level of the averaged spectrum. We cannot give more precise guidelines here because this is a function of the data. Also, the bin size and sigma values which give good results for one detector may be not optimized for another detector; if this is the case you can always send separate detectors to Average and merge the results back into a single AAR later (remember to STORE each averaged piece before you go to the next). The choice of the averaging algorithm is another parameter you can play with; the Mean and Median methods do not clip outliers and are not recommended. The Standard Clip & Mean does one single pass to discard outliers, while the Special Clip & Mean is more drastic and does iterative passes until there is nothing more to discard. Essentially play with the bin size, sigma and the averaging method until you are happy with your result; the precise combination of these three parameters will be a function of your data.

This will be the result of Average using all data together:

Fig. 9

and this is the result for the spectrum averaged keeping the scan directions separate:

Fig. 10

This should be compared with Fig. 6 . We see that we have done a good job especially on detector LW1, which is now better aligned with SW5 and LW2. In the final steps of the analysis you should use either of the two averaged data sets depending if the detector you are analysing has or not systematic differences between scan directions. For the detectors which show this difference remember to work on the scan direction which is correct for the particular wavelength range: remember the discussion about detector LW2 (above).

• Defringe. The particular data set we are using in this cookbook does not have fringing problems, meaning that the source is not extended and there are not other sources off-axis. If your spectrum is fringed, than you should defringe it before any kind of detector shift is even attempted. Defringe only detectors for which you see fringes. Do not defringe because you are not sure if there is a fringe or not and "it makes no harm anyway"; it can make harm, if you do things blindly.

• Extended Source Correction. A new functionality is available under ISAP v2.0 and later: the possibility to apply a correction to your spectrum to make the LWS calibration valid for extended sources. This is needed because the calibration files used to reduce your data were derived using a point-like source. By comparing the amount of source flux which is clipped by the focal plane aperture in the two extreme cases of point-like and infinitely extended and uniform source, the instrument team could determine a set of correction factors which, after interpolating, can be applied to your data. This is a gain correction which can reduce your LWS fluxes by as much as 40%. This correction should be applied for extended sources only.

 

 
 
 

Q. "This is all very nice. But my source is not infinitely extended and it not uniformly bright!! What should I do ?"
A. This is of course a most tricky and common situation, but unfortunately there is no definite indication; the truth will be in the middle. However, from tests done on real cases (extended galaxies, hence non uniform) it seems that the correction works well for sources more extended than 3-4 arcmin.

• And now ....? Now the fun starts!!

• Convert from flux to brightness
• Fit lines and derive integrated fluxes
• Fit combinations of blackbodies to have a first idea of the properties of the continuum spectrum
• Integrate the SED under the IRAS filters to compare with the PSC.
• Smooth the spectrum
• Rebin the spectrum to a different wavelength scale, or to the same scale of another observation
• Make arithmetic operations between two spectra
• ....