Data Reduction Recipe for an LWS01 AOT.
The case of a medium brightness source.
0. About this recipe...
What this recipe is:
What this recipe is not:
A list of checks to be done and procedures to be followed during reduction
& analysis of your LWS01 observation.
A place to look for tips about how to deal with the aspects of the data
reduction which are critical for a medium-brightness source.
This recipe is a worked example of real LWS data reduction and analysis;
it covers many of the peculiar instrumental effects that affect the quality
of your data, but certainly not all. In these cases we encourage you to
check the LWS and LIA FAQs; should your
question remain unanswered, please contact us at email@example.com
if in the U.S, or firstname.lastname@example.org
if in Europe.
A tutorial about the use of ISAP
An exhaustive description of ISAP and LIA functionality (refer to the tutorial
A detailed description of the ISAP and LIA Graphical User Interfaces
Recent changes: Sects 3.2 and
1. Definitions & Requirements
A medium-brightness source, for the purpose of this document, has a flux
higher than 100 Jy but lower than 1000 Jy around 100um.
This recipe assumes you have a basic knowledge of the LWS instrument and
its main sub-systems: detectors, grating, illuminators.
This recipe assumes you are already familiar with ISAP's and LIA's GUIs,
as well as with the main functionality available in those packages.
This recipe assumes the following packages are available to you:
ISAP Version 1.6 or later. An earlier ISAP version does not have all the
functionality needed to follow all the steps and procedures described in
LIA Version 7.2 or later
This recipe assumes the following files pertaining to your ISO LWS observation
are available to you, and have been obtained with Version 7 of the ESA
automatic data reduction pipeline (OLP):
LSPD - Raw science data at an intermediate level of reduction; this will
be the starting point for our data reduction
LIPD - Calibration data obtained towards internal
calibration sources during your observation
LIAC - File holding the Dark
Current estimates performed by the ESA automatic data reduction pipeline
LGIF - File holding the absolute responsivity
correction factors estimated by the ESA automatic data reduction pipeline
LSAN - Calibrated science data produced by the ESA automatic data reduction
(IIPH - Pointing history file, contains information about the instantaneous
pointing of the satellite during your observation)
2. Inspecting your Data
A preliminary inspection of your data is useful to get a feeling of what
you have in your hands, and to spot potential problems you will have to
take care of during the data reduction. Let's start by entering ISAP:
Change directory where the files containing the data of your observations
Enter ISAP by typing the command isap
at the prompt
Load the LSAN file into ISAP
Select all Scans, Detectors, and the two scan directions, and plot the
data. Likely, you will see something like this:
This is your calibrated spectrum; not very informative at this stage,
but we can note that the noise of the spectrum is higher in the region
with lambda < 93um, corresponding to the SW detectors. There is nothing
wrong with this observation; infact, this is common for the LWS and
it is due to the combined effect of higher NEPs and narrower spectral
element size for the SW detectors. Let's make a couple of checks for memory
effects and detector intercalibration:
Memory effects. There are two different
types of memory effects:
Effects caused by energetic particles hitting the detectors. This event
causes an abrupt modification of the detector responsivity resulting in
a jump of the signal output of the detector ('glitch') followed by a more
or less slow recovery to normal behaviour. The recovery time is unpredictable;
we have seen cases of almost instantaneous recovery, and cases where the
detector took hours to go back to normal behaviour. Here is an example
where 5 'glitches' follow one another during a grating scan (here the grating
is scanning towards lower wavelengths):
We tried to model, among other things, the recovery time as a function
of the signal jump, but without success; the first three points of each
jump have actually been discarded by the 'deglitching' algorithm present
in the first stage (ERD ===> SPD) of the automatic data reduction pipeline
and they do not appear in Fig. 2. The only possibility
is then to zap the crap; later, in ISAP, we will have to discard the trailing
edge of each glitch.
caused by the average amount of flux falling onto the detector. There is
a certain flux range for each detector (e.g. between 300 and 1000 Jy for
detector LW2) where the response time of the detectors is a strong function
of the temporal gradient of the signal; in other words the observed spectrum
will be different for the two grating scan directions. For medium-brightness
sources we expect this effect to be present, especially in detectors LW2
and LW3. To visualize this effect we have to average the data; for this
particular purpose we can use the default averaging options and use Special
Clip & Mean as the averaging method. When you are back in
the main ISAP widget, change the Plot Style to Line
Style: Connected and Colour Coding
Preference: Scan Direction and plot. You should see something
The green and the blue lines represent the two scan directions; you
can zoom and closely inspect different portions of your spectrum. Without
even zooming portions of the above plot, we can already note dramatic differences
among scan directions for detectors SW1, SW2 and LW2; a closer inspection
show we have similar problems on SW3 and LW3.
Why these problems ? The response time
of the detectors is different depending if the signal increases or decreases
with time; in particular, detectors are faster to respond to decreasing
I have deliberately used the term signal and not the term flux because
what the detector sees is the source flux convoluted with its Relative
Spectral Response Function (RSRF). Since intrinsic source spectra are in
general shallow (not considering lines and spectral features in general)
over the wavelength range covered by each detector, the RSRF is the function
that tells you when the detectors is experiencing an increasing or a decreasing
signal. Now, let's examine in more detail the situation of detector LW2:
N.B. the green is for Scan Direction
=0, where the grating scans from higher to lower wavelengths.
And now let's see what is the RSRF for this detector:
The turnover of the RSRF is around 120 microns. Since the green scans
are done towards decreasing wavelengths, the part of a green
scan from 133 to 120 microns will experience an increasing signal and the
detector will be "slow" in responding; the recorder signal will therefore
be lower than the real signal, because the detector has not enough time
to adjust its response to the increased signal before the grating moves
again. Below 120 microns the green scans will be ok. On the other hands
the blue scans start from 102 microns towards higher wavelengths, and the
detector will experience increasing incoming signals up to 120 microns;
for this range the detector output will lead to an underestimate of the
source flux. Summarizing: trust the green scans longward of 120 microns,
and the blue scans shortward of 120 microns.
You should make similar considerations for all detectors for which
you see this problem occuring in Fig. 3. The RSRF
for all detectors can be found at http://www.ipac.caltech.edu/iso/lws/lia/glossary.html#Relative
Spectral Response Function (RSRF).
How do I proceed in the analysis ? When you will have completed
you LIA reduction (see below) and you will start the analysis, remember
not to average across scan direction for the detectors where the above
problems are spotted.
Detector Intercalibration. Now change the plot style to Colour
Coding Preference: Detector number and plot; you will now see something
The spectra of the 10 detectors should line up nicely; the overlapping
portions of the spectra should match both in shape and in absolute value
(once the correct scan direction is picked for that particular detector),
but this generally does not happens. What are the possible reasons for
a mismatch ?
Incorrect Dark Current subtraction. This should not be an issue for medium
brightness sources; it is anyway advisable to check the DCs.
Incorrect estimate of the Absolute Responsivity Correction factors.
LWS flux calibration (both absolute and relative) is based on a point-like
source (a planet). If the source is extended, we may expect detector mismatches
depending on the extension as a function of wavelength (e.g., the source
may be more extended at short wavelengths than at long wavelengths). A
tool to apply a correction for the source extension (assuming extension
larger than the beam and uniform brightness distribution) will be available
in ISAP Version 2.0.
Now you should quantify a bit this mismatch. Generally speaking
you might be happy with a 10-15% mismatch, but this of course depends on
the source. If the source is pointlike, you should expect less mismatch
above) the primary calibrator for the LWS is also pointlike and the
beam is stable at all wavelengths. If your source is extended or there
another source close enough (<80" for a 300 Jy source) to induce
fringing on your spectrum (which is not the case for the example in Fig.
6), then the situation is more critical because the LWS beam is modulated
and its FWHM may vary of about +/- 10% within each detector band (the amplitude
of the effect increases with wavelength). Examine your spectrum and take
notes about which detectors are more critical in the aspects we have just
3. Reprocessing your
Data with LIA
Let's start this Section with a recommendation: it is always advisable
to have a go with LIA, also if the preliminary analysis with ISAP did not
show any particular problem. It is good to have a look at the dark
current and at the raw data to check that everything is all right; a trend
of decreasing dark current, e.g., may reflect in an increased scatter of
your grating scans (i.e. an increased noise of your averaged spectrum).
Besides, the comparison of your data with the Dark Currents will tell you
if you are detecting signal or you just reached the sensitivity limit of
There are 4 steps in the LIA reprocessing of an LWS01 AOT:
3.1 Dark Currents
The tool to be used is IA_DARK. A full tutorial describing its functionality
is accessible at http://www.ipac.caltech.edu/iso/lws/lia/dark.html;
in the following, it is assumed that you are familiar with that document
(or at least have it at hand).
While at the ISAP> prompt, type:
tdt='TDT', where TDT is the eight digit number attached to the filename
of all you data files. If you are running ISAP in a directory which is
different from the one where your data files are stored, an additional
parameter has to be given; the call would then be: IA_DARK,tdt='TDT',dir='your
directory'. Once the widget is up, click on detector SW3. You will
The red crosses are the DC measurements, while the white points are
your (uncalibrated) data; everything is plotted as a function of time.
If your observations was taken at a revolution number earlier than about
400 you may see DC measurements (red crosses) taken in between the observation;
however only the first and the last (before and after your observation)
DC measurements can actually be used.
Here the various scans of the grating can be recognized as an oscillating
pattern on your data. Remember that this data is still uncalibrated at
this stage and the transmission
profiles (called RSRF, 1 per detector) of the instrument are still
to be divided out. To check where the different scans start and end, go
with the mouse on a point and click the middle button of the mouse: the
text area at the bottom of the IA_DARK widget will give you all the information
regarding the nearest point to the mouse actual position. Again,
it is important to remember that at this stage the instrumental transmission
is not yet divided out and, if there is sufficient flux from your source,
it will manifest itself as a periodic pattern throughout your dataset;
it is not anything you want to model and subtract
as a DC or Gain trend.
You will note that the average signal from the source is about a factor
4 higher than the DC values. In a case like this an incorrect DC estimate
will not affect your data unless the DC is wrong by a factor 4 of course.
It is always an advisable practice to check and possibily refine the DC
estimates also if the refinement is not going to affect your data,
because the DCs are also subtracted from your illuminator flash data before
estimating the Absolute Responsivity Correction Factors. An increasing
trend is clearly visible in the data, but since the DCs are much lower
than that, it must be a gain trend; we can correct for this in IA_DRIFT
or in ISAP.
Speaking of instrumental trends, If
you have a Raster Map you will be in the uncomfortable situation
to have a signal variation during the observation. In a single pointing
observation, you are sure that the intrinsic signal from the source is
not varying so that you can assign trends either to DC or Gain variations
(and remove them). In a raster, unless you have an extended uniform brightness
source, the intrinsic signal is varying; it is practically impossible to
spot a trend in the instrument behaviour in this conditions.
3.2 Responsivity Drifts
With the possibility of `shifting spectral scans' which will be available
with ISAP version 2.0, the routine IA_DRIFT, used to correct for temporal
responsivity drifts, is likely to become obsolete. This routine was needed
because the pipeline incorrectly estimate the responsivity drift before
subtracting the DC; it will be much easier and less time consuming to do
this with the new ISAP functionality.
The only plausible reason to still use IA_DRIFT is when there is evidence
for a non-linear responsivity drift: please refer to the tutorial available
However, if a memory effect is spotted for a detector, i.e. there is a
systematic difference between the two scan directions, it is important
you remember the following:
If you have a raster map, remeber what we said above,
before trying to estimate and remove a suspect gain trend.
If you are removing the Drift with IA_DRIFT: once you have the key
points and you have to choose the interpolation method to divide
the drift out, always choose the 1st or 2nd order
polynomial interpolation. Choosing the Linear Interpolation method will
fit the systematic difference between scan directions as a drift with the
result of corrupting the good scan direction.
If you are removing the Drift by Shifting the Scans in ISAP: do
the scan shifting for each scan direction separately.
Responsivity Correction Factors
The absolute responsivity correction factors
can revised using the routine IA_ABSCORR, whose tutorial is available at
Since the estimate of these factors is based on the Illuminator Flash data,
the characteristics of the source observed are not important; the tutorial
has all the information you need to proceed.
The recalibration of your data is independent on the type of source observed.
Instructions to recalibrate your data are found in http://www.ipac.caltech.edu/iso/lws/lia/lia.html#SHORT_AAL.
If you did not use IA_DRIFT to remove the relative responsivity drifts,
remember to run SHORT_ALL with the `/nodrift` option.
4. Data Analysis with ISAP
At this time, you should have in your directory a file called LSANxxxxxxxx_SHORT.FITS
produced by SHORT_AAL; load this file into ISAP. You can do a straight
average and compare the result with the averaged spectrum for the original
LSAN file produced by the OLP. Once you are done, we can start with the
real work to analyse this data.
Zap the crap. We have to manually edit the data for each detector
separately to identify and discard the clear outliers resulting from glitches.
If you have a large enough data set (i.e. more than 10 scans) you might
want to split apart your AAR according top the detector number. Infact
after each zapping ISAP has to reload all the data structure and it may
take few (say 10) seconds to do it; if you multiply these 10 seconds by
the number of zappings that you will have to do this will result in an
unacceptable overhead. If you find out that this is the case, it is better
to SPLIT your AAR and work on smaller datasets. If in the preliminary analysis
you identified detectors which showed memory
effects, it is better to work on each scan direction separately. Be
careful to only zap clear outliers; in the example below we can clearly
recognize few glitch trails (the green, magenta and yellow scans): zap
Do not start zapping in the noise; you should only zap what's really
systematic otherwise you will modify the statistics of the measurement
and the averaged spectrum will be perturbed.
Do this zapping for all detectors; remember to STORE the AAR after
finishing zapping each detector. At that moment, if you SPLITted apart
your AAR, you will have to merge all the pieces back into a single AAR.
Shift Scans. If you did not estimate and divide out the gain drift
from your data with IA_DRIFT, it is time to do it now. This step has to
be done for each scan direction separately for those detectors where you
saw the systematic differences between scan directions. The easiest thing
is to do as follows (let's assume that the zapped and merged data set is
Select all data with the right mouse button and choose Shift
in the toolbox; choose the Gain
option and Store the resulting
AAR(let's call it "all").
Go back to the previous (zapped) data set (make it the prime data
set) and select all data with Scan Direction
0 and make an AAR out of it; send it to Shift
and choose the Gain option; Store
the resulting AAR (let's call it "sdir0").
Go back to the previous (zapped) data set (make it the prime data
set) and select all data with Scan Direction
1 and make an AAR out of it; send it to Shift
and choose the Gain option; Store
the resulting AAR (let's call it "sdir1").
Merge sdir0 with sdir1 (let's Store
the result as "allsdir").
Average. Averaging of data must be done separately for each scan
direction if you have detectors with systematic dufferences between scan
directions. Adopting the nomenclature of the example in previous item,
you will have to use Average with
data sets "all" and "allsdir". In the first run of Average
you will leave the default selection for the check boxes at the top left
of the widget; in the second run of Average
you will de-select the Scan Direction
Here are some guidelines for averaging. If you have a limited number
of scans you may want to increase the bin size to gain a bit in S/N at
the expense of the spectral resolution. Consider that unless you are observing,
e.g., a supernova remnant where the line can be broader than 1500 km/s,
your line will be unresolved and there is nothing bad in increasing a bit
the bin size to push down a bit the noise. Play also a bit with the
clipping threshold; a too high value will not discard enough outliers,
while a too low value will start discarding too many points; there should
be an optimum value which minimizes the noise level of the averaged spectrum.
We cannot give more precise guidelines here because this is a function
of the data. Also, the bin size and sigma values which give good results
for one detector may be not optimized for another detector; if this is
the case you can always send separate detectors to Average
and merge the results back into a single AAR later (remember to STORE
each averaged piece before you go to the next). The choice of the averaging
algorithm is another parameter you can play with; the Mean
and Median methods do not clip
outliers and are not recommended. The Standard
Clip & Mean does one single pass to discard outliers, while
the Special Clip & Mean is
more drastic and does iterative passes until there is nothing more to discard.
Essentially play with the bin size, sigma and the averaging method until
you are happy with your result; the precise combination of these three
parameters will be a function of your data.
This will be the result of Average using all data together:
and this is the result for the spectrum averaged keeping the scan directions
This should be compared with Fig. 6 . We see that
we have done a good job especially on detector LW1, which is now better
aligned with SW5 and LW2. In the final steps of the analysis you should
use either of the two averaged data sets depending if the detector you
are analysing has or not systematic differences between scan directions.
the detectors which show this difference remember to work on the scan direction
which is correct for the particular wavelength range: remember the discussion
about detector LW2 (above).
Defringe. The particular data set we are using in this cookbook
does not have fringing problems, meaning that the source is not extended
and there are not other sources off-axis. If your spectrum is fringed,
than you should defringe it before any kind of detector shift is even attempted.
Defringe only detectors for which you see fringes. Do not defringe because
you are not sure if there is a fringe or not and "it makes no harm anyway";
it can make harm, if you do things blindly.
Extended Source Correction. A new functionality is available under
ISAP v2.0 and later: the possibility to apply a correction to your spectrum
to make the LWS calibration valid for extended sources. This is needed
because the calibration files used to reduce your data were derived using
a point-like source. By comparing the amount of source flux which is clipped
by the focal plane aperture in the two extreme cases of point-like and
infinitely extended and uniform source, the instrument team could determine
a set of correction factors which, after interpolating, can be applied
to your data. This is a gain correction which can reduce your LWS fluxes
by as much as 40%. This correction should be applied for extended sources
Q. "This is all very nice. But my source is not
infinitely extended and it not uniformly bright!! What should I do ?"
A. This is of course a most tricky and common
situation, but unfortunately there is no definite indication; the truth
will be in the middle. However, from tests done on real cases (extended
galaxies, hence non uniform) it seems that the correction works well for
sources more extended than 3-4 arcmin.
And now ....? Now the fun starts!!