Recent changes: Sects 3.2 and 3.4.
This is your calibrated spectrum; not very informative at this stage,
but we can note that the noise of the spectrum is higher in the region
with lambda < 93um, corresponding to the SW detectors. There is nothing
wrong with this observation; infact, this is common for the LWS and
it is due to the combined effect of higher NEPs and narrower spectral
element size for the SW detectors. Let's make a couple of checks for memory
effects and detector intercalibration:
The green and the blue lines represent the two scan directions; you
can zoom and closely inspect different portions of your spectrum. For faint
sources you should not see any systematic differences among the green and
blue lines (for the same detectors); if you instead see a significative
difference (well above the noise of the green and blue lines) consider
that the effect has not been modelled, so that it is not possible to correct
for it. In case a line is present it is strongly advisable to keep the
two scan directions separate and estimate the line parameters separately;
the two line fluxes can be averaged later, but the two determinations will
give you the best estimate of the uncertainty to assign to the average
line flux. This problem will be analysed in more detail when we will discuss
the data reduction of a medium-flux source (http://www.ipac.caltech.edu/iso/lws/lia/lws01_medium.html).
In principle, the spectra of the 10 detectors should line up nicely;
the overlapping portions of the spectra should match both in shape and
in absolute value, but this generally does not happens. What are the possible
reasons for a mismatch ?
Now you should quantify a bit this mismatch. Generally speaking you might be happy with a 10-15% mismatch, but this of course depends on the source. If the source is pointlike, you should expect less mismatch because (see above) the primary calibrator for the LWS is also pointlike and the beam is stable at all wavelengths. If your source is extended or there another source close enough (<80" for a 300 Jy source) to induce fringing on your spectrum, then the situation is more critical because the LWS beam is modulated and its FWHM may vary of about +/- 10% within each detector band (the amplitude of the effect increases with wavelength). Examine your spectrum and take notes about which detectors are more critical in the aspects we have just discussed.
There are 4 steps in the LIA reprocessing of an LWS01 AOT:
While at the ISAP> prompt, type: IA_DARK, tdt='TDT', where TDT is the eight digit number attached to the filename of all you data files. If you are running ISAP in a directory which is different from the one where your data files are stored, an additional parameter has to be given; the call would then be: IA_DARK,tdt='TDT',dir='your directory'. Once the widget is up, click on detector SW1. You will see this:
The red crosses are the DC measurements, while the white points are your (uncalibrated) data; everything is plotted as a function of time. If your observations was taken at a revolution number earlier than about 400 you may see DC measurements (red crosses) taken in between the observation; however only the first and the last (before and after your observation) DC measurements can actually be used.
Here the various scans of the grating can be recognized as an oscillating pattern on your data. Remember that this data is still uncalibrated at this stage and the transmission profiles (called RSRF, 1 per detector) of the instrument are still to be divided out. To check where the different scans start and end, go with the mouse on a point and click the middle button of the mouse: the text area at the bottom of the IA_DARK widget will give you all the information regarding the nearest point to the mouse actual position. Again, it is important to remember that at this stage the instrumental transmission is not yet divided out and, if there is sufficient flux from your source, it will manifest itself as a periodic pattern throughout your dataset; it is not anything you want to model and subtract as a DC or Gain trend. For a moment you think that all the spikes that you see on your data are lines, but your illusion only lasts few tenths of seconds. Zoom wherever you want on the above plot (on your LIA widget, of course) and you will find out that most of them are 'glitches' resulting from the detector being hit by an energetic particle, each followed by its decaying trend marking the recovery to normal conditions. Most of the glitches are located on the trailing edge of a previous glitch, so you may wonder what 'normal condition' means for this detector. Let's make a note that we will have to do a lot of zapping later in ISAP. The important thing to notice here is that glitches are present also during the dark current measurements; if you bring up the first DC measurement you will see this:
You will note the couple of glitch trails at the end of the measurement. The median-clipped averaged performed on the whole set of points above gives the DC estimated made by the OLP, which is visible in the box 'OLP Used...' for 'Dark 0' in Fig. 6; it is clear however that this is an overestimate because the second part of the measurement is clearly corrupted. Also the first group of 7 points in the plot above looks like a recovery from a glitch happened prior to the start of the observation. It seems that the best estimate we can do of the dark current should be based on the central group of point in Fig. 6. Now go ahead and make your DC estimate also for the second DC measurement.
When you come to the subtraction of your estimated DCs from the data, a lot of common sense is needed. Take the example in Fig. 7.
The only thing you want to try here is to remove trend in DC during your observations, and so you may want to try and interpolate following the lower envelope of your data.
If you have a Raster Map you will be in the uncomfortable situation to have a signal variation during the observation. In a single pointing observation, you are sure that the intrinsic signal from the source is not varying so that you can assign trends either to DC or Gain variations (and remove them). In a raster, unless you have an extended uniform brightness source, the intrinsic signal is varying; it is practically impossible to spot a trend in the instrument behaviour in this conditions.
You should by now be convinced that the variability you see in the signal
(white points) in Fig. 7 is just the relative spectral
response function of the detector which has not yet been divided out; do
not try to model this stuff, as it will disappear when we will recalibrate
our spectrum later. The best (and most conservative) thing you want to
do at this stage is to assume that the DC is constant through out the observations.
In general, if the DC measurements done before and after the observation
are comparable, within the noise of the single measurements, we recommend
you choose the linear interpolation option which uses the average of the
two measurements. This is the option applied by the OLP, and unless the
two measurements are significantly different ot there is a clear DC trend
in your data, you should stick with that option. The case shown in Fig.
7 is very interesting because it represents an exception to this rule.
Assuming a constant DC equal to the average of the two DC measurements
and interpolating this value throughout the observation, we obtain the
upper full line in the above figure. Now it is clear that there is something
wrong with this, because after the DC subtraction more than 1/3 of the
data points would be negative. The fact itself that we see the spectral
profile of the detector during the grating scans is reassuring that some
flux above the DC level must be falling on the detector; the true DC value
must then be lower than suggested by the DC estimates we made above (see
6). We are therefore in the unpleasant situation where we do not
have a reliable independent estimate of the DC; we can only guess that,
based on the considerations above (in italic), the DC must be less or equal
to the minimum value of the data (white points); it cannot be higher than
that because, again, the transmission profile of the detector is clearly
seen and this imply that the detector is seeing flux from the source.
Tip: force a value of DC similar to the minimum value (on average) reached by your data, by typing this value in the two Dark Measures boxes visible in Fig. 6 and hit return; then interpolate the DC assuming it constant and equal to this value.
Write down that you gave your best guess for the DC of this detector, since a further adjustment might be needed when you will look at your recalibrated spectrum.
Another peculiar situation we find in the current example is found for detector LW4:
Apart from the periodic pattern due to the transmission profiles, we
do see a decreasing temporal trend/drift in the data. Is this a DC or a
responsivity trend ? It is a tough question and you might not always be
in the condition to answer. In this case, however, we have a way out.
Tip: check the peak-to-peak amplitude of each RSRF pattern. Is this decreasing with time ? If YES, this is likely to be a responsivity (gain) trend; if NO, this will be a DC trend.
Keep in mind the considerations we have been doing so far, while proceeding with the DC estimate and subtraction for the rest of the detectors.
The only plausible reason to still use IA_DRIFT is when there is evidence for a non-linear responsivity drift: please refer to the tutorial available at http://www.ipac.caltech.edu/iso/lws/lia/drift.html.
We have just one special recommendation for you, when you are dealing with faint sources. Be careful that the drift you will fit to your data does not cross 0. If this happens, a portion of your data will be divided by very small numbers and the calibrated spectrum will contain points with enormous flux values (both positive and negative). When the source is faint, put a lot of care in selecting the highest signal portions of your grating scans to estimate the responsivity drift. When fitting trends in an observation which is a raster map, remember what we said above. Please note that this problem will affect OLP Version 8 data.
Do not start zapping in the noise; you should only zap what's really systematic otherwise you will modify the statistics of the measurement and the averaged spectrum will be perturbed.
Do this zapping for all detectors; remember to STORE the AAR after finishing zapping each detector. If you SPLITted apart your AAR, it time now to merge all the pieces back into a single AAR.
Here are some guidelines for averaging. If you have a limited number of scans you may want to increase the bin size to gain a bit in S/N at the expense of the spectral resolution. Consider that unless you are observing, e.g., a supernova remnant where the line can be broader than 1500 km/s, your line will be unresolved and there is nothing band in increasing a bit the bin size to push down a bit the noise. Play also a bit with the sigma clipping threshold; a too high value will not discard enough outliers, while a too low value will start discarding too many points; there should be an optimum value which minimizes the noise level of the averaged spectrum. We cannot give more precise guidelines here because this is a function of the data. Also, the bin size and sigma values which give good results for one detector may be not optimized for another detector; if this is the case you can always send separate detectors to Average and merge the results back into a single AAR later (remember to STORE each averaged piece before you go to the next). The averaging algorithm is another parameter you can play with; the Mean and Median methods do not clip outliers and are not recommended. The Standard Clip & Mean does one single pass to discard outliers, while the Special Clip & Mean is more drastic and does iterative passes until there is nothing more to discard. Essentially play with the bin size, sigma and the averaging method until you are happy with your result; the precise combination of these three parameters will be a function of your data.
As is the case in general for faint sources, memory effects were not present in the observation we are using as an example in this cookbook. The averaged spectrum, coloured by detector number, will be something like this:
This should be compared with Fig. 4 (plotted on the same Y scale). We see that we have done a good job especially on detector SW1 (the first on the left), which is now aligned with SW2. The situation got a little bit worse for LW4; can we do something about it ? Sure ! You know that for LW4 you did your best guess with the absolute DC values (There are no doubts for the removal of the decreasing trend that was present for the DCs - see above); well, this guess was probably not correct and we have a way to correct for this in ISAP. In the example above, I will plot the last three detectors only, to see in detail how LW4 compares with the adjacent LW3 and LW5. This is a blow-up of the situation:
We select the three detectors and we choose the option to Shift
them; use indifferently LW3 or LW5 as the Good
choose the LWS Dark Correction shifting method. This method puts back the detector transmission profiles onto your spectra, applies the proper offset to the detector that must be shifted, and divides the result by the transmission profiles. The result is shown in Fig. 12
The result is very good and this is confirmed by the fact that the height of the [CII] line is the same in both detectors. Had we Shifted LW4 with any of the other two methods the line would have looked different in the two detectors, casting severe doubts on the procedure. A couple of question may arise:
Q. "Why should I lose my time with the detailed
inspection of DCs done before in LIA, if I can always adjust the spectra
in this way in ISAP ?"
A. We could apply this method of shifting LW4 with the Dark LWS Correction option, because we carefully checked the DCs in LIA. You will remember that LW4 was a detector for which the DCs were given as a best guess (see above), so we were authorized to make this correction afterwards. If you do not do the IA_DARK step, who is going to tell you if a mismatch is due to a DC problem or a responsivity (gain) problem ?
Q. "In the previous case I might have tried all
three Shifting methods and accept that for which the line height in the
two detectors comes out to be the same; why bothering with all the rest
A. In principle is true; but what's going to happen when there is no line to help you in the overlapping region between two adjacent detectors ?
Q. "This is all very nice. But my source is not
infinitely extended and it not uniformly bright!! What should I do ?"
A. This is of course a most tricky and common situation, but unfortunately there is no definite indication; the truth will be in the middle. However, from tests done on real cases (extended galaxies, hence non uniform) it seems that the correction works well for sources more extended than 3-4 arcmin.