IMAP - An IPAC Tool for PHT32 Data Reduction
The IMAP (IPAC MAP package, imap.pro) is
a visual and interactive tool that allows the user to inspect, clean, and
perform empirical corrections for a number of instrumental effects (e.g.,
vignetting, long-term drift, flatfield) on the PHT32 observation data.
Bright sources (or "negative dips") in the map can also be masked out temporarily
for improved fits of these instrumental effects in the data. The
IMAP takes a SPD level data from the PIA (ISOPHOT
and generates a new one with its corrections applied. There are two
ways to access this tool. It is implemented into the PIA, so that
you can run it by selecting the appropriate option, "Launch IMAP P32 processing"
on the pull-down menu for the "Process" button on the PIA SPD Data Display
window. The other option is to run it outside the PIA, but this requires
you to save the SPD data into a file and feed the output back into the
PIA for the final map construction.
To use the IMAP implemented into the PIA, all you need to do is to click
on the option, "Launch IMAP P32 processing" on the pull-down menu for the
"Process" button on the PIA SPD Date Display window and follow instructions
at the prompt. Otherwise, you need to follow the steps outlined below
when using the imap.pro in a separate IDL session.
Using the imap.pro
1. PIA --> SPD level
The P32 data needs to be saved by the PIA as an internal SPD
file at the SPD level . This file ends with ".ISPD".
2. Feed the SPD result into IMAP
Start an IDL session from the directory which contains the
imap.pro file and the saved SPD file. Then type the following at
the IDL prompt. You should have imap.pro saved
to a file in advance.
3. PIA --> Astrophysical Work --> map construction
IDL> .r imap.pro
Then follow the comments and questions at the IDL prompt.
The imap.pro displays several images at appropriate times to aid the
user visually. In addition, five of these displays, one from each
of the major sections of program, are saved to imap_result.ps file,
so that the user can produce hardcopies easily.
In the end, the user has the option of saving the imap.pro results to
a new internal SPD file or exit without saving any changes. The new
file has ".ISPD_IMAP" ending to distinguish it from the original
Now you can load this new file into the PIA and construct the
map as usual. Please note that to read this file into the PIA properly,
you need to delete the _IMAP ending. Refer to the PIA User Manual
for questions regarding the map construction. Also, you should disable
the Flat Fielding with the PIA mapping because IMAP already applied skyflattening.
The following is the description of each step of the IMAP program.
The green and blue capital letters are used to represent the program labels
and to clearly show where the program moves to next. The DISPLAY
signals a new IDL window display and gives the description of what gets
depicted visually. Click on the DISPLAY
to view an example.
UP DATA IN IDL
Restores the PIA SPD file and extracts headers and data arrays.
Prints out the following information on the screen :
name of the object as defined in the header
name of the PI
wavelength of the filter used in the measurement
size of the map
step sizes in both y and z directions
Assigns relative (Y,Z) coordinates to each pixel
at each chopper plateau.
The position of each pixel at each chopper plateau
is defined by the following arrays.
phtispd.det contains the detector information (C100
phtispd.cpos contains the chopper positions (ranging
from -90 to +90 degrees)
phtispd.rpid contains the raster point IDs
(Please refer to the Appendix B: Internal PIA
Buffer Structure Definitions of the PIA User Manual.)
Determines Y and Z coordinates of the detector center
at each chopper plateau (i.e., the center of the central pixel if it is
the C100 detector, and in the case of the C200 detector, the mid-point
where the gaps between the pixels cross). These are defined as (y0,z0)
in the program. Note that y0 and z0 are arrays with their lengths
equal to the number of chopper plateaus, n.
From the (y0, z0) arrays, the Y and Z coordinates
of each pixel center, (yout, zout), are calculated from the known detector
dimensions. The pixel arrangement for the C100 detector is
shown below. The center of the detector coincides exactly with the
center of the pixel #5.
The pixel arrangement for the C200 detector is as follows.
For both diagrams above, the +Y is to
the right and +Z is upwards.
Displays the individual pixel centers with yellow dots, and the detector
centers at each chopper plateau with green plus signs. Above the
plot, the name of the object, TDT number, date of observation, and wavelength
of the filter used are given. The gray square boxes represent
the individual detector pixels for the top left most chopper plateau.
These boxes are correctly centered at the pixel centers. However,
these are NOT drawn to scale, and the aspect ratio of the display is not
correct; this is simply to aid the user with the visualization.
OUT BRIGHT SOURCES
Divides the map into a user-defined number of cells
for masking out deviated points.
First, prints out the size of the region cover by the map (specifically
by the pixel centers, the yellow dots in the display) in arcsecs, then
asks the user to input values for a cell size in arcsecs and for a threshold
in units of r.m.s. noise for sigma clipping. Examples are given to
guide the user.
After the number of cells and their locations are calculated, the result
is plotted. As before, the yellow dots represent the pixel centers,
and the green plus signs show the chopper plateaus. Now, the red
asterisks represent the centers of the cells. The cells should not
be too closely spaced.
The data points are grouped into different cells.
For each given cell, calculates the mean and sigma
of the phtispd.mnpw (mean in-band power per chopper plateau) values of
the data points grouped into that cell and whose corresponding phtispd.flag
values are zero. In phtispd.flag array, a value of 0 means the signal
is valid, 1 means the uncertainty associated with the signal is not valid,
and 2 mean the signal is not valid. Then, all the signals that deviate
from their corresponding cell mean by more than the sigma times the user-defined
threshold are flagged (by setting their FLAG values equal to 3).
For each cell, the following parameters are printed
out on the screen. The user may need to make the window wider to
avoid the output spilling over to the next line. The (Y-cell, Z-cell)
are index numbers used to designate each cell whose coordinates are given
by (Y-center, Z-center). For each cell, the number of valid data
points, their mean and sigma, and the number of points flagged are printed.
||number of valid data points
||number of points flagged
Now asks the user if he/she would like to ignore
this result and try using different cell size and/or different threshold
for the sigma-clipping. This is a yes/no question. The user
should make the decision based on the displayed cell positions and the
above table - the number of valid points per cell should not be too small.
If the user answers yes, then all the FLAG = 3 will be reset back to 0
and the program goes back to the top of this section marked with A.
If the answer is no, then the program moves to the next section keeping
all the results.
Masks out bright point sources using a sigma-based
threshold and a growing radius. This is to pick out the background
region to be used for further corrections.
Using only the data points with FLAG = 0, the MEAN
power is recalculated for each cell. Then, the mean and sigma of
these MEAN powers are calculated. These are called MEANMEAN and MEANSIGMA
respectively in the program.
At the command line, the user is asked to input a threshold such that any
cell whose mean signal deviates by more than this number of sigmas will
be masked out (e.g. 2.5). Next, it asks the user to input a value
for the growth radius using the units of the cell size. Any points
found within this radius around the masked out cells (if any) will also
be masked out. If this instruction is not clear, the user is encouraged
to try different values and study the changes in the display result.
An IDL display window designed to fit four plots appears. The two
plots on the left are surface plots of the mean values of all the cells,
and each of the two plots on the right shows the view of the corresponding
surface plot to its left as seen from the point directly above it with
overplots of contours and dots which represent the centers of each pixel
at each chopper plateau. The X in the lower right plot will show
all the flagged points. If there is no X present in the lower right
plot, then the two surface plots on the left are exactly the same except
for the scale along the z-direction. This display is saved into a
postscript file named imap_result.ps.
All cells whose mean is above the mean of all cell
means by more than the threshold times the sigma will be masked out, and
all points found within the radius of these cell centers with FLAG = 0
will also be masked out by having their FLAG changed to 4. The total
number of cells and the number of masked out cells are printed on the screen,
and the result completes the current IDL window display.
Now the user is asked if he/she would like to ignore
this result and try using different threshold and/or growing radius.
Again, this is a yes/no question, where no means the program moves to the
next section saving all the work done so far. If the answer is yes,
then all the FLAG = 4 gets reset to 0 and the program moves back to the
of the program to start this section over again.
Statistically determines the chopper vignetting.
First, defines a new two-dimensional array CPVIGN
whose first index has size equal to the total number of pixels (9 for C100
and 4 for C200) and the second has size equal to the total number of different
chopper positions. Hence, in the case of C200, the 1st index had
4 elements and the 2nd index has 7 elements. In the case of C100,
the 1st index has 9 elements and the 2nd has 13 elements.
Using only those with FLAG = 0, the program groups
the data points according to their pixel and chopper position. The
phtispd.cstp array which contains the chopper step information (contains
repeating 1-13 for C100 and 1-7 for C200) is used in a loop over pixels.
An array CPV is used to store all data points according to different pixels
and chopper positions. The CPVIGN is simply the median of CPV.
That is, CPVIGN is the median of all data points for a given pixel and
a given chopper position.
For each pixel, the normalized vignetting factors
are found by dividing all of the CPVIGN elements by its middle element
which corresponds to the chopper throw or position of 0 degree.
Now these vignetting factors for each pixel are displayed
along with their polynomial fits. After each pixel display, the user
is asked to select which fit to use for correction.
For each pixel, its CPVIGN is fitted with polynomials of order 2, 3 and
4 which are plotted as yellow solid line, green dotted line, and red dashed
line respectively. Please refer to the following setup for the pixel
arrangement on the IDL display window.
| pixel #1
|| pixel #2
|| pixel #3
| pixel #4
|| pixel #5
|| pixel #6
C100 pixel arrangement
| pixel #7
|| pixel #8
|| pixel #9
| pixel #1
|| pixel #2
C200 pixel arrangement
| pixel #3
|| pixel #4
Asks the user if he/she would like to apply these
selected fittings to the data. If yes, then the the correction is
applied to all data points irrespective of their FLAG values by dividing
each signal by the corresponding CPVIGN element (i.e., the same pixel and
the chopper position). If no, the vignetting correction is skipped.
Corrects for the long-term drift using a moving median
Uses a moving median method to calculate the average drift of each pixel.
First, it prints out the length of the time series N and the suggested
width for the moving median DELTA which is equal to 2/3 the length of the
time series. DELTA is the length over which the median is taken.
Next, it asks the user to enter his/her choice of the width for median.
The value should not be too small.
Using only the data points with FLAG = 0, it calculates
the moving median for each pixel P and calls the result MEDIAN(P,N).
To understand how a moving median is computed in the program, follow the
example given below.
An Example: Suppose the data
has 1000 points (i.e., N = 1000), and the chosen width for the moving median
is 600 (i.e., DELTA = 600). Then, the quarter of DELTA is 150, and
this number of points at the start all have the same value which is equal
to their median. The value assigned to the 151th point is the median
of all points between the first and the 300th point (This is the half width
of DELTA). Again, for the 152th point, it is the median of all points
between the second and the 301th point. This continues similarly
till the 300th point. After that, median is taken from 600 data points
(the full width of DELTA) until the 300th point from the last point is
reached. Then, only the half of DELTA is used again, and the last
150 points all have the same value which is their median. Hence,
the pattern of grouping the data points for the median is done symmetrically
around the mid-point.
For each pixel, the data points are plotted along with the moving median
just calculated. The x-axis is the time in seconds, and the y-axis
is the signal in watts. The green dots represent the data points
with FLAG = 0 and 1, yellow plus signs show the masked out bright
signals, red dots denote the data points with FLAG = 2, red Xs depict the
flagged points using the cells, and finally the white line is the calculated
moving median. This result is also saved to the postscript file,
imap_result.ps. In the postscript file, different symbols are used
to distinguish between different data points instead of colors as in the
actual window display.
Immediately, the user is asked if he/she wants to
zoom in on the moving median curves. The y-axis scale gets blown
up considerably when they are zoomed in, so the user should not be alarmed
by the greatly exaggerated structures in the moving median curves.
The user is now asked if he/she wants to heavily
smooth these curves. The user is encouraged to smooth the curves
to rid of any isolated features in the curves. Each time the answer
is yes, the moving median curves are smoothed five times using a box car
average of width, DELTA/5.
If the answer to any of the above two questions is yes, then the zoomed
in plots show up. All the conventions are the same as before, but
now the smoothed curves will be overplotted.
Unusually low signals are detected where there is
a bright source. To correct for this, the program asks the user to
input a number of sigmas such that any data points found below the smoothed
curves by more than this amount should be flagged. The value of 1.0
is suggested, but a large value can be used to skip this sigma clipping.
These bad points get their FLAG set to 5.
The result of this clipping under the bright source is displayed.
The green dots show all the good data points, the red Xs represent the
flagged points from the cleaning with cells, and the red dots show the
flagged points whose signals are below the moving median by more than the
user-defined number of sigmas. This result is saved to the imap_result.ps
file. In this postscript result, only the data points with FLAG =
0 and the moving median curves are displayed to avoid clutter.
Asks the user to enter one of the following
If the answer is 1, then the MEDIAN is first normalized
by its final part. This means each element in the MEDIAN is divided
by the median of its last DELTA number of points. Next, each data
point is divided by this resulting MEDIAN irrespective of their FLAG values
for the long-term drift correction. If the answer is 2, then the
program moves back to the point Cof
the program. If the answer is 0 or anything other than 1 and 2, then
the program moves to the next part without applying this correction to
all data points; however, the result of any points flagged from the clipping
below the bright source is saved.
enter 1 to accept these fits (i.e., the moving median
curves) as the long-term drifts
enter 2 to redo the fits using a different width
enter 0 to skip this long-term drift correction all
Using only the data points found within the central
region of the map, 92 arcsecs smaller in all four sides (referring to the
plot of pixel centers - i.e., yellow dots in the first two displays at
the beginning of the program), and have FLAG = 0, the program takes the
median of all the signals for each pixel (SKYFLAT is the resulting array
in the program). Taking the mean of SKYFLAT, the skyflat factors
are calculated by dividing this mean value by individual SKYFLAT elements.
The result is printed out on the screen with the
following table format. The first one is for C100 and the next for
Pixel Layout Multiplying Factor from Skyflat
9 6 3
skyflat(9) skyflat(6) skyflat(3)
8 5 2
skyflat(8) skyflat(5) skyflat(2)
7 4 1
skyflat(7) skyflat(4) skyflat(1)
Pixel Layout Multiplying Factor from Skyflat
Asks the user if this skyflattening correction be applied to the data.
If the answer is yes, then all the data points are multiplied by the skyflat
factor according to their pixel label irrespective of their FLAG values.
The final result is plotted on two different display windows. The
first one shows the individual pixel data with time in seconds as the x-axis
and the signal in watts as the y-axis. The yellow dots represent
the valid data points, and the red dots show all the flagged points.
On the second display window, 3D surface plots are displayed (These are
very much like the ones shown before when masking bright sources).
On the top, it shows the data as before any of it changed from this
program, and the final resulting data is plotted on the bottom. These
two displays are also saved to the imap_result.ps file.
The results need to be saved in a PIA/SPD file format
if it is to be fed back into the PIA for mapping.
Asks the user if he/she want to save the results
to a new PIA/SPD file. If the answer is yes, then set all FLAG =
3 and 5 to FLAG = 2 and FLAG = 4 to FLAG =0, and a new PIA/SPD file is
created with ".ISPD_IMAP" ending containing the modified data. If
the answer is no, then the results are not saved but lost.
Appropriate message is issued on the screen before "DONE" shows up to signal
the end of the program.
If you have any questions or comments, please do not hesitate to contact
Nanyao Y. Lu
Min Y. Hur
Mail Stop 100-22
770 S. Wilson Avenue
Pasadena, CA 91125
Fax : (626) 397-7018
Last update : April 2000