From mazz@ipac.caltech.eduWed Feb 26 08:32:35 1997
Date: Wed, 26 Feb 1997 08:11:49 -0800 (PST)
From: Joe Mazzarella
To: ISIS -- Nian-Ming Chiu ,
Roc Cutri , Tracey Evans ,
John Good , Carol Lonsdale ,
Joe Mazzarella ,
Sherry Wheelock
Cc: Tom Chester
Subject: 25-Feb-97 2MASS DBWG (ISISWG) Meeting Minutes
***************************************************************************
25-Feb-97 2MASS DBWG (ISISWG) Meeting Minutes
ATTENDED: Roc C., Nian C., Tracy E., John G., Joe M., Sherry W.
---------------------------------------------------------------------------
GENERAL DISCUSSION AND FACTOIDS
Roc:
1) Significent events coming up:
~18 March: First end-to-end dry run of pipeline. This should include DBMAN,
an Informix 'bit bucket' for one night's data, and the ability to perform
some basic queries. We should also plan to provide "postage-stamp" server
access for the 20/1 compressed dry-run images.
Space estimate summary for one night's data:
DBMAN tabular DB output: ~0.5 GB in Informix. (See previous meeting minutes.)
Images: 10.4 GB survey + 1.7 GB calibrations ~= 12 GB total.
20/1 compress images: ~= 0.6 GB
31Mar-1 Apr: IPAC User's Committee Meeting. This will include a presentation
and demo of progress regarding 2MASS (and IPAC) Web services.
Joe:
1) An archive of the code from our Informix class excersises is available at
/2massdba/tools/import/InformixClass.
2) 2MASS DB-access Web (ISIS) needs a machine to host the Web server that can
handle the rate of hits, etc. The issues are non-trivial: network bandwidth
(isolation of "production" from "users"), security, etc. We should have a
small meeting of network, Informix, & HTTP config. experts to engineer this
right. The new 2MASS Web docs can be split out from ISIS proper and
configured on phedra if this is desireable.
3) Concern: Coding for the dry-run on ~18 Mar should likely have higher
priority than writing more documentation and preparing details of IPAC
ISIS meeting. This would argue for the later 8 Apr. meeting option.
Opinions welcome!
Tracy:
1) For March dry-run, the DB output of DBMAN will be one relation (table)
for point sources and one for extended sources. Revisions for efficient
vertical (column) and horizontal (row) partitioning will come later.
There is sufficient space (~0.5 GB) in tmass db to catch one night's data.
2) Needs a 'tmassdbo' account/userid to load and maintain DBMAN output,
rather than her personal userid ('evans'). Nian will work this.
3) Brought up the need for a GUI (ISIS extensions or stand-alone) for
tmass db loading tasks. This would be a front end to tasks:
dbadd - add a scan or scans for a night's run
dbdelete - zap scans
dbclean - TBD
We agree this should be an important (internal-only) branch of ISIS, but that
details of designing/coding this should wait until after March dry-run.
Command-line programs will hopefully suffice until then.
All:
1) Informix training class was very good and got us all off to a good start
with standard relational SQL business. We will need follow-up training with
the OO/DataBlade extensions of Universal Server (formerly Illustra) later.
(There is currently no single class or Informix instructor that can cover it
all.)
2) Tracy asked about "catalog production", as opposed to "working DB"
production via DBMAN. All agreed that this should amount to QA and science
analysis to determe proper thresholds, data subsets, etc. *using ISIS and
Informix*. Official "catalogs" will likely amount to DB views (and possibly
periodic bulk extractions?) of proper subsets, etc. Thus, catalog production
does not impose requirements on DBMAN itself. Of course, details are TBD and
part of our long-term work!
---------------------------------------------------------------------------
COMPLETED ACTIONS
John:
o Developing/testing/document
Tracy:
o DBMAN progress
Joe:
o Progress on WebCatScan interface.
o Email to reschedule ISIS meeting including draft agenda was
distributed on 24 Feb. No clear answer yet on 19 Mar vs. 8 Apr.
We should be prepared for a 19 Mar presentation!
Sherry:
o Restored the 2MASS protocam data postage-stamp server on
karloff (it was down after subnet move and Solaris upgrade).
It is available at http://karloff.ipac.caltech.edu:8080/
Mazz adds: There is a hook to it in the prototype ISIS interface
http://crystal:8080/isis/. We will eventually revamp (modularize)
portions of postage-stamp access and integrate it with ISIS
workspace protocols, etc.
---------------------------------------------------------------------------
PRIORITY ONGOING ACTIONS (In rough priority order.
Long-term roles/responsibilities/tasks omitted here. See also our Web docs.)
Tracy:
o Develope and test initial DBMAN for end-March run-through.
o DBMAN->Informix loading/indexing issues:
-w/ and w/o various indexes, clustered indexes, etc.
-Since Informix user access will have to be locked during
DBMAN runs, what is the best strategy and period
(nightly, weekly, ...?) for updates?
o Refinements on SIS's for DBMAN & WSDB field definitions:
extended sources, mag & color uncertainties and upper limits.
o Treat galaxy data. (See previous memos).
o Calibration scans need SIS's and tables in the DBMS.
~40 fields expected; define indexing scheme.
John:
o Testing, documentation and configuration in /2massdba of
infrastructure (RSM/LSM, structure handling and workspace) code
so others can begin using it for new apps and interface code.
Coord w/ Joe.
o Design User System & Infrastructure Apps (w/ Joe)
o Agenda and viewgraphs for IPAC ISIS meeting (w/ Joe&Carol&Roc)
o Infrastructure (& User System w/ Joe) design docs.
o Design and document the data "containers" (partitions) we
will use for the source DB. How will table fields be partitioned
vertically (split into relations), and how will objects=rows be
partitioned in data-space to maximize query efficiency, etc.
o Call Cassini Project and get their story on their raid disk.
o Summarize findings (w/ Nian) about HP/Informix platform.
Does this hardware/software bundle offer significant benefits?
Summarize in email to team.
Nian:
o Working on high-priority Survey Operations DB (non-Informix) project.
~1 more month?
o Plan for incorporating /k* disks into Informix space as the
disks are cleared and become available (Probably soon after
real survey data start poring in, when protocam data can be
taken off mag. disk and archived near- or off-line.)
Roc:
o Refine mag/color uncertainties and upper limits (w/ Sci Team)
for DNMAN SIS (to Tracy).
Joe:
o Develope (w/ John) prototype "WebCatScan" interface. Needed for:
March/April dry-run queries, ISIS meeting, Users Committee demos
o Design User System & Infrastructure Apps (w/ John)
o Agenda and viewgraphs for IPAC ISIS meeting (w/ John&Carol&Roc)
o Discussions (w/ John) with IPAC mission managers in preparation
for general ISIS req./design review meeting.
ISO/SIRTF (George & Rick): Done.
WIRE (Carol, Tim, Perry): TBD
MSX (Mehrdad): TBD
o User System (& Infrastructure w/ John) design docs.
o Make an "RTB" for Sybase->Informix db migration (FSC to start)
Carol:
o Agenda and viewgraphs for IPAC ISIS meeting (w/ Joe&John&Roc)
---------------------------------------------------------------------------
NEW ACTIONS (See also ONGOING above and in Web Task List)
Joe,John,Carol,...:
o Plan & prepare presentation and demo for User's Committee meeting.
Joe:
o Get experts advice on (at least an initial) solution for where
to set up the 2MASS/DB-access Web server. See DISCUSSION above.
Transfer new 2MASS doc pages (split out from ISIS proper) to
public Web ASAP (Jarrett needs this.) Coordinate with Rick.
Joe & John:
o Initial *detailed* design (SIS) of Informix SQL access programs.
(Best of XCATSCAN's "scan" and "maktbl" and John's Sybase
SQL client/server code revamped for ISIS.)
*** NOTE: A temp/prototype solution will have to be used
for the dry run due to our strict time constraints and
multitasking loads! John has volunteered to make this based
on one of our last Informix exercises. Long-term we will need
programming help (cog-E FTEs) for these and related ESQL/C apps.
Nian:
o Transfer tmass db/table ownership from 'evans' to 'tmassdbo'.
o Answer for us whether a user with an Informix account can
transfer table ownership to another user, or whether that is
possible only for the db-admin (you).
Tracy:
o Now new/additional tasks (lucky you!)
Roc:
o Provide 2 or 3 high priority queries desired for dry-run DB.
o Provide QA/analysis task priorities for ISIS development.
Sherry:
o Plan for postage stamp image access for pipeline dry-run output.
Work w/ Joe & John to revise and modularize code for ISIS
processes.
---------------------------------------------------------------------------