NAME:
phot2db
PURPOSE:
Process a file of external photometry and add to phot database.
DESCRIPTION:
Reads a file of external photometry data with format specified below.
It creates entries in two tables of the phot database,
reference and data. The file defines a REFID in its header section,
which corresponds to an entry in phot.reference. Ordinarily this will
be a new REFID- if not, the existing entry in phot.reference will be
updated and existing entries in phot.data will be purged.
CATEGORY:
Photometry
CALLING SEQUENCE:
phot2db, fn
INPUTS:
fn filename to be read and processed.
OPTIONAL INPUT PARAMETERS:
KEYWORD INPUT PARAMETERS:
ADMIN - String, if specified, interpreted as an email address (or list)
to cc any email generated. If the contact information
obtained from the data file is unavailable or bad,
e-mail will addressed directly to ADMIN in the'to' field.
DATABASE - Name of database in which the reference and data tables reside,
by default 'phot'.
DATATABLE - Name of data table in data base, by default 'data'.
NOCONTACT - Flag, if set, the contact information specified in the data
file will not be used to send e-mail. The mail will be
sent to ADMIN only. If NOCONTACT is specified without ADMIN
it is equivalent to NOMAIL. Note that the contact field
is always mandatory in the data file.
NOMAIL - Flag, if set, the normal email sent in case of errors and
anomalies is suppressed. Errors will be printed to screen.
PATH - String, if specified, is a path that will be prepended to fn.
REFTABLE - Name of reference table in data base, by default 'reference'.
ESQUELCH - Maximum number of error lines in email messages, default 150.
Note that the preamble and data base update summary is always
printed regardless of ESQUELCH and that 'error lines' pertain
to header and data sections of the file, as well as a listing
of the header section of the file appended to the end. ESQUELCH
must be greater than 0.
TEST --Flag, if set, no databases are modified, queries that would be
generated are listed to the screen.
OUTPUTS:
KEYWORD OUTPUT PARAMETERS:
COMMON BLOCKS:
SIDE EFFECTS:
Updates tables data and reference in the phot database. It normally sends
email to the specified contact person (and admin) if errors or anomalies are
found in the file read.
RESTRICTIONS:
The header section of the file must specify at least REFID, OBSCODE, and
CONTACT.
The data section of the file must have at least 1 valid line, otherwise,
no changes are made to the database.
The REFID must not begin with "LO-" which is reserved for Lowell Observatory
internal use.
PROCEDURE:
The program is normally run as part of a cron job on a directory of incoming
photometry data. The data base tables are silently updated, unless errors or
anomalies are found. These are reported through email only. Once processed,
an external script moves data files to an accumulating archive.
The format of the photometry data files (as of 2006 Nov 21)::
The data file is composed of two sections, the header information and
the data. The header can be seen as an unbroken list of lines that have
the form of an identifier followed by text. It is generally expected
that one file will contain all the data that might result from a single
night of observation. However, this is not required. It is allowed
for a file to contain more than one night of data in which case the
RunDate information will not be provided. It is NOT allowed to have
multiple files for a single night on a single telescope and instrument
that share a RefID. The basic rule will be that one file = one RefID.
Also, RefID must be unique among all possible datasets.
Also, RefID must be unique among all possible datasets.
The specimen file below includes the recognized header keywords.
The order is not important- however the DATA line must follow
all the header lines.
REFID.....:
RUNDATE...:
INSTRUMENT:
OBSCODE...:
TELESCOPE.:
OBSERVER..:
MEASURER..:
CONTACT...:
COMMENTS..:
DATA:
# JD RA DEC FILTER MAG ERR OBJNAME
# 2443741.49403 12:12:12.1 +35:35:35 V 13.456 0.023 SAO 160688
# 2443741.59403 x x R 14.567 0.031 SAO 120107
# 2443741.69403 14:12:34.2 +17:23:12 R 14.567 0.031
#
The header contains keywords that are formatted as a name of up to
10 characters in length and padded to a length of 10 by periods.
The padded name is to be followed by a colon and then a space before
the data it contains. The order is not important. Some keywords
can usefully appear more than once providing for multi-line entries.
These fields are collected in the order seen in the header. It is an
error for a non-multi-line keyword to appear more than once. Note that a
multi-line comment need not be contiguous in the header. Blank (empty)
lines and any line that starts with '#' are ignored no matter where
they appear in the file. The end of the header is noted with 'DATA:'
(not padded). After this line is seen no further header information is
allowed and the rest of the file is expected to contain the data.
The header keywords contain the following information:
REFID
This is a string, currently limited to a maximum of 20 characters.
This string must be unique across all datasets. The string really
should be uniquely tied to a single file. When the database is loaded
from the file, the REFID value is used to identify old values that may
have been previously loaded. Prior to loading the new file, these old
values will be deleted in their entireity and the new values will be
placed into the database. No attempt is made to match up values in
the file with previously loaded data. The actual construction of the
REFID is not defined here but a good working example is to construct a
string that looks like: XXX-YYMMDD-IIII where XXX are the initials of
the contact (presumed to be responsible for the data), YYMMDD is the
rundate YY last two digits of the year, MM the month, and DD the day
in UT. In cases where the UT date changes during the night choose the
dominant UT date for the night. For example, at CTIO, the data often
changes an hour or so into the night. In this case the UT date at local
midnight is appropriate. Actually, the UT date of local midnight is
usually appropriate though keep in mind that local midnight does not mean
"when is it midnight according to a local clock". Finally, IIII is a
string that identifies the instrument (does not need to be exactly four
characters long).
It is currently required that the REFID be a string with three non-null
substrings separated by hyphen (- not _). For example,
XXX-YYMMDD-IIII is legal but YYMMDD or XXX-IIII are not (nor is XXX--AAAA).
This restriction is needed to avoid propagating RefID's that could conflict
with internal usage at Lowell Observatory.
RUNDATE
This field is not required but is very helpful in many cases. This should
be a 6-digit string, YYMMDD with the same meaning discussed under REFID.
If not supplied this will be set to NULL in the database. If the data
being loaded apply to a single night this field makes a lot of sense.
Data that may come from a larger collective source (SDSS photometry database
for instance) the RUNDATE will be meaningless.
for instance) the RUNDATE will be meaningless.
INSTRUMENT
INSTRUMENT
This is a short identifying string, maximum of 12 characters. There should
be a one-to-one correspondence between the instrument being used and this
name. It is most useful if this really is a useful identifying string that
is unique and constant for a given instrument.
is unique and constant for a given instrument.
OBSCODE
OBSCODE
This is the 3-character standard observatory code as defined by the
Minor Planet center.
Minor Planet center.
TELESCOPE
TELESCOPE
Descriptive name of the telescope used to collect the data. Maximum length
saved in the database is 80 characters.
saved in the database is 80 characters.
OBSERVER
OBSERVER
[Multiple line field] Name(s) of the observers who collected the data at
the telescope. This field is not required but is strongly encouraged for
normal nightly datasets. You can put multiple names on one line or use
one line per observer.
one line per observer.
MEASURER
MEASURER
[Multiple line field] Name(s) of the who processed the raw data and generated
the numbers in this file. Usually this will be one person but multiple names
are allowed and encouraged if appropriate.
are allowed and encouraged if appropriate.
CONTACT
CONTACT
Email address of the primary contact regarding the data. This should be the
contact address of someone that can be contacted long after the data are
reduced and posted in case of questions. It is not appropriate to use a
transient worker as the contact (eg., a summer student). It is also assumed
and normal to expect that the contact is mentioned by name as one of the
measurers.
measurers.
COMMENTS
COMMENTS
[Multiple line field] Any comments that should accompany the data would be
placed in this field. For that sake of formatting by any system that reads
these commands and generates output, a blank comment line (keyword but no
text) will be preserved and will indicate a paragraph break. Line breaks
within a "paragraph" may or may not be preserved.
within a "paragraph" may or may not be preserved.
DATA:
DATA:
Once this keyword is seen the file processing switches to data mode
reading. The data format is not really a fixed format file. Instead,
the lines are read by word tokens and there must be a minimum of 6
"words" on every line. Words are separated by one or more blanks.
Leading and trailing blanks are ignored. The first word is the Julian
Date and an F13.5 format is recommended though more digits can be provided
if desired. The second word is the J2000 Right Ascension of the object
in either sexigesimal format (HH:MM:SS.s) or decimal hours. The third
word is the Declination (DD:MM:SS or decimal degrees). The RA and Dec
should NOT be supplied if this is not a measured quantity. In other
words, do not provide catalog positions or ephemeris calculations.
If you do not have measured positions, then put a single 'x' character
in place of each of the two fields. The fourth word is the name of the filter.
Blanks are NOT allowed in a filter name. This name should be a standard
name to the extent possible. For instance, Johnson filters would be V
and SDSS would use something like r'. The filter string may not execeed 10
characters in length. The next two words are the standard
magnitude and its error. Use as many digits of precision as appropriate.
The remainder of the line is used for the object name. Embedded blanks are
allowed in the object names though multiple blanks will get collapsed to a
single blank. It is allowed (and normal) to omit the object name when a
position is provided. If the position is not provided then the object name
is required. Again, blank lines and lines starting with # are skipped when
reading.
MODIFICATION HISTORY:
2006/11/21, Written by Peter L. Collins, Lowell Observatory