Starlink User Note256
Mark Taylor
23 October 2008
$Id: sun256.xml,v 1.192 2008-10-23 15:21:34 mbt Exp $
stilts
command
addcol
addskycoords
assert
badval
cache
check
clearparams
colmeta
delcols
every
explodeall
explodecols
head
keepcols
meta
progress
random
replacecol
replaceval
select
sequential
setparam
sort
sorthead
stats
tablename
tail
transpose
uniq
sky
: Sky Matching
skyerr
:
Sky Matching with Per-Object Errors
sky3d
: Spherical Polar Matching
exact
: Exact Matching
1d
, 2d
, ...:
Isotropic Cartesian Matching
2d_anisotropic
, ...:
Anisotropic Cartesian Matching
calc
: Evaluates expressions
coneskymatch
: Crossmatches table on sky position against Cone Search service
funcs
: Browses functions used by algebraic expression langauage
plot2d
: 2D Scatter Plot
plot3d
: 3D Scatter Plot
plothist
: Histogram
regquery
: Queries the VO registry
server
: Runs an HTTP server to perform STILTS commands
sqlclient
:
Executes SQL statements
sqlskymatch
:
Crossmatches table on sky position against SQL table
sqlupdate
: Updates values in an SQL table
tcat
: Concatenates multiple similar tables
tcatn
: Concatenates multiple tables
tcopy
: Converts between table formats
tcube
: Calculates N-dimensional histograms
tjoin
: Joins multiple tables side-to-side
tmatch1
: Performs a crossmatch internal to a single table
tmatch2
: Crossmatches 2 tables using flexible criteria
tmatchn
: Crossmatches multiple tables using flexible criteria
tpipe
: Performs pipeline processing on a table
tskymatch2
: Crossmatches 2 tables on sky position
votcopy
: Transforms between VOTable encodings
votlint
: Validates VOTable documents
STILTS is a set of command-line tools for processing tabular data. It has been designed for, but is not restricted to, use on astronomical data such as source catalogues. It contains both generic (format-independent) table processing tools and tools for processing VOTable documents. Facilities offered include crossmatching, format conversion, format validation, column calculation and rearrangement, row selection, sorting, plotting, statistical calculations and metadata display. Calculations on cell data can be performed using a powerful and extensible expression language.
The package is written in pure Java and based on STIL, the Starlink Tables Infrastructure Library. This gives it high portability, support for many data formats (including FITS, VOTable, text-based formats and SQL databases), extensibility and scalability. Where possible the tools are written to accept streamed data so the size of tables which can be processed is not limited by available memory. As well as the tutorial and reference information in this document, detailed on-line help is available from the tools themselves.
STILTS is available under the GNU General Public Licence.
STILTS provides a number of command-line applications which can be used for manipulating tabular data. Conceptually it sits between, and uses many of the same classes as, the packages STIL, which is a set of Java APIs providing table-related functionality, and TOPCAT, which is a graphical application providing the user with an interactive platform for exploring one or more tables. This document is mostly self-contained - it covers some of the same ground as the STIL and TOPCAT user documents (SUN/252 and SUN/253 respectively).
Currently, this package consists of commands in the following categories:
tcopy
,
tpipe
,
tcat
,
tcatn
,
tjoin
and
tcube
(see Section 5).
tmatch1
,
tmatch2
,
tmatchn
and
tskymatch2
(see Section 6).
plot2d
,
plot3d
and
plothist
(see Section 7).
votcopy
and
votlint
.
regquery
and
coneskymatch
.
sqlclient
,
sqlupdate
and
sqlskymatch
.
calc
,
funcs
and
server
.
There are many ways you might want to use these tools; here are a few possibilities:
server
command may help,
but is not required, for use in these situations.
stilts
command
All the functions available in this package can be used from
a single command, which is usually referred to in this document
simply as "stilts
". Depending on how you have installed
the package, you may just type "stilts
",
or something like
java -jar some/path/stilts.jaror
java -classpath topcat-lite.jar uk.ac.starlink.ttools.Stiltsor something else - this is covered in detail in Section 3.
In general, the form of a command is
stilts <stilts-flags> <task-name> <task-args>The forms of the parts of this command are described in the following subsections, and details of each of the available tasks along with their arguments are listed in the command reference at the end of this document. Some of the commands are highly configurable and have a variety of parameters to define their operation. In many cases however, it's not complicated to use them. For instance, to convert the data in a FITS table to VOTable format you might write:
stilts tcopy cat.fits cat.vot
Some flags are common to all the tasks in the STILTS package,
and these are specified after the stilts
invocation itself
and before the task name. They generally have the same effect
regardless of which task is running. These generic flags are as
follows:
-help
stilts
command
itself and exits. The message contains a listing of all the
known tasks.
-version
-verbose
-disk
-Dstartable.storage=disk
.
-debug
-prompt
-prompt
flag,
then you will be prompted for every parameter you have not
explicitly specified to give you an opportunity to enter a value
other than the default.
-batch
-batch
flag,
then you won't be prompted at all.
-bench
-checkversion <vers>
<vers>
. If it is not, STILTS will exit with
an error. This can be useful when executing in certain controlled
environments to ensure that the correct version of the application
is being picked up.
-stdout <file>
-
" for <file>
will restore this default behaviour.
-stderr <file>
-
" for <file>
will restore this default behaviour.
If you are submitting an error report, please include the result of
running stilts -version
and the output of the troublesome
command with the -debug
flag specified.
The <task-name>
part of the command line is the
name of one of the tasks listed in Appendix B - currently
the available tasks are:
calc
funcs
coneskymatch
plot2d
plot3d
plothist
regquery
server
sqlclient
sqlskymatch
sqlupdate
tcat
tcatn
tcopy
tcube
tjoin
tmatch1
tmatch2
tmatchn
tpipe
tskymatch2
votcopy
votlint
The <task-args>
part of the command line is a
list of parameter assignments,
each giving the value of one of the named parameters belonging to
the task which is specified in the <task-name>
part.
The general form of each parameter assignment is
<param-name>=<param-value>If you want to set the parameter to the null value, which is legal for some but not all parameters, use the special string "
null
".
In some cases you can optionally leave out the <param-name>
part of the assignment (i.e. the parameter is positionally determined);
this is indicated in the task's usage description if the parameter
is described like [<param-name>=]<param-value>
rather than <param-name>=<param-value>
.
If the <param-value>
contains spaces or other special
characters, then in most cases, such as from the Unix shell, you will
have to quote it somehow. How this is done depends on your platform,
but usually surrounding the whole value in single quotes will do the trick.
Tasks may have many parameters, and you don't have to set all of them explicitly on the comand line. For a parameter which you don't set, two things can happen. In many cases, it will default to some sensible value. Sometimes however, you may be prompted for the value to use. In the latter case, a line like this will be written to the terminal:
matcher - Name of matching algorithm [sky]:This is prompting you for the value of the parameter named
matcher
. "Name of matching algorithm" is a short
description of what that parameter does. "sky
" is
the default value (if there is no default, no value will appear
in square brackets).
At this point you can do one of four things:
null
".null
" means the null value,
which is legal for some, but not all parameters.
If the value you enter is not legal, you will see an error
message and you will be invited to try again. help
" or a question mark "?
".
This will output a message
giving a detailed description of the parameter
and prompt you again.stilts
command itself (see Section 2.1).
If you supply the -prompt
flag, then you will be prompted
for every parameter you have not explicitly set. If you supply
-batch
on the other hand, you won't be prompted for
any parameters (and if you fail to set any without legal default
values, the task will fail).
If you want to see the actual values of the parameters for a task
as it runs,
including prompted values and defaulted ones
which you haven't specified explicitly,
you can use the -verbose
flag after the stilts
command:
% stilts -verbose tcopy cat.fits cat.vot ifmt=fits INFO: tcopy in=cat.fits out=cat.vot ifmt=fits ofmt=(auto)
Extensive help is available from stilts
itself about task and its parameters, as described in the next section.
As well as the command descriptions in this document (especially the reference section Appendix B) you can get help for STILTS usage from the command itself. Typing
stilts -helpresults in this output:
Usage: stilts [-help] [-version] [-verbose] [-disk] [-debug] [-prompt] [-batch] [-bench] [-checkversion <vers>] [-stdout <file>] [-stderr <file>] <task-name> <task-args> stilts <task-name> help[=<param-name>|*] Known tasks: calc coneskymatch funcs regquery plot2d plot3d plothist server sqlclient sqlskymatch sqlupdate tcat tcatn tcopy tcube tjoin tmatch1 tmatch2 tmatchn tpipe tskymatch2 votcopy votlint
For help on the individual tasks, including their parameter lists,
you can supply the word help
after the task name, so for instance
stilts tcopy helpresults in
Usage: tcopy ifmt=<in-format> ofmt=<out-format> [in=]<table> [out=]<out-table>
Finally, you can get help on any of the parameters of a task
by writing help=<param-name>
, like this:
stilts tcopy help=ingives
Help for parameter IN in task TCOPY ----------------------------------- Name: in Usage: [in=]<table> Summary: Location of input table Description: The location of the input table. This is usually a filename or URL, and may point to a file compressed in one of the supported compression formats (Unix compress, gzip or bzip2). If it is omitted, or equal to the special value "-", the input table will be read from standard input. In this case the input format must be given explicitly using the ifmt parameter.If you use "
*
" instead of a parameter name in this usage,
help for all the parameters will be printed. Note that in most shells
you will probably need to quote the asterisk, so you should write
stilts tcopy help='*'
In some cases, as described in Section 2.3, you will be prompted for the value of a parameter with a line something like this:
matcher - Name of matching algorithm [sky]:In this case, if you enter "
help
" or a question mark,
then the parameter help entry will be printed to the screen, and
the prompt will be repeated.
For more detailed descriptions of the tasks, which includes explanatory comments and examples as well as the information above, see the full task descriptions in the Command Reference.
There are a number of ways of invoking the stilts
command,
depending on how you have installed the package.
If you're using a Unix-like operating system,
the easiest way is to use the stilts
script.
If you have a full starjava installation it is in the
starjava/bin
directory.
Otherwise you can download it separately from wherever you got your
STILTS installation in the first place, or find it
at the top of the stilts.jar
or topcat-*.jar
that contains your STILTS installation, so do something like
unzip stilts.jar stilts chmod +x stiltsto extract it (if you don't have
unzip
,
try jar xvf stilts.jar stilts
).
stilts
is a simple shell script which just invokes java with the
right classpath and the supplied arguments.
To run using the stilts
script, first make sure that
both the java
executable and the stilts
script itself are on your path,
and that the stilts.jar
or topcat-*.jar
jar file is in the same directory as stilts
.
Then the form of invocation is:
stilts <java-flags> <stilts-flags> <task-name> <task-args>A simple example would be:
stilts votcopy format=binary t1.xml t2.xmlin this case, as often, there are no
<java-flags>
or
<stilts-flags>
.
If you use the -classpath
argument or have a CLASSPATH environment variable set,
then classpath elements thus specified will be added to the classpath
required to run the command.
The examples in the
command descriptions below use this form for convenience.
If you don't have a Unix-like shell available however,
you will need to invoke
Java directly with the appropriate classes on your classpath.
If you have the file stilts.jar
, in most cases you can
just write:
java <java-flags> -jar stilts.jar <stilts-flags> <task-name> <task-args>which in practice would look something like
java -jar /some/where/stilts.jar votcopy format=binary t1.xml t2.xml
In the most general case, Java's -jar
flag might be
no good, for one of the following reasons:
stilts.jar
file (such as topcat-full.jar
)java <java-flags> -classpath <class-path> uk.ac.starlink.ttools.Stilts <stilts-flags> <task-name> <task-args>The example above in this case would look something like:
java -classpath /some/where/topcat-full.jar uk.ac.starlink.ttools.Stilts votcopy format=binary t1.xml t2.xml
Finally, as a convenience, it is possible to run STILTS from a
TOPCAT
installation by using its -stilts
flag, like this:
topcat <java-flags> -stilts <stilts-flags> <task-name> <task-args>This is possible because TOPCAT is built on top of STILTS, so contains a superset of its code.
The
<stilts-flags>
,
<task-name>
and
<task-args>
parts of these invocations are explained in Section 2,
and the
<class-path>
and
<java-flags>
parts are explained in the following subsections.
The classpath is the list of places that Java looks to find
the bits of compiled code that it uses to run an application.
Depending on how you have done your installation the core STILTS
classes could be in various places, but they are probably in a
file with one of the names
stilts.jar
,
topcat-lite.jar
or
topcat-full.jar
.
The full pathname of one of these files can therefore be used as
your classpath. In some cases these files are self-contained and
in some cases they reference other jar files in the filesystem -
this means that they may or may not continue to work if you
move them from their original location.
Under certain circumstances the tools might need additional classes, for instance:
In most cases it is not necessary to specify any additional arguments to the Java runtime, but it can be useful in certain circumstances. The two main kinds of options you might want to specify directly to Java are these:
-Dname=value
.
So for instance to ensure that temporary files are written to
the /home/scratch
directory, you could use the flag
-Djava.io.tmpdir=/home/scratch
-Xmx
flag. To set the heap
memory size to 256 megabytes, use the flag
-Xmx256M(don't forget the 'M' for megabyte). You will probably find performance is dreadful if you specify a heap size larger than the physical memory of the machine you're running on.
Note however that encouraging STILTS to use disk files
rather than memory for temporary storage is often a
better idea than boosting the heap memory -
this is done by specifying the -disk
flag
(stilts -disk <task-name> ...
),
or possibly setting the system property
-Dstartable.storage=disk
(see Section 2.1).
You can specify other options to Java such as tuning and profiling flags etc, but if you want to do that sort of thing you probably don't need me to tell you about it.
If you are using the stilts
command-line script,
any flags to it starting -D
or -X
are passed
directly to the java
executable.
You can pass other flags to Java with the stilts
script's
-J
flag; for instance:
stilts -Xmx4M -J-verbose:gc calc 'mjdToIso(0)'is equivalent to
java -Xmx4M -verbose:gc -jar stilts.jar calc 'mjdToIso(0)'
System properties are a way of getting information into the
Java runtime - they are a bit like environment variables.
There are two ways to set them when using STILTS: either
on the command line using arguments of the form
-Dname=value
(see Section 3.2)
or in a file in your home directory named
.starjava.properties
, in the form of a
name=value
line.
Thus submitting the flag
-Dvotable.strict=trueon the command line is equivalent to having the following in your
.starjava.properties
file:
# Force strict interpretation of the VOTable standard. votable.strict=true
The following system properties have special significance to STILTS:
java.awt.headless
true
" if running the
plotting tasks on a headless server.
You only need to worry about this if you see error messages
complaining about headlessness.
java.io.tmpdir
-disk
flag has been
specified (see Section 2.1).
jdbc.drivers
jel.classes
mark.workaround
mark()
/reset()
methods of some java
InputStream
classes. These are rather common,
including in Sun's J2SE system libraries.
Use this if you are seeing errors that say something like
"Resetting to invalid mark
".
Currently defaults to "false".startable.readers
startable.storage
disk
" has basically the same effect as
supplying the "-disk
" argument on the command line
(see Section 2.1). Other possible values are
"memory
", "sideways
" and "discard
";
see SUN/252.
startable.writers
votable.strict
true
for strict enforcement of the VOTable standard
when parsing VOTables. This prevents the parser from working round
certain common errors, such as missing arraysize
attributes on FIELD
or PARAM
elements with datatype="char"
.
False by default.
This section describes additional configuration which must be done to allow the commands to access SQL-compatible relational databases for reading or writing tables. If you don't need to talk to SQL-type databases, you can ignore the rest of this section. The steps described here are the standard ones for configuring JDBC (which sort-of stands for Java Database Connectivity), described in more detail on Sun's JDBC web page.
To use STILTS with SQL-compatible databases you must:
jdbc.drivers
system property to the name of the
driver class as described in Section 3.3
These steps are all standard for use of the JDBC system. See SUN/252 for information about JDBC drivers known to work with STIL (the short story is that at least MySQL and PostreSQL will work).
Here is an example of using tcopy
to write the results
of an SQL query on a table in a MySQL database as a VOTable:
stilts -classpath /usr/local/jars/mysql-connector-java.jar \ -Djdbc.drivers=com.mysql.jdbc.Driver \ tcopy \ in="jdbc:mysql://localhost/db1#SELECT id, ra, dec FROM gsc WHERE mag < 9" \ ofmt=votable gsc.votor invoking Java directly:
java -classpath stilts.jar:/usr/local/jars/mysql-connect-java.jar \ -Djdbc.drivers=com.mysql.jdbc.Driver \ uk.ac.starlink.ttools.Stilts tcopy \ in="jdbc:mysql://localhost/db1#SELECT id, ra, dec FROM gsc WHERE mag < 9" \ ofmt=votable out=gsc.votYou have to exercise some care to get the arguments in the right order here - see Section 3.
Alternatively, you can set some of this up beforehand to make the invocation easier. If you set your CLASSPATH environment variable to include the driver jar file (and the STILTS classes if you're invoking Java directly rather than using the scripts), and if you put the line
jdbc.drivers=com.mysql.jdbc.Driverin the
.starjava.properties
file in your home directory,
then you could avoid having to give the -classpath
and
-Djdbc.drivers
flags respectively.
Most of the tools in this package either read one or more tables as input, or write one or more tables as output, or both. This section explains what kind of tables the tools can read and write, and how you tell them where to find the tables to operate on.
In most cases input and output table specificcations are given by parameters with the following names (or similar ones):
in
ifmt
out
ofmt
The location of tables for input and output are usually given using
the in
and out
parameters respectively.
These are often, but not always, filenames. The possibilities are these:
http:
ftp:
file:
jar:
myspace:
myspace:/survey/iras_psc.xml
",
and can access files in the myspace are that the user is currently
logged into.
These URLs can be used for both input and output of tables.
To use them you must have an AstroGrid account and the AstroGrid
WorkBench or similar must be running; if you're not currently
logged in a dialogue will pop up to ask you for name and
password.ivo:
ivo://uk.ac.le.star/filemanager#node-2583
".
These URLs can be used for both input and output of tables.
To use them you must have an AstroGrid account and the AstroGrid
WorkBench or similar must be running; if you're not currently
logged in a dialogue will pop up to ask you for name and
password.jdbc:
-
")
In any of these cases, for input locations compression is taken care
of automatically. That means that you can give the filename or URL
of a file which is compressed using gzip
, bzip2
or Unix compress
and the program will uncompress it on the fly.
The generic table commands in STILTS
(currently tpipe
,
tcopy
,
tcat
,
tcatn
,
tcube
,
tjoin
,
tmatch1
,
tmatch2
,
tmatchn
,
tskymatch2
,
plot2d
,
plot3d
,
plothist
,
coneskymatch
,
sqlskymatch
and
regquery
)
have no native format for table storage, they can process
data in a number of formats equally well.
STIL has its own model of what a table
consists of, which is basically:
The formats the package knows about are dependent on the input and output handlers currently installed. The ones installed by default are listed in the following subsections. More may be added in the future, and it is possible to install new ones at runtime - see the STIL documentation for details.
Some of the tools in this package ask you to specify the format
of input tables using the ifmt
parameter.
The following list gives the values usually allowed for this
(matching is case-insensitive):
fits
colfits
votable
TABLE
element is used,
but this can be altered
by supplying the 0-based index after a '#
' sign,
so "table.xml#4" means the fifth TABLE
element in the document.
ascii
csv
tst
ipac
wdc
In some cases (when using VOTable or FITS format tables) the tools can detect the table format automatically, and no explicit specification is necessary. If this isn't the case and you omit the format specification, the tool will fail with a suitable error message. It is always safe to specify the format explicitly; this will be slightly more efficient, and may lead to more helpful error messages in the case that the table can't be read correctly.
Some of the tools ask you to specify the format of output tables
using the ofmt
parameter.
The following list gives the values usually allowed for this;
in some cases as you can see there are several variants of a given format.
You can abbreviate these names, and the first match in the list below
will be used, so for instance specifying votable
is equivalent
to specifying votable-tabledata
and fits
is equivalent to fits-plus
.
Matching is case-insensitive.
fits-plus
fits-basic
for most purposes)fits-basic
colfits-plus
colfits-basic
votable-tabledata
votable-binary-inline
STREAM
elementvotable-binary-href
votable-fits-href
votable-fits-inline
STREAM
elementascii
text
csv
csv-noheader
tst
html
TABLE
elementhtml-element
TABLE
elementlatex
tabular
environmentlatex-document
tabular
environmentmirage
In some cases the tools may guess what output format you want by looking at the extension of the output filename you have specified.
Several of the tasks available in STILTS take one or more input tables, do something or other with them, and produce an output table. This is a pretty obvious way to go about things, and in the most straightforward case that's exactly what happens: you name one or more input tables, specify the processing parameters, and name an output table; the task then reads the input tables from disk, does the processing and writes the output table to disk.
However, many of the tasks in STILTS allow you to do pre-processing of the input tables before the main job, post-processing of the output table after the main job, and to decide what happens to the final tabular result, without any intermediate storage of the data. Examples of the kind of pre-processing you might want to do are to rearrange the columns so that they have the right units for the main task, or replace 'magic' values such as -999 with genuine blank values; the kind of post-processing you might want to do is to sort the rows in the output table or delete some of the columns you're not interested in. As for the destination of the final table, you might want to write it to disk, but equally you might not want to store it anywhere, but only be interested in counting the number of rows, or seeing the minima/maxima of a few of the columns, or you might want to send it straight to TOPCAT or some other table viewing application for interactive analysis.
Clearly, you could achieve the same effect by running multiple applications: preprocess your original input tables to write intermediate files on disk, run the main processing application which reads those files from disk and writes a new output file, run another application to postprocess the output file and write a new final output file, and finally do something with this such as counting the rows in it or viewing it in TOPCAT. However, by doing it all within a single task instead, no intermediate results have to be stored, and the whole sequence can be very much more efficient. You can think of this (if it helps) like a Unix pipeline, except what is being streamed from the start to the end of the pipe is not bytes, but table metadata and data. In most cases, the table data is streamed through the pipeline a row at a time, meaning that the amount of memory required is small (though in some cases, for instance row sorting and crossmatching, this is not possible).
Tasks which allow this pre/post-processing, or "filtering",
have parameters with names like "cmd
" which you
use to specify processing steps.
Tasks with multiple input tables
(tmatch2
,
tskymatch2
,
tcatn
,
tjoin
)
may have parameters named icmd1
, icmd2
, ...
for preprocessing the different input tables and
ocmd
for postprocessing the output table.
tpipe
does nothing except
filtering, so there is no distinction between pre- and post-processing,
and its filter parameter is just named cmd
.
tpipe
additionally has a script
parameter which allows you to use a text file to write the
commands in, to prevent the command line getting too long.
In both cases there is a parameter named omode
which defines the "output mode", that is, what happens to the
post-processed output table that comes out of the end of the pipeline.
Section 5.1 lists the processing steps available,
and explains how to use them,
Section 5.2 and Section 5.3 describe the syntax
used in some of these filter commands for specifying columns,
and Section 5.4 describes the available output modes.
See the examples in the
command reference,
and particularly the
tpipe
examples,
for some examples putting all this together.
This section lists the filter commands which can be used for
table pipeline processing, in conjunction with cmd
-
or script
-type parameters.
You can string as many of these together as you like.
On the command line, you can repeat the cmd
(or icmd1
, or ocmd
...) parameter
multiple times, or use one cmd
parameter and
separate different filter specifiers with semicolons (";
").
The effect is the same.
It's important to note that each command in the sequence of processing steps acts on the table at that point in the sequence. Thus either of the two identical invocations:
stilts tpipe cmd='delcols 1; delcols 1; delcols 1' stilts tpipe cmd='delcols 1' cmd='delcols 1' cmd='delcols 1'has the same effect as
stilts tpipe cmd='delcols "1 2 3"'since in the first case the columns are shifted left after each one is deleted, so the table seen by each step has one fewer column than the one before. Note also the use of quotes in the latter of the examples above, which is necessary so that the
<colid-list>
of the delcols
command is interpreted as one argument not
three separate words.
The available filters are described in the following subsections.
addcol
Usage:
addcol [-after <col-id> | -before <col-id>] [-units <units>] [-ucd <ucd>] [-desc <description>] <col-name> <expr>
Add a new column called <col-name>
defined
by the algebraic expression <expr>
.
By default the new column appears after the last column
of the table, but you can position it either before or
after a specified column using the -before
or -after
flags respectively.
The -units
, -ucd
and
-desc
flags can be used to define
metadata values for the new column.
Syntax for the <expr>
and <col-id>
arguments is described in the manual.
addskycoords
Usage:
addskycoords [-epoch <expr>] [-inunit deg|rad|sex] [-outunit deg|rad|sex] <insys> <outsys> <col-id1> <col-id2> <col-name1> <col-name2>
Add new columns to the table representing position on the sky.
The values are determined by converting a sky position
whose coordinates are contained in existing columns.
The <col-id>
arguments give identifiers for
the two input coordinate columns
in the coordinate system named by
<insys>
, and
the <col-name>
arguments name
the two new columns,
which will be in the coordinate system named by
<outsys>
.
The <insys>
and <outsys>
coordinate system specifiers are one of
icrs
: ICRS (Hipparcos) (Right Ascension, Declination)fk5
: FK5 J2000.0 (Right Ascension, Declination)fk4
: FK4 B1950.0 (Right Ascension, Declination)galactic
: IAU 1958 Galactic (Longitude, Latitude)supergalactic
: de Vaucouleurs Supergalactic (Longitude, Latitude)ecliptic
: Ecliptic (Longitude, Latitude)The -inunit
and -outunit
flags
may be used to indicate the units of the existing coordinates
and the units for the new coordinates respectively;
use one of
degrees
, radians
or
sexagesimal
(may be abbreviated),
otherwise degrees will be assumed.
For sexagesimal, the two corresponding columns must be
string-valued in forms like hh:mm:ss.s and dd:mm:ss.s
respectively.
For certain conversions, the value specified by the
-epoch
flag is of significance.
Where significant its value defaults to 2000.0.
Syntax for the <expr>
, <col-id1>
and <col-id2>
arguments is described in the manual.
assert
Usage:
assert <expr>
Check that a boolean expression is true for each row.
If the expression <expr>
does not
evaluate true for any row of the table, execution terminates
with an error.
As long as no error occurs, the output table is identical
to the input one.
The exception generated by an assertion violation is of class
uk.ac.starlink.ttools.filter.AssertException
although that is not usually obvious if you are running from
the shell in the usual way.
Syntax for the <expr>
argument is described in Section 8.
badval
Usage:
badval <bad-val> <colid-list>
For each column specified in <colid-list>
any occurrence of the value <bad-val>
is replaced by a blank entry.
Syntax for the <colid-list>
argument is described in Section 5.3.
cache
Usage:
cache
Stores in memory or on disk a temporary copy of the table at this point in the pipeline. This can provide improvements in efficiency if there is an expensive step upstream and a step which requires more than one read of the data downstream. If you see an error like "Can't re-read data from stream" then adding this step near the start of the filters might help.
check
Usage:
check
Runs checks on the table at the indicated point in the processing pipeline. This is strictly a debugging measure, and may be time-consuming for large tables.
clearparams
Usage:
clearparams <pname> ...
Clears the value of one or more named parameters.
Each of the <pname>
values supplied may be
either a parameter name or a simple wildcard expression
matching parameter names. Currently the only wildcarding
is a "*
" to match any sequence of characters.
clearparams *
will clear all the parameters
in the table.
It is not an error to supply <pname>
s
which do not exist in the table - these have no effect.
colmeta
Usage:
colmeta [-name <name>] [-units <units>] [-ucd <ucd>] [-desc <descrip>] <colid-list>
Modifies the metadata of one or more columns.
Some or all of the name, units, ucd and description of
the column(s), identified by <colid-list>
can be set by using some or all of the listed flags.
Typically, <colid-list>
will simply be
the name of a single column.
Syntax for the <colid-list>
argument is described in Section 5.3.
delcols
Usage:
delcols <colid-list>
Delete the specified columns. The same column may harmlessly be specified more than once.
Syntax for the <colid-list>
argument is described in Section 5.3.
every
Usage:
every <step>
Include only every <step>
'th row in the
result, starting with the first row.
explodeall
Usage:
explodeall [-ifndim <ndim>] [-ifshape <dims>]
Replaces any columns which is an N-element arrays with N scalar columns. Only columns with fixed array sizes are affected. The action can be restricted to only columns of a certain shape using the flags.
If the -ifndim
flag is used, then only columns
of dimensionality <ndim>
will be exploded.
<ndim>
may be 1, 2, ....
If the -ifshape
flag is used, then only columns
with a specific shape will be exploded;
<dims>
is a space- or comma-separated list
of dimension extents, with the most rapidly-varying first,
e.g. '2 5
' to explode all 2 x 5 element array
columns.
explodecols
Usage:
explodecols <colid-list>
Takes a list of specified columns which represent N-element
arrays and replaces each one with N scalar columns.
Each of the columns specified by <colid-list>
must have a fixed-length array type,
though not all the arrays need to have the same number
of elements.
Syntax for the <colid-list>
argument is described in Section 5.3.
head
Usage:
head <nrows>
Include only the first <nrows>
rows of
the table.
If the table has fewer than <nrows>
rows
then it will be unchanged.
keepcols
Usage:
keepcols <colid-list>
Select the columns from the input table which will be
included in the output table.
The output table will include only those columns listed in
<colid-list>
, in that order.
The same column may be listed more than once,
in which case it will appear in the output table more than once.
Syntax for the <colid-list>
argument is described in Section 5.3.
meta
Usage:
meta [<item> ...]
Provides information about the metadata for each column. This filter turns the table sideways, so that each row of the output corresponds to a column of the input. The columns of the output table contain metadata items such as column name, units, UCD etc corresponding to each column of the input table.
By default the output table contains columns for the following items:
Index
: Position of column in tableName
: Column nameClass
: Data type of objects in columnShape
: Shape of array valuesUnits
: Unit stringDescription
: Description of data in the columnUCD
: Unified Content DescriptorHowever, the output may be customised by supplying
one or more <item>
headings.
These may be selected from the above as well as the following:
UCD_desc
: Textual description of UCDAny table parameters of the input table are propagated to the output one.
progress
Usage:
progress
Monitors progress by displaying the number of rows processed so far on the terminal (standard error). This number is updated every second or thereabouts; if all the processing is done in under a second you may not see any output. If the total number of rows in the table is known, an ASCII-art progress bar is updated, otherwise just the number of rows seen so far is written.
random
Usage:
random
Ensures that steps downstream see the table as random access. Only useful for debugging.
replacecol
Usage:
replacecol [-name <name>] [-units <units>] [-ucd <ucd>] [-desc <descrip>] <col-id> <expr>
Replaces the content of a column with the value of an
algebraic expression.
The old values are discarded in favour of the result of
evaluating <expr>
.
You can specify the metadata for the new column using the
-name
, -units
, -ucd
and -desc
flags; for any of these items which you
do not specify, they will take the values from the column
being replaced.
It is legal to reference the replaced column in the
expression,
so for example "replacecol pixsize pixsize*2
"
just multiplies the values in column pixsize
by 2.
Syntax for the <col-id>
and <expr>
arguments is described in the manual.
replaceval
Usage:
replaceval <old-val> <new-val> <colid-list>
For each column specified in <colid-list>
any instance of <old-val>
is replaced by
<new-val>
.
The value string 'null
' can be used for either
<old-value>
or <new-value>
to indicate a blank value
(but see also the badval
filter).
Syntax for the <colid-list>
argument is described in Section 5.3.
select
Usage:
select <expr>
Include in the output table only rows for which the
expression <expr>
evaluates to true.
<expr>
must be an expression which
evaluates to a boolean value (true/false).
Syntax for the <expr>
argument is described in Section 8.
sequential
Usage:
sequential
Ensures that steps downstream see the table as sequential access. Only useful for debugging.
setparam
Usage:
setparam [-type byte|short|int|long|float|double|boolean|string] [-desc <descrip>] [-unit <units>] [-ucd <ucd>] <pname> <pval>
Sets a named parameter in the table to a given value.
The parameter named <pname>
is set
to the value <pval>
.
By default the type of the parameter is determined automatically
(if it looks like an integer it's an integer etc)
but this can be overridden using the -type
flag.
The parameter description may be set using the
-desc
flag.
sort
Usage:
sort [-down] [-nullsfirst] <key-list>
Sorts the table according to the value of one or more
algebraic expressions.
The sort key expressions appear,
as separate (space-separated) words,
in <key-list>
; sorting is done on the
first expression first, but if that results in a tie then
the second one is used, and so on.
Each expression must evaluate to a type that
it makes sense to sort, for instance numeric.
If the -down
flag is used, the sort order is
descending rather than ascending.
Blank entries are by default considered to come at the end
of the collation sequence, but if the -nullsfirst
flag is given then they are considered to come at the start
instead.
Syntax for the <key-list>
argument is described in Section 8.
sorthead
Usage:
sorthead [-tail] [-down] [-nullsfirst] <nrows> <key-list>
Performs a sort on the table according to the value of
one or more algebraic expressions, retaining only
<nrows>
rows at the head
of the resulting sorted table.
The sort key expressions appear,
as separate (space-separated) words,
in <key-list>
; sorting is done on the
first expression first, but if that results in a tie then
the second one is used, and so on.
Each expression must evaluate to a type that
it makes sense to sort, for instance numeric.
If the -tail
flag is used, then the
last <nrows>
rows rather than the first
ones are retained.
If the -down
flag is used the sort order is
descending rather than ascending.
Blank entries are by default considered to come at the end
of the collation sequence, but if the -nullsfirst
flag is given then they are considered to come at the start
instead.
This filter is functionally equivalent to using
sort
followed by head
,
but it can be done in one pass and is usually cheaper
on memory and faster, as long as <nrows>
is significantly lower than the size of the table.
Syntax for the <key-list>
argument is described in Section 8.
stats
Usage:
stats [<item> ...]
Calculates statistics on the data in the table. This filter turns the table sideways, so that each row of the output corresponds to a column of the input. The columns of the output table contain statistical items such as mean, standard deviation etc corresponding to each column of the input table.
By default the output table contains columns for the following items:
Name
: Column nameMean
: AverageStDev
: Population Standard deviationMinimum
: Numeric minimumMaximum
: Numeric maximumNGood
: Number of non-blank cellsHowever, the output may be customised by supplying one or more
<item>
headings. These may be selected
from the above as well as the following:
NBad
: Number of blank cellsVariance
: Population VarianceSampStDev
: Sample Standard DeviationSampVariance
: Sample VarianceSkew
: Gamma 1 skewness measureKurtosis
: Gamma 2 peakedness measureSum
: Sum of valuesMinPos
: Row index of numeric minimumMaxPos
: Row index of numeric maximumCardinality
: Number of distinct values in column; values >100 ignoredMedian
: Middle value in sequenceQuartile1
: First quartileQuartile2
: Second quartileQuartile3
: Third quartileQ.25
: First quartileQ.625
: Fifth octileAny parameters of the input table are propagated to the output one.
Note that quantile calculations (including median and quartiles) can be expensive on memory. If you want to calculate quantiles for large tables, it may be wise to reduce the number of columns to only those you need the quantiles for earlier in the pipeline. No interpolation is performed when calculating quantiles.
tablename
Usage:
tablename <name>
Sets the table's name attribute to the given string.
tail
Usage:
tail <nrows>
Include only the last <nrows>
rows
of the table.
If the table has fewer than <nrows>
rows
then it will be unchanged.
transpose
Usage:
transpose [-namecol <col-id>]
Transposes the input table so that columns become rows
and vice versa.
The -namecol
flag can be used to specify a column
in the input table which will provide the column names for
the output table.
The first column of the output table will contain the
column names of the input table.
Syntax for the <col-id>
argument is described in Section 5.2.
uniq
Usage:
uniq [-count] [<colid-list>]
Eliminates adjacent rows which have the same values. If used with no arguments, then any row which has identical values to its predecessor is removed.
If the <colid-list>
parameter is given
then only the values in the specified columns must be equal
in order for the row to be removed.
If the -count
flag is given, then an additional
column with the name DupCount will be
prepended to the table giving a count of the number of duplicated
input rows represented by each output row. A unique row
has a DupCount value of 1.
Syntax for the <colid-list>
argument is described in Section 5.3.
If an argument is specified in the help text for a
command with the symbol <col-id>
it means you must give a string which identifies one of the
existing columns in a table.
There are three ways you can specify a column in this context:
-
').
It is usually matched case insensitively. If multiple columns
have the same name, the first one that matches is selected.
Tip: if counting which column has which index is giving you a
headache, running tpipe
with omode=meta
or
omode=stats
on the table may help.
ucd$<ucd-spec>
". Depending on the version of
UCD scheme used, UCDs can contain various punctuation marks such
as underscores, semicolons and dots; for the purpose of this syntax
these should all be represented as underscores ("_
").
So to identify a column which has the UCD "phot.mag;em.opt.R
",
you should use the identifier "ucd$phot_mag_em_opt_r
".
Matching is not case-sensitive. Futhermore, a trailing underscore
acts as a wildcard, so that the above column could also be referenced
using the identifier "ucd$phot_mag_
". If multiple
columns have UCDs which match the given identifer, the first one
will be used.
If an argument is specified in the help text for a command
with the symbol <colid-list>
it means you
must give a string which identifies a list of zero, one or more
of the existing columns in a table.
The string you specify is a separated into separate tokens by
whitespace, which means that you will normally
have to surround it in single or double quotes to ensure
that it is treated as a single argument and not several of them.
Each token in the <colid-list>
string may be one of
the following:
-
').
It is usually matched case insensitively. If multiple
columns have the same name, the first one that matches is selected.
Tip: if counting which column has which index is giving you a
headache, running tpipe
with omode=meta
or
omode=stats
on the table may help.
*
'
which matches any sequence of characters. To match an unknown
sequence at the start or end of the string an asterisk must be
given explicitly. Other than that, matching is usually
case insensitive. The order of the expanded list is the same
as the order in which the columns appear in the table.
Thus "col*
" will match columns named
col1
, Column2
and COL_1024
,
but not decOld
.
"*MAG*
" will match columns named
magnitude
, ABS_MAG_U
and JMAG
.
"*
" on its own
expands to a list of all the columns of the table in order.
Specifying a list which contains a given column more than once is not usually an error, but what effect it has depends on the function you are executing.
This section lists the output modes which can be used as
the value of the omode
parameter of
tpipe
and other commands.
Typically, having produced a result table by pipeline processing
an input one, you will write it out by specifying
omode=out
(or not using the omode
parameter at all -
out
is the default). However, you can do other things
such as calculate statistics, display metadata, etc. In some of
these cases, additional parameters are required. The different
output modes, with their associated parameters, are described in
the following subsections.
cgi
Usage:
omode=cgi ofmt=<out-format>
Writes a table to standard output in a way suitable for
use as output from a CGI (Common Gateway Interface) program.
This is very much like out
mode
but a short CGI header giving the MIME Content-Type
is prepended to the output
Additional parameters for this output mode are:
ofmt = <out-format>
[Default: votable
]
count
Usage:
omode=count
Counts the number of rows and columns and writes the result to standard output.
discard
Usage:
omode=discard
Reads all the data in the table in sequential mode
and discards it.
May be useful in conjunction with
the assert
filter.
meta
Usage:
omode=meta
Prints the table metadata to standard output. The name and type etc of each column is tabulated, and table parameters are also shown.
See the meta
filter for more flexible output of table metadata.
out
Usage:
omode=out out=<out-table> ofmt=<out-format>
Writes a new table.
Additional parameters for this output mode are:
out = <out-table>
[Default: -
]
ofmt = <out-format>
(auto)
"
(the default),
then the output filename will be
examined to try to guess what sort of file is required
usually by looking at the extension.
If it's not obvious from the filename what output format is
intended, an error will result.
[Default: (auto)
]
plastic
Usage:
omode=plastic transport=string|file client=<app-name>
Broadcasts the table to any registered Plastic-aware applications. PLASTIC, the PLatform for AStronomical Tool InterConnection, is a tool interoperability protocol. A Plastic hub must be running in order for this to work.
Additional parameters for this output mode are:
transport = string|file
string
:
VOTable serialized as a string and passed as a call parameter
(ivo://votech.org/votable/load
).
Not suitable for very large files.file
:
VOTable written to a temporary file and the filename passed as
a call parameter
(ivo://votech.org/votable/loadFromURL
).
The file ought to be deleted once it has been loaded.
Not suitable for inter-machine communication.null
) then a decision will
be taken based on the apparent size of the table.
client = <app-name>
stats
Usage:
omode=stats
Calculates and displays univariate statistics for each of the numeric columns in the table. The following entries are shown for each column as appropriate:
See the stats
filter
for more flexible statistical calculations.
topcat
Usage:
omode=topcat
Displays the output table directly in TOPCAT. If a TOPCAT instance (version 1.6 or later) is already running on the local host, the table will be opened in that, otherwise a new TOPCAT instance will be launched for display. The latter mode only works if the TOPCAT classes are on the class path.
A variety of mechanisms (e.g. PLASTIC and SOAP) are attempted to transfer the table, depending on what running instances of TOPCAT can be found. Depending on the transport mechanism used, there may be limits to the size of table which can be transmitted to the application in this way.
tosql
Usage:
omode=tosql protocol=<jdbc-protocol> host=<value> db=<db-name> dbtable=<table-name> write=create|dropcreate|append user=<username> password=<passwd>
Writes a new table to an SQL database.
You need the appropriate JDBC drivers and
-Djdcb.drivers
set as usual
(see Section 3.4).
Additional parameters for this output mode are:
protocol = <jdbc-protocol>
mysql
,
and for PostgreSQL's driver it is postgres
.
For other drivers, you may have to consult the driver
documentation.
host = <value>
[Default: localhost
]
db = <db-name>
dbtable = <table-name>
write = create|dropcreate|append
create
: Creates a new table before writing. It is an error if a table of the same name already exists.dropcreate
: Creates a new database table before writing. If a table of the same name already exists, it is dropped first.append
: Appends to an existing table. An error results if the named table has the wrong structure (number or types of columns) for the data being written.[Default: create
]
user = <username>
[Default: mbt
]
password = <passwd>
STILTS offers flexible and efficient facilities for crossmatching tables. Crossmatching is identifying different rows, which may be in the same or different tables, that refer to the same item. In an astronomical context such an item is usually, though not necessarily, an astronomical source or object. This operation corresponds to what in database terminology is called a join.
There are various complexities to specifying such a match. In the first place you have to define what is the condition that must be satisfied for two rows to be considered matching. In the second place you must decide what happens if, for a given row, more than one match can be found. Finally, you have to decide what to do having worked out what the matched rows are; the result will generally be presented as a new output table, but there are various choices about what columns and rows it will consist of. Some of these issues are discussed in this section, and others in the reference sections on the tools themselves in Appendix B.
Matching can in general be a computationally intensive process.
The algorithm used by the tmatch*
tasks in STILTS,
except in pathological cases, scales as O(N log(N)) or thereabouts,
where N is the total number of rows in all the tables being matched.
No preparation (such as sorting) is required on the tables prior to
invoking the matching operation.
It is reasonably fast; for instance an RA, Dec positional match
of two 105-row catalogues takes of the order of 60 seconds
on current (2005 laptop) hardware. Attempting matches with large tables can
lead to running out of memory; the calculation just mentioned required a
java heap size of around 200Mb (-Xmx200M
).
In the current release of STILTS the following tasks are provided for crossmatching between local tables:
tmatch2
tskymatch2
tmatch2
provided for convenience when the
full generality is not required.
tmatch1
tmatchn
tjoin
Determining whether one row represents the same item as another is done by comparing the values in certain of their columns to see if they are the same or similar. The most common astronomical case is to say that two rows match if their celestial coordinates (right ascension and declination) are within a given small radius of each other on the sky. There are other possibilities; for instance the coordinates to compare may be in a Cartesian space, or have a higher (or lower) dimensionality than two, or the match may be exact rather than within an error radius....
If you just need to match two tables according to sky position
with fixed errors you are recommended to use the simplified
tskymatch2
task.
For other cases, this section describes how to specify much more
flexible match criteria for use with
tmatch1
, tmatch2
or tmatchn
by setting the following parameters:
matcher
params
values*
tmatch2
you must specify both
values1
and values2
.
For example, suppose we wish to locate objects in two tables which are
within 3 arcseconds of each other on the sky. One table has columns
RA and DEC which give coordinates in degrees, and the other has columns
RArad and DECrad which give coordinates in radians. These are the
arguments which would be used to tell tmatch2
what the match
criteria are:
matcher=sky params=3 values1='RA DEC' values2='radiansToDegrees(RArad) radiansToDegrees(DECrad)'It is clearly important that corresponding values are comparable (in the same units) between the tables being matched, and in geometrically sensitive cases such as matching on the sky, it's important that they are the units expected by the matcher as well. To determine what those units are, either consult the roster below, or run the following command:
stilts tmatch2 help=matcherwhich will tell you about all the known matchers and their associated
params
and values*
parameters.
The following subsections list the basic matcher
types and the
requirements of their associated params
and
values*
parameters. The units of the required values
are given where significant.
sky
: Sky Matching
matcher=sky values*='<ra/degrees> <dec/degrees>' params='<max-error/arcsec>'
The sky
matcher compares positions on the celestial sphere
with a fixed error radius.
Rows are considered to match when the two (ra
, dec
)
positions are within max-error
arcseconds of each other
along a great circle.
In fact this matching is not restricted to equatorial coordinates -
the ra
and dec
parameters may represent
any longitude-like and latitude-like coordinates in degrees,
since the spherical geometry for the matching is unchanged under
such transformations.
skyerr
:
Sky Matching with Per-Object Errors
matcher=skyerr values*='<ra/degrees> <dec/degrees> <error/arcsec>' params='<max-error/arcsec>'
The skyerr
matcher compares positions on the celestial
sphere using error radii which can be different for each row.
Rows are considered to match when the separation between the
two ra
, dec
positions is smaller than
both the fixed max-error
value
and the sum of the two per-row error
values.
If either of the error
values is blank,
then any separation up to max-error
is considered a match.
According to these rules, you might decide to set max-error
to an arbitarily large number so that only the sum of error
s
will determine the actual match criteria.
However please don't do this, since max-error
also functions as a tuning parameter for the matching algorithm,
and ought to be reasonably close to the actual maximum acceptable
separation.
As with sky
matching,
other longitude/latitude coordinate pairs may be used in place of
right ascension and declination.
sky3d
: Spherical Polar Matching
matcher=sky3d values*='<ra/degrees> <dec/degrees> <distance>' params='<error/Units of distance>'
The sky3d
matcher compares positions in the volume of
the sky taking account of distance from the observer.
The position in three-dimensional space is calculated for each
row using the ra
, dec
and distance
as spherical polar coordinates, where distance
is the
distance from the observer along the line of sight.
Rows are considered to match when their positions in this space are
within error
units of each other.
The units of error
are the same as those of
distance
.
As with sky
matching,
other longitude/latitude coordinate pairs may be used in place of
right ascension and declination.
exact
: Exact Matching
matcher=exact values*='<matched-value>'
The exact
matcher compares arbitrary key values
for exact equality.
Rows are considered to match only if the values in their
matched-value
columns are exactly the same.
These values can be strings, numbers, or anything else.
A blank value never matches, not even with another blank one.
Since the params
parameter holds no values,
it does not have to be specified.
1d
, 2d
, ...:
Isotropic Cartesian Matching
matcher=1d values*='<x>' params='<error>'
matcher=2d values*='<x> <y>' params='<error>'
The 1d
matcher compares positions in 1-dimensional
Cartesian space.
Rows are considered to match if their x
column
values differ by no more than error
.
The 2d
matcher compares postions in 2-dimensional
Cartesian space.
Rows are considered to match if the difference in their
(x
,y
) positions reckoned using
Pythagoras is less than error
.
Matching in any number of Cartesian dimensions can be done by extending this syntax in the obvious way.
2d_anisotropic
, ...:
Anisotropic Cartesian Matching
matcher=2d_anisotropic values*='<x> <y>' params='<error-in-x> <error-in-y>'
The 2d_anisotropic
matcher compares positions in
2-dimensional Cartesian space using an anisotropic metric.
Rows are considered to match if their (x
,y
)
positions fall within an error ellipse with radii
error-in-x
,error-in-y
of each other.
This kind of match will typically be used for non-'spatial' spaces,
for instance (magnitude,redshift) space, in which the metrics along
different axes are not related to each other.
Matching in any number of dimensions of Cartesian space using an anisotropic metric can be done by extending this syntax in the obvious way.
In addition to the matching criteria listed in the previous subsections,
you can build your own by combining any of these.
To do this, take the two (or more)
matchers that you want to use, and separate their names with a
"+
" character. The values*
parameters
of the combined matcher should then hold the concatenation of the
values*
entries of the constituent matchers, and the
same for the params
parameter.
So for instance the matcher "sky+1d
" could be used
with the following syntax:
matcher=sky+1d values*='<ra/degrees> <dec/degrees> <x>' params='<max-error/arcsec> <error>'This would compare positions on the sky with an additional scalar constraint. Rows are considered to match if both their
ra
, dec
positions are within
max-error
arcseconds of each other along a great circle
(as for matcher=sky
)
and
their x
values differ by no more than error
(as for matcher=1d
).
This example might be used for instance to identify objects from two catalogues which are within a couple of arcseconds and also 0.5 blue magnitudes of each other. Rolling your own matchers in this way can give you very flexible match constraints.
The generic matching in STILTS is determined by specified
match criteria, as described in Section 6.1.
These criteria give conditions for whether two items (table rows)
count as matched with each other. In the case of a pair match,
as provided by tmatch2
,
it is clear how this is to be interpreted.
However, some of the matching tasks
(tmatchn
in group mode and
tmatch1
)
search for match groups which may have more than two members.
This section explains precisely how STILTS applies the pair-wise
matching criteria it is given to identifying multi-object groups.
In a multi-object match context, the matcher identifies a matched group as the largest possible group of objects in which each is linked by a pair match to any other object in the group. Formally, the set of matched groups is a set of disjoint graphs whose nodes are input table rows and whose edges are successful pair matches, where no successful pair match exists between nodes in different elements of that set. Thus the set has a minimal number of elements, and each of its elements is a matched group of maximal size. The important point to note is that for any particular pair in a matched group, there is no guarantee that the two objects match each other, only that you can hop from one to the other via pairs which do match.
So in the case of a multi-object sky match on a field which is very crowded compared to the specified error radius, it is quite possible for all the objects in the input table(s) to end up as part of the same large matching group. Results at or near this percolation threshold are (a) probably not useful and (b) likely to take a long time to run. Some care should therefore be exercised when specifying match criteria in multi-object match contexts.
As of Version 2.0 (October 2008), STILTS offers table plotting commands. These acquire a data point from each line of one or more input tables, perhaps influenced by the pipelining operations described in Section 5, and generate some kind of graphical plot from the result. At time of writing, the following plot types are available:
but see alsotcube
for generating
N-dimensional histograms as FITS files.
It is hoped to add more plot types in future releases.
The plotting commands offer considerable control over what is plotted and how it is represented, and thus unavoidably have rather a large number of parameters. When looking at the command documentation in Appendix B the Usage sections may look rather daunting. However, the discussion below and the Examples sections should help. Generating a simple plot is straightforward and can be done with only three or four parameters; if you have more complicated requirements for data selection or specific preferences for appearance then you can consult the documentation for the additional options.
As a simple example, if a file "cat.fits" contains the columns RMAG and BMAG for red and blue magnitudes, you can draw a two-dimensional colour-magnitude scatter plot with the command:
stilts plot2d in=cat.fits xdata=RMAG ydata=BMAG-RMAGSince an output file is not specified, the plot is shown on the screen for convenience. To send the output to a PNG file, do instead:
stilts plot2d in=cat.fits xdata=RMAG ydata=BMAG out=plot.png ofmt=pngin some cases (including the above), the
ofmt
parameter is not
required since STILTS may be able to guess the format from the output
file name.
Various other options for output and graphics formats are described in
Section 7.2 and Section 7.3
Some of the parameters use suffixes to define data sets and therefore behave a bit differently from the parameters elsewhere in STILTS - a discussion of these is given in the following subsection. Some other plotting-specific topics are also discussed below.
Some of the parameters for the plotting tasks behave a little bit
differently to other parameters in STILTS, in order to accommodate
related sets of values. If you look at the usage of one of the
plotting commands, for instance in Appendix B.4.1,
you will see that a number of the parameters have the
suffixes "N
" or "NS
".
These suffixes can be substituted with any convenient string to identify
parameters which relate to the same input datasets or subsets.
Specifically:
N
":
inN
parameter
must be given to identify the source of the data; any other parameters
with the same value of the N
suffix relate to that dataset.
A dataset here refers to a particular set of plot data from a
table; in most cases each input table corresponds to a different dataset,
though two datasets may correspond to different sets of columns from
the same table.
NS
":
N
.
At least the subsetNS
parameter must be given to
identify the expression by which the subset is defined;
any other parameters with the same value of the NS
suffix
relate to that subset.
Some examples will help to illustrate. The following will generate a Cartesian plot of catalogue position from a single dataset:
stilts plot2d in=gals.fits xdata=RA ydata=DECIn this case the
N
suffix is present on each of the parameters
in
, xdata
and ydata
, but is
equal to the empty string, hence invisible. This is perfectly legal,
and convenient when only a single table is in use.
If we wish to overplot two datasets however, the dataset suffixes
(or one of them at least) have to be made explicit so that different ones
can be used, for instance:
stilts plot2d in1=gals.fits xdata1=RA ydata1=DEC in2=stars.fits xdata2=RAJ2000 ydata2=DEJ2000The suffix values "
1
" and "2
" are quite arbitrary
and can be chosen as convenient, so the following would do exactly the
same as the previous example:
stilts plot2d in_GAL=gals.fits xdata_GAL=RA ydata_GAL=DEC in_STAR=stars.fits xdata_STAR=RAJ2000 ydata_STAR=DEJ2000The other parameters which have the
N
suffix apply only
to the matching dataset, so for instance the following:
stilts plot2d in1=gals.fits xdata1=RA ydata1=DEC txtlabel1=NGC_ID in2=stars.fits xdata2=RAJ2000 ydata2=DEJ2000would draw text labels adjacent to the points from only the gals.fits file giving the contents of its NGC_ID column.
The NS
suffix identifies distinct row subsets
within the same or different datasets. A subset is defined by supplying
a boolean inclusion expression (each row is included only if the
expression evaluates true for that row) as the value of a
subsetNS
parameter.
If, as in all the examples we have seen so far,
no subsetNS
parameter is supplied for a given dataset,
then it is treated as a special case, as if a single subset with a name
equal to the empty string (S=""
)
containing all rows has been specified.
So our earlier simple example:
stilts plot2d in=gals.fits xdata=RA ydata=DECis equivalent to
stilts plot2d in=gals.fits xdata=RA ydata=DEC subset=trueIf we wish to split the plotted points into two sets based on their R-B colours, we can write something like:
stilts plot2d in=gals.fits xdata=RA ydata=DEC subsetX='RMAG-BMAG>0' subsetY='RMAG-BMAG<=0'This will generate a plot with two subsets shown using different colours and/or plotting symbols. These colours and symbols are selected automatically. More control over the appearance can be exercised by setting values for some of the other parameters with
NS
suffixes, for instance
stilts plot2d in=gals.fits xdata=RA ydata=DEC subset_A='RMAG-BMAG>0' colour_A=blue subset_B='RMAG-BMAG<=0' colour_B=redAgain, the suffix strings can be chosen to have any value as convenient.
The dataset- and subset-specific parameters must be put together if there are multiple datasets with multiple subsets to plot simultaneously, for instance:
stilts plot2d in_1=gals.fits xdata_1=RA ydata_1=DEC subset_1_A='RMAG-BMAG>0' colour_1_A=blue subset_1_B='RMAG-BMAG<=0' colour_1_B=red in_2=stars.fits xdata_2=RAJ2000 ydata_2=DEJ2000 colour_2=green
Finally, it's not quite true that the suffixes chosen have no effect
on the plot; they may influence the order in which sets are plotted.
Markers drawn for sets plotted earlier may be obscured by the markers
drawn for sets plotted later, so this can affect the appearance of the plot.
If you want to control this, use the sequence
parameter.
For instance, to ensure that star data appears on top of galaxy data
in the plot, do the following:
stilts plot2d in_GAL=gals.fits xdata_GAL=RA ydata_GAL=DEC in_STAR=stars.fits xdata_STAR=RAJ2000 ydata_STAR=DEJ2000 sequence=_GAL,_STAR
More examples can be found in the Examples subsections of the individual plotting command descriptions in Appendix B.
The plots generated by the plotting commands can be used in various
different ways. One thing you might want to do is to write the
output to a file in a given graphics format
(out
);
another is to preview it directly on the screen
(swing
).
By default one or other of these will happen depending on whether you
specify an output file. However there are other possibilities;
these are listed in the following subsections.
Except for display to the screen, these modes should work happily
on a headless machine (one with no graphics display, as may be the case
for a web server).
When running headless, you may find it necessary to
set the java system property
"java.awt.headless
" to true
- see Section 3.3.
The default output mode is auto
,
which means that output is to a file if an output file is specified,
or to the screen if it is not. So often you don't need to specify the
omode
parameter explicitly.
swing
Usage:
omode=swing
Plot will be displayed in a window on the screen.
out
Usage:
omode=out out=<out-file> ofmt=png|gif|jpeg|eps|eps-gzip
Plot will be written to a file given by out
using the graphics format given by ofmt
.
cgi
Usage:
omode=cgi ofmt=png|gif|jpeg|eps|eps-gzip
Plot will be written in a way suitable for CGI use direct from a web server.
The output is in the graphics format given by ofmt
,
preceded by a suitable "Content-type" declaration.
discard
Usage:
omode=discard
Plot is drawn, but discarded. There is no output.
auto
Usage:
omode=auto [out=<out-file>]
Behaves as swing
or out
mode depending on presence of out
parameter
Several of the plot output modes write the plot in some graphics format or other. When selecting an output format it is important to understand the distinction between bitmapped and vector formats; basically bitmapped formats represent the image as a grid of finite-sized pixels while vector formats notionally draw smooth lines. Bitmapped formats are fine for a computer screen, but for high quality paper printouts you will want a vector format. You can convert from vector to bitmapped but not (usefully) in the other direction. There are a couple of subtleties to this distinction specific to STILTS graphical output as discussed below.
The following formats are the available values for the
ofmt
parameter of the various plot commands:
png
gif
jpeg
eps
eps-gzip
.
eps-gzip
eps
format above except that the output
is automatically compressed using the GZIP format as it is written.
Postscript compresses well (typically a factor of 5-10).
The intention is in future releases for STILTS to provide all the plot types and facilities which are available from TOPCAT. STILTS may additionally offer more detailed options for controlling plot appearance, for instance of font and colour selection and tick mark placement. At time of writing however, only the 2d scatter plot, 3d Cartesian scatter plot and histogram plot types are available, though these do include the most useful plot types and the most of the options from TOPCAT for these plot types are available from STILTS too.
As well as the advantage (in some contexts) of being able to generate plots in a scriptable fashion rather than from a graphical interactive interface, STILTS allows plots to be made from datasets of unlimited size. While TOPCAT has an effective limit of a few million rows, STILTS can stream data from tables to do its plotting, so a plot can be made representing an unlimited number of rows without large memory requirements. In some cases this might lead to plotting times which are a bit slower than TOPCAT - if this becomes an issue something may be done about it.
The tpipe
command allows you to use algebraic
expressions when making row selections and defining new synthetic
columns. They can also be used in defining the quantities to
match against in matching commands such as tmatch2
.
In both cases you are defining an expression which
has a value in each row as a function of the values in the existing
columns in that row.
This is a powerful feature which permits you to manipulate and select
table data in very flexible ways.
The syntax for entering these expressions is explained in this section.
What you write are actually expressions in the Java language, which are compiled into Java bytecode before evaluation. However, this does not mean that you need to be a Java programmer to write them. The syntax is pretty similar to C, but even if you've never programmed in C most simple things, and many complicated ones, are quite intutitive.
The following explanation gives some guidance and examples for writing these expressions. Unfortunately a complete tutorial on writing Java is beyond the scope of this document, but it should provide enough information for even a novice to write useful expressions.
The expressions that you can write are basically any function
of all the column values which apply
to a given row; the function result can then be used in
one of tpipe
's commands,
e.g. to define the per-row value of a new column
(addcol
, replacecol
)
make a row selection
(select
),
and some other places.
If the built-in operators and functions are not sufficient,
or it's unwieldy to express your function in one line of code,
it is possible to add new functions by writing your own classes -
see Section 8.7.3.
Note that since these algebraic expressions often contain spaces, you may need to enclose them in single or double quotes so that they don't get confused with other parts of the command string.
Note: if Java is running in an environment with certain security restrictions (a security manager which does not permit creation of custom class loaders) then algebraic expressions won't work at all. It's not particularly likely that security restrictions will be in place if you are running from the command line though.
To create a useful expression which can be evaluated for each row in a table, you will have to refer to cells in different columns of that row. You can do this in three ways:
ucd$<ucd-spec>
". Depending on the version of
UCD scheme used, UCDs can contain various punctuation marks such
as underscores, semicolons and dots; for the purpose of this syntax
these should all be represented as underscores ("_
").
So to identify a column which has the UCD "phot.mag;em.opt.R
",
you should use the identifier "ucd$phot_mag_em_opt_r
".
Matching is not case-sensitive. Futhermore, a trailing underscore
acts as a wildcard, so that the above column could also be referenced
using the identifier "ucd$phot_mag_
". If multiple
columns have UCDs which match the given identifer, the first one
will be used.
Note that the same syntax can be used for referencing table parameters (see the next section); columns take preference so if a column and a parameter both match the requested UCD, the column value will be used.
There is a special column whose name is "Index" and whose ID is "$0". The value of this is the same as the row number (the first row is 1).
The value of the variables so referenced will be a primitive
(boolean, byte, short, char, int, long, float, double) if the
column contains one of the corresponding types. Otherwise it will
be an Object of the type held by the column, for instance a String.
In practice this means: you can write the name of a column, and it will
evaluate to the numeric (or string) value that that column contains
in each row. You can then use this in normal algebraic expressions
such as "B_MAG-U_MAG
" as you'd expect.
Some tables have constant values associated with them; these may represent such things as the epoch at which observations were taken, the name of the catalogue, an angular resolution associated with all observations, or any number of other things. Such constants are known as table parameters (not to be confused with parameters passed to STILTS commands) and can be thought of as extra columns which have the same value for every row. The values of such parameters can be referenced in STILTS algebraic expressions as follows:
param$
.
ucd$
. Any punctuation marks in the
UCD should be replaced by underscores, and a trailing underscore
is interpreted as a wildcard. See Section 8.1 for
more discussion.
When no special steps are taken, if a null value (blank cell) is encountered in evaluating an expression (usually because one of the columns it relies on has a null value in the row in question) then the result of the expression is also null.
It is possible to exercise more control than this, but it
requires a little bit of care,
because the expressions work in terms of primitive values
(numeric or boolean ones) which don't in general have a defined null
value. The name "null
"
in expressions gives you the java null
reference, but this cannot be matched against a primitive value
or used as the return value of a primitive expression.
For most purposes, the following two tips should enable you to work with null values:
NULL_
"
(use upper case) to the column name or $ID. This
will yield a boolean value which is true if the column contains
a blank, and false otherwise.
NULL
"
(upper case). To return a null value from a non-numeric expression
(e.g. a String column) use the name "null
" (lower case).
Null values are often used in conjunction with the conditional
operator, "? :
"; the expression
test ? tval : fvalreturns the value
tval
if the boolean expression test
evaluates true, or fval
if test
evaluates false.
So for instance the following expression:
Vmag == -99 ? NULL : Vmagcan be used to define a new column which has the same value as the
Vmag
column for most values, but if Vmag
has the "magic" value -99 the new column will contain a blank.
The opposite trick (substituting a blank value with a magic one) can
be done like this:
NULL_Vmag ? -99 : VmagSome more examples are given in Section 8.6.
The operators are pretty much the same as in the C language. The common ones are:
+
(add)
-
(subtract)
*
(multiply)
/
(divide)
%
(modulus)
!
(not)
&&
(and)
||
(or)
^
(exclusive-or)
==
(numeric identity)
!=
(numeric non-identity)
<
(less than)
>
(greater than)
<=
(less than or equal)
>=
(greater than or equal)
(byte)
(numeric -> signed byte)
(short)
(numeric -> 2-byte integer)
(int)
(numeric -> 4-byte integer)
(long)
(numeric -> 8-byte integer)
(float)
(numeric -> 4-type floating point)
(double)
(numeric -> 8-byte floating point)
+
(string concatenation)
[]
(array dereferencing)
?:
(conditional switch)
instanceof
(class membership)
Many functions are available for use within your expressions, covering standard mathematical and trigonometric functions, arithmetic utility functions, type conversions, and some more specialised astronomical ones. You can use them in just the way you'd expect, by using the function name (unlike column names, this is case-sensitive) followed by comma-separated arguments in brackets, so
max(IMAG,JMAG)will give you the larger of the values in the columns IMAG and JMAG, and so on.
The functions available for use by default
are listed by class in the following subsections
with their arguments and short descriptions.
The funcs
command provides another
way to browse these function descriptions online.
Functions for angle transformations and manipulations. In particular, methods for translating between radians and HH:MM:SS.S or DDD:MM:SS.S type sexagesimal representations are provided.
DEGREE
HOUR
ARC_MINUTE
ARC_SECOND
radiansToDms( rad )
rad
(floating point): angle in radiansrad
radiansToDms( rad, secFig )
rad
(floating point): angle in radianssecFig
(integer): number of decimal places in the seconds fieldrad
radiansToHms( rad )
rad
(floating point): angle in radiansrad
radiansToHms( rad, secFig )
rad
(floating point): angle in radianssecFig
(integer): number of decimal places in the seconds fieldrad
dmsToRadians( dms )
dm[s]
, or some others.
Additional spaces and leading +/- are permitted.dms
(String): formatted DMS stringdms
hmsToRadians( hms )
hm[s]
, or some others.
Additional spaces and leading +/- are permitted.hms
(String): formatted HMS stringhms
dmsToRadians( deg, min, sec )
In conversions of this type, one has to be careful to get the
sign right in converting angles which are between 0 and -1 degrees.
This routine uses the sign bit of the deg
argument,
taking care to distinguish between +0 and -0 (their internal
representations are different for floating point values).
It is illegal for the min
or sec
arguments
to be negative.
deg
(floating point): degrees part of anglemin
(floating point): minutes part of anglesec
(floating point): seconds part of anglehmsToRadians( hour, min, sec )
In conversions of this type, one has to be careful to get the
sign right in converting angles which are between 0 and -1 hours.
This routine uses the sign bit of the hour
argument,
taking care to distinguish between +0 and -0 (their internal
representations are different for floating point values).
hour
(floating point): degrees part of anglemin
(floating point): minutes part of anglesec
(floating point): seconds part of angleskyDistance( ra1, dec1, ra2, dec2 )
ra1
(floating point): right ascension of point 1 in radiansdec1
(floating point): declination of point 1 in radiansra2
(floating point): right ascension of point 2 in radiansdec2
(floating point): declination of point 2 in radiansskyDistanceDegrees( ra1, dec1, ra2, dec2 )
ra1
(floating point): right ascension of point 1 in degreesdec1
(floating point): declination of point 1 in degreesra2
(floating point): right ascension of point 2 in degreesdec2
(floating point): declination of point 2 in degreeshoursToRadians( hours )
hours
(floating point): angle in hoursdegreesToRadians( deg )
deg
(floating point): angle in degreesradiansToDegrees( rad )
rad
(floating point): angle in radiansraFK4toFK5( raFK4, decFK4 )
raFK4
(floating point): right ascension in B1950.0 FK4 system (radians)decFK4
(floating point): declination in B1950.0 FK4 system (radians)decFK4toFK5( raFK4, decFK4 )
raFK4
(floating point): right ascension in B1950.0 FK4 system (radians)decFK4
(floating point): declination in B1950.0 FK4 system (radians)raFK5toFK4( raFK5, decFK5 )
raFK5
(floating point): right ascension in J2000.0 FK5 system (radians)decFK5
(floating point): declination in J2000.0 FK5 system (radians)decFK5toFK4( raFK5, decFK5 )
raFK5
(floating point): right ascension in J2000.0 FK5 system (radians)decFK5
(floating point): declination in J2000.0 FK5 system (radians)raFK4toFK5( raFK4, decFK4, bepoch )
bepoch
parameter is the epoch at which the position in
the FK4 frame was determined.raFK4
(floating point): right ascension in B1950.0 FK4 system (radians)decFK4
(floating point): declination in B1950.0 FK4 system (radians)bepoch
(floating point): Besselian epochdecFK4toFK5( raFK4, decFK4, bepoch )
bepoch
parameter is the epoch at which the position in
the FK4 frame was determined.raFK4
(floating point): right ascension in B1950.0 FK4 system (radians)decFK4
(floating point): declination in B1950.0 FK4 system (radians)bepoch
(floating point): Besselian epochraFK5toFK4( raFK5, decFK5, bepoch )
raFK5
(floating point): right ascension in J2000.0 FK5 system (radians)decFK5
(floating point): declination in J2000.0 FK5 system (radians)bepoch
(floating point): Besselian epochdecFK5toFK4( raFK5, decFK5, bepoch )
raFK5
(floating point): right ascension in J2000.0 FK5 system (radians)decFK5
(floating point): declination in J2000.0 FK5 system (radians)bepoch
(floating point): Besselian epochStandard arithmetic functions including things like rounding, sign manipulation, and maximum/minimum functions.
roundUp( x )
x
(floating point): a value.x
rounded uproundDown( x )
x
(floating point): a valuex
rounded downround( x )
x
(floating point): a floating point value.x
rounded to the nearest integerroundDecimal( x, dp )
float
(32-bit floating point value),
so this is only suitable for relatively low-precision values.
It's intended for truncating the number of apparent significant
figures represented by a value which you know has been obtained
by combining other values of limited precision.
For more control, see the functions in the Formats
class.x
(floating point): a floating point valuedp
(integer): number of decimal places (digits after the decimal point)
to retainx
but with a
limited apparent precisionabs( x )
x
(integer): the argument whose absolute value is to be determinedabs( x )
x
(floating point): the argument whose absolute value is to be determinedmax( a, b )
a
(integer): an argument.b
(integer): another argument.a
and b
.max( a, b )
a
(floating point): an argument.b
(floating point): another argument.a
and b
.maxReal( a, b )
a
(floating point): an argumentb
(floating point): another argumenta
and b
min( a, b )
a
(integer): an argument.b
(integer): another argument.a
and b
.min( a, b )
a
(floating point): an argument.b
(floating point): another argument.a
and b
.minReal( a, b )
a
(floating point): an argumentb
(floating point): another argumenta
and b
Functions for conversion between flux and magnitude values. Functions are provided for conversion between flux in Janskys and AB magnitudes.
Some constants for approximate conversions between different magnitude scales are also provided:
JOHNSON_AB_*
, for Johnson <-> AB magnitude
conversions
(http://www.astro.utoronto.ca/~patton/astro/mags.html,
citing Frei and Gunn 1995).VEGA_AB_*
, for Vega <-> AB magnitude
conversions
(Blanton et al., Astronomical Journal 127, 2562-2578 (2005),
eqs.(5)).
JOHNSON_AB_V
JOHNSON_AB_V
.JOHNSON_AB_B
JOHNSON_AB_B
.JOHNSON_AB_Bj
JOHNSON_AB_Bj
.JOHNSON_AB_R
JOHNSON_AB_R
.JOHNSON_AB_I
JOHNSON_AB_I
.JOHNSON_AB_g
JOHNSON_AB_g
.JOHNSON_AB_r
JOHNSON_AB_r
.JOHNSON_AB_i
JOHNSON_AB_i
.JOHNSON_AB_Rc
JOHNSON_AB_Rc
.JOHNSON_AB_Ic
JOHNSON_AB_Ic
.JOHNSON_AB_uPrime
JOHNSON_AB_uPrime
=u'AB.JOHNSON_AB_gPrime
JOHNSON_AB_gPrime
=g'AB.JOHNSON_AB_rPrime
JOHNSON_AB_rPrime
=r'AB.JOHNSON_AB_iPrime
JOHNSON_AB_iPrime
=i'AB.JOHNSON_AB_zPrime
JOHNSON_AB_zPrime
=z'AB.VEGA_AB_J
VEGA_AB_J
.VEGA_AB_H
VEGA_AB_H
.VEGA_AB_K
VEGA_AB_K
.abToJansky( magAB )
F/Jy=10(23-(AB+48.6)/2.5)
magAB
(floating point): AB magnitude valuejanskyToAb( fluxJansky )
AB=2.5*(23-log10(F/Jy))-48.6
fluxJansky
(floating point): flux in JanskyluminosityToFlux( lumin, dist )
F=lumin/(4 x Pi x dist2)
lumin
(floating point): luminositydist
(floating point): luminosity distancefluxToLuminosity( flux, dist )
lumin=(4 x Pi x dist2) F
flux
(floating point): fluxdist
(floating point): luminosity distanceFunctions which operate on array-valued cells. You can only use these functions on values which are already arrays. In most cases that means on values in table columns which are declared as array-valued. FITS and VOTable tables can have columns which contain array values, but other formats such as CSV cannot.
sum( array )
array
is not a numeric array, null
is returned.array
(Object): array of numbersarray
mean( array )
array
is not a numeric array, null
is returned.array
(Object): array of numbersarray
minimum( array )
array
is not a numeric array, null
is returned.array
(Object): array of numbersarray
maximum( array )
array
is not a numeric array, null
is returned.array
(Object): array of numbersarray
size( array )
array
is not an array, zero is returned.array
(Object): arrayarray
Pixel tiling functions for the celestial sphere.
htmIndex( level, ra, dec )
level
(integer): HTM levelra
(floating point): right ascension in degreesdec
(floating point): declination in degreeshealpixNestIndex( k, ra, dec )
k
(integer): resolution parameter - log to base 2 of nsidera
(floating point): right ascension in degreesdec
(floating point): declination in degreeshealpixRingIndex( k, ra, dec )
k
(integer): resolution parameter - log to base 2 of nsidera
(floating point): right ascension in degreesdec
(floating point): declination in degreeshealpixK( pixelsize )
k
value is the logarithm to base 2 of the
Nside parameter.pixelsize
(floating point): pixel size in degreesk
healpixResolution( k )
k
This k
value is the logarithm to base 2 of the
Nside parameter.k
(integer): HEALPix resolution parameter k
htmLevel( pixelsize )
level
parameter suitable for a given
pixel size.pixelsize
(floating point): required resolution in degreeshtmResolution( level )
level
(integer): HTM depthString manipulation and query functions.
concat( s1, s2 )
s1+s2
, but blank values can sometimes appear as
the string "null
" if you do it like that.s1
(String): first strings2
(String): second strings1
followed by s2
concat( s1, s2, s3 )
s1+s2+s3
, but blank values can sometimes appear as
the string "null
" if you do it like that.s1
(String): first strings2
(String): second strings3
(String): third strings1
followed by s2
followed by s3
concat( s1, s2, s3, s4 )
s1+s2+s3+s4
,
but blank values can sometimes appear as
the string "null
" if you do it like that.s1
(String): first strings2
(String): second strings3
(String): third strings4
(String): fourth strings1
followed by s2
followed by s3
followed by s4
equals( s1, s2 )
s1==s2
,
which can (for technical reasons) return false even if the
strings are the same.s1
(String): first strings2
(String): second stringequalsIgnoreCase( s1, s2 )
s1
(String): first strings2
(String): second stringstartsWith( whole, start )
whole
(String): the string to teststart
(String): the sequence that may appear at the start of
whole
whole
are
the same as start
endsWith( whole, end )
whole
(String): the string to testend
(String): the sequence that may appear at the end of
whole
whole
are
the same as end
contains( whole, sub )
whole
(String): the string to testsub
(String): the sequence that may appear within whole
sub
appears within
whole
length( str )
str
(String): stringstr
matches( str, regex )
str
(String): string to testregex
(String): regular expression stringregex
matches str
anywherematchGroup( str, regex )
str
(String): string to match againstregex
(String): regular expression containing a grouped sectionregex
didn't match str
)replaceFirst( str, regex, replacement )
str
(String): string to manipulateregex
(String): regular expression to match in str
replacement
(String): replacement stringstr
, but with the first match (if any) of
regex
replaced by replacement
replaceAll( str, regex, replacement )
str
(String): string to manipulateregex
(String): regular expression to match in str
replacement
(String): replacement stringstr
, but with all matches of
regex
replaced by replacement
substring( str, startIndex )
str
(String): the input stringstartIndex
(integer): the beginning index, inclusivestr
, omitting the first
startIndex
characterssubstring( str, startIndex, endIndex )
startIndex
and continues to the character at index endIndex-1
Thus the length of the substring is endIndex-startIndex
.str
(String): the input stringstartIndex
(integer): the beginning index, inclusiveendIndex
(integer): the end index, inclusivestr
toUpperCase( str )
str
(String): input stringstr
toLowerCase( str )
str
(String): input stringstr
trim( str )
str
(String): input stringpadWithZeros( value, ndigit )
value
(long integer): numeric value to padndigit
(integer): the number of digits in the resulting stringvalue
with
at least ndigit
charactersFunctions for formatting numeric values.
formatDecimal( value, dp )
value
(floating point): value to formatdp
(integer): number of decimal places (digits after the decmal point)formatDecimalLocal( value, dp )
formatDecimal
function.value
(floating point): value to formatdp
(integer): number of decimal places (digits after the decmal point)formatDecimal( value, format )
format
string is as defined by Java's
java.text.DecimalFormat
class.value
(floating point): value to formatformat
(String): format specifierformatDecimalLocal( value, format )
formatDecimal
function.value
(floating point): value to formatformat
(String): format specifierStandard mathematical and trigonometric functions.
E
PI
RANDOM
sin( theta )
theta
(floating point): an angle, in radians.cos( theta )
theta
(floating point): an angle, in radians.tan( theta )
theta
(floating point): an angle, in radians.asin( x )
x
(floating point): the value whose arc sine is to be returned.acos( x )
x
(floating point): the value whose arc cosine is to be returned.atan( x )
x
(floating point): the value whose arc tangent is to be returned.exp( x )
x
(floating point): the exponent to raise e to.log10( x )
x
(floating point): argumentln( x )
x
(floating point): argumentsqrt( x )
x
(floating point): a value.x
.
If the argument is NaN or less than zero, the result is NaN.atan2( y, x )
x
,y
)
to polar (r
,theta
).
This method computes the phase
theta
by computing an arc tangent
of y/x
in the range of -pi to pi.y
(floating point): the ordinate coordinatex
(floating point): the abscissa coordinatetheta
component (radians) of the point
(r
,theta
)
in polar coordinates that corresponds to the point
(x
,y
) in Cartesian coordinates.pow( a, b )
a
(floating point): the base.b
(floating point): the exponent.ab
.sinh( x )
x
(floating point): parametercosh( x )
x
(floating point): parametertanh( x )
x
(floating point): parameterasinh( x )
x
(floating point): parameteracosh( x )
x
(floating point): parameteratanh( x )
x
(floating point): parameterFunctions for conversion of time values between various forms. The forms used are
yyyy-mm-ddThh:mm:ss.s
, where the T
is a literal character (a space character may be used instead).
Based on UTC.
Therefore midday on the 25th of October 2004 is
2004-10-25T12:00:00
in ISO 8601 format,
53303.5 as an MJD value,
2004.81588 as a Julian Epoch and
2004.81726 as a Besselian Epoch.
Currently this implementation cannot be relied upon to better than a millisecond.
isoToMjd( isoDate )
isoDate
argument is
yyyy-mm-ddThh:mm:ss.s
, though some deviations
from this form are permitted:
T
' which separates date from time
can be replaced by a spaceZ
' (which indicates UTC) may be appended
to the time1994-12-21T14:18:23.2
",
"1968-01-14
", and
"2112-05-25 16:45Z
".isoDate
(String): date in ISO 8601 formatisoDate
dateToMjd( year, month, day, hour, min, sec )
year
(integer): year ADmonth
(integer): index of month; January is 1, December is 12day
(integer): day of month (the first day is 1)hour
(integer): hour (0-23)min
(integer): minute (0-59)sec
(floating point): second (0<=sec<60)dateToMjd( year, month, day )
year
(integer): year ADmonth
(integer): index of month; January is 1, December is 12day
(integer): day of month (the first day is 1)decYearToMjd( decYear )
decYear
(floating point): decimal yearmjdToIso( mjd )
yyyy-mm-ddThh:mm:ss
.mjd
(floating point): modified Julian datemjd
mjdToDate( mjd )
yyyy-mm-dd
.mjd
(floating point): modified Julian datemjd
mjdToTime( mjd )
hh:mm:ss
.mjd
(floating point): modified Julian datemjd
mjdToDecYear( mjd )
mjd
(floating point): modified Julian DateformatMjd( mjd, format )
java.text.SimpleDateFormat
class.
The default output corresponds to the string
"yyyy-MM-dd'T'HH:mm:ss
"mjd
(floating point): modified Julian dateformat
(String): formatting pattternmjd
mjdToJulian( mjd )
mjd
(floating point): modified Julian datejulianToMjd( julianEpoch )
julianEpoch
(floating point): Julian epochmjdToBesselian( mjd )
mjd
(floating point): modified Julian datebesselianToMjd( besselianEpoch )
besselianEpoch
(floating point): Besselian epochunixMillisToMjd( unixMillis )
unixMillis
(long integer): milliseconds since the Unix epochmjdToUnixMillis( mjd )
mjd
(floating point): modified Julian dateFunctions for converting between different measures of cosmological distance.
The following parameters are used:
For a flat universe, omegaM
+omegaLambda
=1
The terms and formulae used here are taken from the paper by D.W.Hogg, Distance measures in cosmology, astro-ph/9905116 v4 (2000).
SPEED_OF_LIGHT
METRE_PER_PARSEC
SEC_PER_YEAR
MpcToM( distMpc )
distMpc
(floating point): distance in MpcmToMpc( distM )
distM
(floating point): distance in mzToDist( z )
Warning: this makes some reasonable assumptions about the cosmology and returns the luminosity distance. It is only intended for approximate use. If you care about the details, use one of the more specific functions here.
z
(floating point): redshiftzToAge( z )
Warning: this makes some reasonable assumptions about the cosmology. It is only intended for approximate use. If you care about the details use one of the more specific functions here.
z
(floating point): redshiftz
in GyrcomovingDistanceL( z, H0, omegaM, omegaLambda )
z
(floating point): redshiftH0
(floating point): Hubble constant in km/sec/MpcomegaM
(floating point): density ratio of the universeomegaLambda
(floating point): normalised cosmological constantcomovingDistanceT( z, H0, omegaM, omegaLambda )
z
(floating point): redshiftH0
(floating point): Hubble constant in km/sec/MpcomegaM
(floating point): density ratio of the universeomegaLambda
(floating point): normalised cosmological constantangularDiameterDistance( z, H0, omegaM, omegaLambda )
z
(floating point): redshiftH0
(floating point): Hubble constant in km/sec/MpcomegaM
(floating point): density ratio of the universeomegaLambda
(floating point): normalised cosmological constantluminosityDistance( z, H0, omegaM, omegaLambda )
z
(floating point): redshiftH0
(floating point): Hubble constant in km/sec/MpcomegaM
(floating point): density ratio of the universeomegaLambda
(floating point): normalised cosmological constantlookbackTime( z, H0, omegaM, omegaLambda )
z
were emitted.z
(floating point): redshiftH0
(floating point): Hubble constant in km/sec/MpcomegaM
(floating point): density ratio of the universeomegaLambda
(floating point): normalised cosmological constantcomovingVolume( z, H0, omegaM, omegaLambda )
z
.z
(floating point): redshiftH0
(floating point): Hubble constant in km/sec/MpcomegaM
(floating point): density ratio of the universeomegaLambda
(floating point): normalised cosmological constantFunctions for converting between strings and numeric values.
toString( value )
value
(floating point): numeric valuevalue
parseByte( str )
str
(String): string containing numeric representationstr
parseShort( str )
str
(String): string containing numeric representationstr
parseInt( str )
str
(String): string containing numeric representationstr
parseLong( str )
str
(String): string containing numeric representationstr
parseFloat( str )
str
(String): string containing numeric representationstr
parseDouble( str )
str
(String): string containing numeric representationstr
toByte( value )
value
(floating point): numeric value for conversionvalue
converted to type bytetoShort( value )
value
(floating point): numeric value for conversionvalue
converted to type shorttoInteger( value )
value
(floating point): numeric value for conversionvalue
converted to type inttoLong( value )
value
(floating point): numeric value for conversionvalue
converted to type longtoFloat( value )
value
(floating point): numeric value for conversionvalue
converted to type floattoDouble( value )
value
(floating point): numeric value for conversionvalue
converted to type doubletoHex( value )
value
(long integer): integer valuevalue
fromHex( hexVal )
hexVal
(String): hexadecimal representation of valuehexVal
Here are some examples for defining new columns;
the expressions below could appear as the <expr>
in a
tpipe
addcol
or sortexpr
command).
(first + second) * 0.5
sqrt(variance)
radiansToDegrees(DEC_radians) degreesToRadians(RA_degrees)
parseInt($12) parseDouble(ident)
toString(index)
toShort(obs_type) toDouble(range)or
(short) obs_type (double) range
hmsToRadians(RA1950) dmsToRadians(decDeg,decMin,decSec)
radiansToDms($3) radiansToHms(RA,2)
min(1000, max(value, 0))
jmag == 9999 ? NULL : jmag
NULL_jmag ? 9999 : jmag
psfCounts[2]
tpipe
select
command)
RA > 100 && RA < 120 && Dec > 75 && Dec < 85
$2*$2 + $3*$3 < 1 skyDistance(ra0,dec0,degreesToRadians(RA),degreesToRadians(DEC))<15*ARC_MINUTE
index <= 100(though you could use
tpipe cmd='head 100'
instead)index % 10 == 0(though you could use
tpipe cmd='every 10'
instead)equals(SECTOR, "ZZ9 Plural Z Alpha") equalsIgnoreCase(SECTOR, "zz9 plural z alpha") startsWith(SECTOR, "ZZ") contains(ph_qual, "U")
matches(SECTOR, "[XYZ] Alpha")
! NULL_ellipticity
This section contains some notes on getting the most out of the algebraic expressions facility. If you're not a Java programmer, some of the following may be a bit daunting - read on at your own risk!
This note provides a bit more detail for Java programmers on what is going on here; it describes how the use of functions in STILTS algebraic expressions relates to normal Java code.
The expressions which you write are compiled to Java bytecode
when you enter them (if there is a 'compilation error' it will be
reported straight away). The functions listed in the previous subsections
are all the public static
methods of the classes which
are made available by default. The classes listed are all in the
package uk.ac.starlink.ttools.func
.
However, the public static methods are all imported into an anonymous
namespace for bytecode compilation, so that you write
(sqrt(x,y)
and not Maths.sqrt(x,y)
.
The same happens to other classes that are imported (which can be
in any package or none) - their public
static methods all go into the anonymous namespace. Thus, method
name clashes are a possibility.
This cleverness is all made possible by the rather wonderful JEL.
There is another category of functions which can be used apart from those listed in Section 8.5. These are called, in Java/object-oriented parlance, "instance methods" and represent functions that can be executed on an object.
It is possible to invoke any of its public
instance methods on any object
(though not on primitive values - numeric and boolean ones).
The syntax is that you place a "." followed by the method invocation
after the object you want to invoke the method on,
hence NAME.substring(3)
instead of substring(NAME,3)
.
If you know what you're doing, feel free to go ahead and do this.
However, most of the instance methods you're likely to want to use
have equivalents in the normal functions listed in the previous section,
so unless you're a Java programmer or feeling adventurous,
you may be best off ignoring this feature.
The functions provided by default for use with algebraic expressions, while powerful, may not provide all the operations you need. For this reason, it is possible to write your own extensions to the expression language. In this way you can specify abritrarily complicated functions. Note however that this will only allow you to define new columns or subsets where each cell is a function only of the other cells in the same row - it will not allow values in one row to be functions of values in another.
In order to do this, you have to write and compile a (probably short) program in the Java language. A full discussion of how to go about this is beyond the scope of this document, so if you are new to Java and/or programming you may need to find a friendly local programmer to assist (or mail the author). The following explanation is aimed at Java programmers, but may not be incomprehensible to non-specialists.
The steps you need to follow are:
jel.classes
system property (colon-separated if there are several)
as described in Section 3.3
Any public static methods defined in the classes thus specified will then be available for use. They should be defined to take and return the relevant primitive or Object types for the function required. For instance a class written as follows would define a three-value average:
public class AuxFuncs { public static double average3( double x, double y, double z ) { return ( x + y + z ) / 3.0; } }and the command
stilts tpipe cmd='addcol AVERAGE "average3($1,$2,$3)"'would add a new column named AVERAGE giving the average of the first three existing columns. Exactly how you would build this is dependent on your system, but it might involve doing something like the following:
AuxFuncs.java
containing the above codejavac AuxFuncs.java
"tpipe
using the flags
"stilts -classpath . -Djel.classes=AuxFuncs tpipe
"This section lists the commands available broken down by the category of function they provide. Some commands appear in more than one category. Detailed descriptions and examples for each command can be found in Appendix B.
tmatch1
:
Performs a crossmatch internal to a single table
tmatch2
:
Crossmatches 2 tables using flexible criteria
tmatchn
:
Crossmatches multiple tables using flexible criteria
tskymatch2
:
Crossmatches 2 tables on sky position
coneskymatch
:
Crossmatches table on sky position against Cone Search service
sqlskymatch
:
Crossmatches table on sky position against SQL table
regquery
:
Queries the VO registry
coneskymatch
:
Crossmatches table on sky position against Cone Search service
sqlclient
:
Executes SQL statements
sqlupdate
:
Updates values in an SQL table
sqlskymatch
:
Crossmatches table on sky position against SQL table
This appendix provides the reference documentation for the commands in the package. For each one a description of its purpose, a list of its command-line arguments, and some examples are given.
calc
: Evaluates expressions
calc
is a very simple utility for evaluating expressions.
It uses the same expression evaluator as is used in tpipe
and the other generic table tasks for things like creating new columns,
so it can be used as a quick test to see what expressions work,
or in order to evaluate expressions using the various algebraic
functions documented in Section 8.5.
Since usually no table is involved, you can't refer to column names in
the expressions.
It has one mandatory parameter, the expression to evaluate, and writes the
result to the screen.
The usage of calc
is
stilts <stilts-flags> calc table=<table> [expression=]<expr>If you don't have the
stilts
script installed,
write "java -jar stilts.jar
" instead of
"stilts
" - see Section 3.
The available <stilts-flags>
are listed
in Section 2.1.
Parameter values are assigned on the command line as explained in Section 2.3. They are as follows:
expression = <expr>
table = <table>
expression
is evaluated.
This parameter is optional, and will usually not be required;
its only purpose is to allow use of constant expressions
(table parameters) associated with the table.
These can be referenced using identifiers of the form
param$*
or
ucd$*
-
see Section 8.2 for more detail.
Here are some examples of using calc
:
stilts calc 1+2
stilts calc 'isoToMjd("2005-12-25T00:00:00")'
stilts calc 'param$author' table=catalogue.xml
coneskymatch
: Crossmatches table on sky position against Cone Search service
coneskymatch
is a utility which performs a
cone search
(cone search is an IVOA standard protocol)
query for each row of an input table and concatenates the results
of all these queries together into one big output table.
You give the service URL
for the cone search server you wish
to use, and expressions (usually column names)
defining how to get the search parameters
(sky position and search radius) for each row of the input table.
The program then goes through the input table and dispatches
a cone search query to the server for each row.
For each of these queries the service should respond with a
VOTable containing the objects it knows about in the specified region;
hopefully the columns will be the same or very similar for
all the different queries since they are using the same service.
The response tables are stitched together top-to-bottom
(in the same way as tcat
)
and the result is output.
The effect of this is like doing a positional crossmatch where one of the catalogues is local and the other is remote and provides a cone search service. Because of both the network communication and the naive algorithm however, it is only suitable if the local catalogue has a rather small number of rows.
The parallel
parameter allows you to perform multiple
cone searches concurrently, so that instead of completing the
first cone search, then the second, then the third,
the program can be executing a number of them at once.
This can speed up operation considerably, especially
in the face of network latency, but beware that submitting a very large
number of queries simultaneously to the same server may overload it,
resulting in some combination of failed queries, ultimately slower runtimes,
and unpopularity with cone search server admins.
Best to start with a low parallelism and cautiously increase it to
see whether there are gains in performance.
Note that when running, coneskymatch
can generate a lot
of WARNING messages. Most of these are complaining about badly formed
VOTables being returned from the cone search services. STILTS does its
best to work out what the service responses mean in this case,
and usually makes a good enough job of it.
Note: this task was known as multicone
in its experimental
form in STILTS v1.2 and v1.3.
The usage of coneskymatch
is
stilts <stilts-flags> coneskymatch ifmt=<in-format> istream=true|false icmd=<cmds> ocmd=<cmds> omode=<out-mode> <mode-args> out=<out-table> ofmt=<out-format> ra=<expr> dec=<expr> sr=<expr> find=best|all copycols=<colid-list> scorecol=<col-name> parallel=<n> erract=abort|ignore|retry|retry<n> ostream=true|false fixcols=none|dups|all suffix0=<label> suffix1=<label> serviceurl=<value> verb=1|2|3 emptyok=true|false [in=]<table>If you don't have the
stilts
script installed,
write "java -jar stilts.jar
" instead of
"stilts
" - see Section 3.
The available <stilts-flags>
are listed
in Section 2.1.
Parameter values are assigned on the command line as explained in Section 2.3. They are as follows:
copycols = <colid-list>
*
", which means that
all columns from the input table are included in the output.
[Default: *
]
dec = <expr>
emptyok = true|false
true
, it is assumed
that the service behaves properly in this respect; if it does not
an error may result. In that case, set this parameter
false
. A consequence of setting it false is that
in the event of no results being returned, the task will
return no table at all, rather than an empty one.
[Default: true
]
erract = abort|ignore|retry|retry<n>
abort
:
failure of any query terminates the task
ignore
:
failure of a query is treated the same as a query which
returns no rows
retry
:
failed queries are retried until they succeed;
use with care - if the failure is for some good, or at least
reproducible reason this could prevent the task from ever
completing
retry<n>
:
failed queries are retried at most a fixed number
<n>
of times
If they still fail the task terminates.
[Default: abort
]
find = best|all
best
is selected, then only the query table row
which best matches the row from the input table will be output.
If all
is selected, then any rows in the query table
which match the input table are output.
[Default: all
]
fixcols = none|dups|all
none
: columns are not renameddups
: columns which would otherwise have duplicate names in the output will be renamed to indicate which table they came fromall
: all columns will be renamed to indicate which table they came fromsuffix*
parameters.
[Default: dups
]
icmd = <cmds>
The value of this parameter is one or more of the filter commands described in Section 5.1. If more than one is given, they must be separated by semicolon characters (";"). This parameter can be repeated multiple times on the same command line to build up a list of processing steps. The sequence of commands given in this way defines the processing pipeline which is performed on the table.
Commands may alteratively be supplied in an external file,
by using the indirection character '@'.
Thus "icmd=@filename
"
causes the file filename
to be read for a list
of filter commands to execute. The commands in the file
may be separated by newline characters and/or semicolons.
ifmt = <in-format>
(auto)
(the default),
then an attempt will be
made to detect the format of the table automatically.
This cannot always be done correctly however, in which case
the program will exit with an error explaining which
formats were attempted.
[Default: (auto)
]
in = <table>
ifmt
parameter.
istream = true|false
in
table
will be read as a stream.
It is necessary to give the
ifmt
parameter
in this case.
Depending on the required operations and processing mode,
this may cause the read to fail (sometimes it is necessary
to read the input table more than once).
It is not normally necessary to set this flag;
in most cases the data will be streamed automatically
if that is the best thing to do.
However it can sometimes result in less resource usage when
processing large files in certain formats (such as VOTable).
[Default: false
]
ocmd = <cmds>
The value of this parameter is one or more of the filter commands described in Section 5.1. If more than one is given, they must be separated by semicolon characters (";"). This parameter can be repeated multiple times on the same command line to build up a list of processing steps. The sequence of commands given in this way defines the processing pipeline which is performed on the table.
Commands may alteratively be supplied in an external file,
by using the indirection character '@'.
Thus "ocmd=@filename
"
causes the file filename
to be read for a list
of filter commands to execute. The commands in the file
may be separated by newline characters and/or semicolons.
ofmt = <out-format>
(auto)
"
(the default),
then the output filename will be
examined to try to guess what sort of file is required
usually by looking at the extension.
If it's not obvious from the filename what output format is
intended, an error will result.
This parameter must only be given if
omode
has its default value of "out
".
[Default: (auto)
]
omode = <out-mode> <mode-args>
out
, which means that
the result will be written as a new table to disk or elsewhere,
as determined by the out
and ofmt
parameters.
However, there are other possibilities, which correspond
to uses to which a table can be put other than outputting it,
such as displaying metadata, calculating statistics,
or populating a table in an SQL database.
For some values of this parameter, additional parameters
(<mode-args>
)
are required to determine the exact behaviour.
Possible values are
out
meta
stats
count
cgi
discard
topcat
plastic
tosql
help=omode
flag
or see Section 5.4 for more information.
[Default: out
]
ostream = true|false
[Default: false
]
out = <out-table>
This parameter must only be given if
omode
has its default value of "out
".
[Default: -
]
parallel = <n>
<n>
, then queries will be overlapped
in such a way that up to approximately <n>
may be running at any one time.
Whether this is a good idea, and what might be a sensible
maximum value for <n>
, depends on the
characteristics of the service being queried.
[Default: 1
]
ra = <expr>
scorecol = <col-name>
[Default: Separation
]
serviceurl = <value>
name=value
", separated by '&' characters).
If this value does not end in either a '?' or a '&',
one will be added as appropriate.
See Appendix B.2.3 for discussion of how to locate service URLs corresponding to given datasets.
sr = <expr>
suffix0 = <label>
fixcols
parameter
is set so that input columns are renamed for insertion into
the output table, this parameter determines how the
renaming is done.
It gives a suffix which is appended to all renamed columns
from the input table.
[Default: _0
]
suffix1 = <label>
fixcols
parameter
is set so that input columns are renamed for insertion into
the output table, this parameter determines how the
renaming is done.
It gives a suffix which is appended to all renamed columns
from the cone result table.
[Default: _1
]
verb = 1|2|3
Here are some examples of coneskymatch
:
stilts coneskymatch serviceurl=http://archive.stsci.edu/hst/search.php \ in=messier.xml ra=RA dec=DEC sr=0.05 \ out=matches.xml
messier.xml
. The result is written to a
new VOTable, matches.xml
. The J2000 positions of each
record in the input file are held in columns named RA and
DEC respectively.
stilts coneskymatch \ serviceurl='http://www.nofs.navy.mil/cgi-bin/vo_cone.cgi?CAT=NOMAD' \ in=vizier.xml#7 \ icmd='addskycoords -inunit sex fk4 fk5 RAB1950 DEB1950 RAJ2000 DEJ2000' \ icmd='progress' ra=RAJ2000 dec=DEJ2000 sr=0.01 \ ocmd='replacecol -units rad RA hmsToRadians(RA[0],RA[1],RA[2])' \ ocmd='replacecol -units rad DEC dmsToRadians(DEC[0],DEC[1],DEC[2])' \ omode=topcat
The input catalogue, which is the 8th TABLE element in a VOTable file,
contains sky positions in sexagesimal FK4 (B1950) coordinates.
The icmd=addskycoords...
parameter specifies a filter which
will add new columns in FK5 (J2000) degrees, which are what the
coneskymatch
command requires.
The icmd=progress
parameter specifies a filter which
will write progress information to the terminal so you can see how
the queries are progressing.
The NOMAD service specified by the serviceurl
parameter
used here happens to return results with the RA/DEC columns represented
in a rather eccentric format, namely 3-element floating point arrays
representing (hours,minutes,seconds)/(degrees,minutes,seconds).
The two ocmd=replacecol...
filters replace the values of
these columns with the scalar equivalents in radians.
Finally, the omode=topcat
parameter causes the result
table to be loaded directly into TOPCAT (if it is available).
stilts coneskymatch serviceurl='http://archive.stsci.edu/iue/search.php?' \ in=queries.txt ifmt=ascii \ ra='$1' dec='$2' \ sr='$3' copycols='$4' \ out=found.fits
sr
parameter)
is taken from the table and so varies for each query.
The copycols
parameter has the value
'$4
',
which means that the value of the fourth column of the input table
will be prepended to each row of the output table for
which it is responsible.
Output is to a FITS table.
To use the coneskymatch
command you need the
service URL (also known as the base URL
or access URL) of a cone search service to use.
If you know one of these representing a service that you wish to use,
you can use it directly.
If you don't, you will need to find the URL from somewhere. It is the job of the Virtual Observatory Registry to keep a record of where you can find various astronomical services, so this is where you should look.
There are various ways you can interrogate the registry; the easiest is probably to use a graphical registry search tool such as AstroGrid's VOExplorer.
It is also possible to interrogate the registry using STILTS's
regquery
command. See that section of the manual for details, but for
instance to locate registered Cone Search services which have
something to do with SDSS data, you could execute the following:
stilts regquery query="serviceType = 'CONE' and title like '%Sloan%'" \ ocmd="keepcols 'shortName serviceUrl'" \ ofmt=asciiWriting just
query="serviceType = 'CONE'"
with no
further qualification would give you all
registered cone search services.
funcs
: Browses functions used by algebraic expression langauage
funcs
is a utility which allows you to browse the
functions you can use in STILTS's algebraic expression language.
Invoking the command causes a window to pop up on the display with
two parts. The left hand panel contains a tree-like representation of the
functions available - the top level shows the classes (categories)
into which the functions are divided, and if you open these up
(by double clicking on them) each contains a list of functions and
constants in that class.
If you click on any of these classes or their constituent functions
or constants, a full descritption of what they are and how to
use them will appear in the right hand panel.
The information available from this command is the same as that given in Section 8.5, but the graphical browser may be a more convenient way to view the documentation. There are no parameters.
The usage of funcs
is
stilts <stilts-flags> funcsIf you don't have the
stilts
script installed,
write "java -jar stilts.jar
" instead of
"stilts
" - see Section 3.
The available <stilts-flags>
are listed
in Section 2.1.
This task has no parameters.
plot2d
: 2D Scatter Plot
plot2d
performs two-dimensional scatter plots,
sending the output to a graphical display or writing it to a file
in some vector or bitmapped graphics format.
You need to supply it with values for one or more X and Y datasets,
in terms of table columns, and it will generate a plot with a point
for each row.
There are many options available to configure the detailed appearance
of the plot, but in its simplest form invocation is quite straightforward.
See Section 7 for more discussion on use of the plotting commands.
The usage of plot2d
is
stilts <stilts-flags> plot2d xpix=<int-value> ypix=<int-value> font=dialog|serif|... fontsize=<int-value> fontstyle=plain|bold|italic|bold-italic title=<value> omode=swing|out|cgi|discard|auto out=<out-file> ofmt=png|gif|jpeg|eps|eps-gzip inN=<table> ifmtN=<in-format> istreamN=true|false cmdN=<cmds> xdataN=<expr> ydataN=<expr> auxdataN=<expr> xlo=<float-value> ylo=<float-value> auxlo=<float-value> xhi=<float-value> yhi=<float-value> auxhi=<float-value> xlog=true|false ylog=true|false auxlog=true|false xflip=true|false yflip=true|false auxflip=true|false xlabel=<value> ylabel=<value> auxlabel=<value> xerrorN=<expr>|[<lo-expr>],[<hi-expr>] yerrorN=<expr>|[<lo-expr>],[<hi-expr>] auxshader=rainbow|pastel|... txtlabelN=<value> subsetNS=<expr> nameNS=<value> colourNS=<rrggbb>|red|blue|... shapeNS=filled_circle|open_circle|... sizeNS=<int-value> transparencyNS=<int-value> lineNS=DotToDot|LinearRegression hideNS=true|false errstyleNS=lines|capped_lines|... grid=true|false antialias=true|false sequence=<suffix>,<suffix>,...If you don't have the
stilts
script installed,
write "java -jar stilts.jar
" instead of
"stilts
" - see Section 3.
The available <stilts-flags>
are listed
in Section 2.1.
Parameter values are assigned on the command line as explained in Section 2.3. They are as follows:
antialias = true|false
[Default: true
]
auxdataN = <expr>
auxflip = true|false
[Default: false
]
auxhi = <float-value>
auxlabel = <value>
auxlo = <float-value>
auxlog = true|false
[Default: false
]
auxshader = rainbow|pastel|...
rainbow
pastel
standard
heat
colour
hue
greyscale
red-blue
hsv_h
hsv_s
hsv_v
intensity
rgb_red
rgb_green
rgb_blue
yuv_y
yuv_u
yuv_v
transparency
[Default: rainbow
]
cmdN = <cmds>
Commands may alteratively be supplied in an external file,
by using the indirection character '@'.
Thus "cmdN=@filename
"
causes the file filename
to be read for a list
of filter commands to execute. The commands in the file
may be separated by newline characters and/or semicolons.
colourNS = <rrggbb>|red|blue|...
ff00ff
" for magenta.
Alternatively it may be the name of one of the
pre-defined colours.
These are currently
red, blue, green, grey, magenta, cyan, orange, pink, yellow, black and white.
For most purposes, either the American or the British spelling is accepted for this parameter name.
errstyleNS = lines|capped_lines|...
none
lines
capped_lines
caps
arrows
ellipse
crosshair_ellipse
rectangle
crosshair_rectangle
filled_ellipse
filled_rectangle
[Default: lines
]
font = dialog|serif|...
bitstream_charter
courier
courier_10_pitch
cursor
default
dialog
dialoginput
lucida_bright
lucida_sans
lucida_sans_typewriter
luxi_mono
luxi_sans
luxi_serif
monospaced
sansserif
serif
utopia
[Default: dialog
]
fontsize = <int-value>
[Default: 12
]
fontstyle = plain|bold|italic|bold-italic
plain
,
bold
,
italic
and
bold-italic
.
[Default: plain
]
grid = true|false
[Default: true
]
hideNS = true|false
[Default: false
]
ifmtN = <in-format>
(auto)
(the default),
then an attempt will be
made to detect the format of the table automatically.
This cannot always be done correctly however, in which case
the program will exit with an error explaining which
formats were attempted.
[Default: (auto)
]
inN = <table>
ifmtN
parameter.
istreamN = true|false
inN
table
will be read as a stream.
It is necessary to give the
ifmtN
parameter
in this case.
Depending on the required operations and processing mode,
this may cause the read to fail (sometimes it is necessary
to read the input table more than once).
It is not normally necessary to set this flag;
in most cases the data will be streamed automatically
if that is the best thing to do.
However it can sometimes result in less resource usage when
processing large files in certain formats (such as VOTable).
[Default: false
]
lineNS = DotToDot|LinearRegression
null
:
No line is plotted.DotToDot
:
Each point is joined to the next one in sequence
by a straight line.LinearRegression
:
A linear regression line is plotted based on all the points
which are visible in the plot.
Note that the regression coefficients take no account of
points out of the visible range.nameNS = <value>
ofmt = png|gif|jpeg|eps|eps-gzip
png
: image/png formatgif
: image/gif formatjpeg
: image/jpeg formateps
: application/postscript formateps-gzip
: application/postscript (gzip) formatout
.
omode = swing|out|cgi|discard|auto
swing
:
Plot will be displayed in a window on the screen.out
:
Plot will be written to a file given by out
using the graphics format given by ofmt
.cgi
:
Plot will be written in a way suitable for CGI use direct from a web server.
The output is in the graphics format given by ofmt
,
preceded by a suitable "Content-type" declaration.discard
:
Plot is drawn, but discarded. There is no output.auto
:
Behaves as swing
or out
mode depending on presence of out
parameter[Default: auto
]
out = <out-file>
sequence = <suffix>,<suffix>,...
NS
"
suffixes which appear on the
parameters which apply to subsets.
The sets which are named
will be plotted in order, so the first-named one will be
at the bottom (most likely to be obscured).
Note that if this parameter is supplied, then only those sets
which are named will be plotted,
so this parameter may also be used to restrict which plots appear
(though it may not be the most efficient way of doing this).
If no explicit value is supplied for this parameter,
sets will be plotted in some sequence decided by STILTS
(probably alphabetic by suffix).
shapeNS = filled_circle|open_circle|...
filled_circle
open_circle
cross
x
open_square
open_diamond
open_triangle_up
open_triangle_down
filled_square
filled_diamond
filled_triangle_up
filled_triangle_down
sizeNS = <int-value>
[Default: -1
]
subsetNS = <expr>
NS
".
This is a boolean expression which may be the name of
a boolean-valued column or any other boolean-valued expression.
Rows for which the expression evaluates true will be included
in the subset, and those for which it evaluates false will not.
title = <value>
transparencyNS = <int-value>
<n>
means that opacity is only
achieved (the background is only blotted out)
when <n>
pixels of this colour have been
plotted on top of each other.
The minimum value is 1, which means opaque markers.
txtlabelN = <value>
xdataN = <expr>
xerrorN = <expr>|[<lo-expr>],[<hi-expr>]
<expr>
: symmetric error value<lo-expr>,<hi-expr>
:distinct lower and upper error values<lo-expr>,
: lower error value only,<hi-expr>
: upper error value onlynull
: no errorsxflip = true|false
[Default: false
]
xhi = <float-value>
xlabel = <value>
xlo = <float-value>
xlog = true|false
[Default: false
]
xpix = <int-value>
[Default: 400
]
ydataN = <expr>
yerrorN = <expr>|[<lo-expr>],[<hi-expr>]
<expr>
: symmetric error value<lo-expr>,<hi-expr>
:distinct lower and upper error values<lo-expr>,
: lower error value only,<hi-expr>
: upper error value onlynull
: no errorsyflip = true|false
[Default: false
]
yhi = <float-value>
ylabel = <value>
ylo = <float-value>
ylog = true|false
[Default: false
]
ypix = <int-value>
[Default: 300
]
Here are some examples of plot2d
in use:
stilts plot2d in=cat.xml xdata=RMAG-BMAG ydata=BMAG
omode
or out
value
has been specified, the plot is posted directly
to the graphics display for inspection.
By adding the parameter
out=xyplot.eps
the plot could be written to an
Encapsulated Postscript file instead.
The generated plot is here.
stilts plot2d in=6dfgs_mini.xml xdata=RMAG-BMAG ydata=BMAG subset1=SGFLAG==1 name1=galaxy colour1=blue shape1=open_circle subset2=SGFLAG==2 name2=star colour2=e010f0 shape2=x size2=3 xlo=-1 xhi=4.5 ylo=10 yhi=20 xpix=500 ypix=250 out=xyplot2.png
1
" and "2
"
with separate sets of parameters applying to each.
The selections for the sets are given by the
subset*
parameters;
set 1 is those rows with the SGFLAG column equal to 1 and
set 2 is those rows with the SGFLAG column equal to 2.
The boundaries of the plot in data coordinates
are set explicitly rather than being determined from
the data (this is faster)
and the plot size in pixels is also set explicitly
rather than taking the default values.
Output is to a PNG file.
The generated plot is here.
stilts plot2d in1=iras_psc.fits cmd1='addskycoords fk5 galactic RA DEC GLON GLAT' xdata1=GLON ydata1=GLAT auxdata1=FNU_100 auxlog=true auxflip=true size1=0 transparency1=3 in2=messier.xml cmd2='addskycoords fk5 galactic RA DEC GLON GLAT' xdata2=GLON ydata2=GLAT txtlabel2=RADIUS>16?("M"+ID):"" cmd2='addcol SIZE sqrt(RADIUS/2)' xerror2=SIZE yerror2=SIZE subset2a=true hide2a=true colour2a=black errstyle2a=ellipse subset2b=true hide2b=true colour2b=black errstyle2b=filled_ellipse transparency2b=6 xlabel='Galactic Longitude' ylabel='Galactic Latitude' title='The Sky' legend=false grid=false fontsize=12 fontstyle=bold-italic xlo=0 xhi=360 ylo=-90 yhi=+90 xpix=600 ypix=300 out=skyplot.png
The generated plot is here.
plot3d
: 3D Scatter Plot
plot3d
performs three-dimensional scatter plots,
sending the output to a graphical display or writing it to a file
in some vector or bitmapped graphics format.
You need to supply it with values for one or more X, Y and Z datasets,
in terms of table columns, and it will generate a plot with a point
for each row.
There are many options available to configure the detailed appearance
of the plot, but in its simplest form invocation is quite straightforward.
See Section 7 for more discussion on use of the plotting commands.
The usage of plot3d
is
stilts <stilts-flags> plot3d xpix=<int-value> ypix=<int-value> font=dialog|serif|... fontsize=<int-value> fontstyle=plain|bold|italic|bold-italic title=<value> omode=swing|out|cgi|discard|auto out=<out-file> ofmt=png|gif|jpeg|eps|eps-gzip inN=<table> ifmtN=<in-format> istreamN=true|false cmdN=<cmds> xdataN=<expr> ydataN=<expr> zdataN=<expr> auxdataN=<expr> xlo=<float-value> ylo=<float-value> zlo=<float-value> auxlo=<float-value> xhi=<float-value> yhi=<float-value> zhi=<float-value> auxhi=<float-value> xlog=true|false ylog=true|false zlog=true|false auxlog=true|false xflip=true|false yflip=true|false zflip=true|false auxflip=true|false xlabel=<value> ylabel=<value> zlabel=<value> auxlabel=<value> xerrorN=<expr>|[<lo-expr>],[<hi-expr>] yerrorN=<expr>|[<lo-expr>],[<hi-expr>] zerrorN=<expr>|[<lo-expr>],[<hi-expr>] auxshader=rainbow|pastel|... txtlabelN=<value> subsetNS=<expr> nameNS=<value> colourNS=<rrggbb>|red|blue|... shapeNS=filled_circle|open_circle|... sizeNS=<int-value> transparencyNS=<int-value> lineNS=DotToDot|LinearRegression hideNS=true|false errstyleNS=lines|capped_lines|... grid=true|false antialias=true|false sequence=<suffix>,<suffix>,... fog=<float-value> phi=<float-value> theta=<float-value>If you don't have the
stilts
script installed,
write "java -jar stilts.jar
" instead of
"stilts
" - see Section 3.
The available <stilts-flags>
are listed
in Section 2.1.
Parameter values are assigned on the command line as explained in Section 2.3. They are as follows:
antialias = true|false
[Default: true
]
auxdataN = <expr>
auxflip = true|false
[Default: false
]
auxhi = <float-value>
auxlabel = <value>
auxlo = <float-value>
auxlog = true|false
[Default: false
]
auxshader = rainbow|pastel|...
rainbow
pastel
standard
heat
colour
hue
greyscale
red-blue
hsv_h
hsv_s
hsv_v
intensity
rgb_red
rgb_green
rgb_blue
yuv_y
yuv_u
yuv_v
transparency
[Default: rainbow
]
cmdN = <cmds>
Commands may alteratively be supplied in an external file,
by using the indirection character '@'.
Thus "cmdN=@filename
"
causes the file filename
to be read for a list
of filter commands to execute. The commands in the file
may be separated by newline characters and/or semicolons.
colourNS = <rrggbb>|red|blue|...
ff00ff
" for magenta.
Alternatively it may be the name of one of the
pre-defined colours.
These are currently
red, blue, green, grey, magenta, cyan, orange, pink, yellow, black and white.
For most purposes, either the American or the British spelling is accepted for this parameter name.
errstyleNS = lines|capped_lines|...
none
lines
capped_lines
caps
arrows
cuboid
ellipse
crosshair_ellipse
rectangle
crosshair_rectangle
filled_ellipse
filled_rectangle
[Default: lines
]
fog = <float-value>
[Default: 1.0
]
font = dialog|serif|...
bitstream_charter
courier
courier_10_pitch
cursor
default
dialog
dialoginput
lucida_bright
lucida_sans
lucida_sans_typewriter
luxi_mono
luxi_sans
luxi_serif
monospaced
sansserif
serif
utopia
[Default: dialog
]
fontsize = <int-value>
[Default: 12
]
fontstyle = plain|bold|italic|bold-italic
plain
,
bold
,
italic
and
bold-italic
.
[Default: plain
]
grid = true|false
[Default: true
]
hideNS = true|false
[Default: false
]
ifmtN = <in-format>
(auto)
(the default),
then an attempt will be
made to detect the format of the table automatically.
This cannot always be done correctly however, in which case
the program will exit with an error explaining which
formats were attempted.
[Default: (auto)
]
inN = <table>
ifmtN
parameter.
istreamN = true|false
inN
table
will be read as a stream.
It is necessary to give the
ifmtN
parameter
in this case.
Depending on the required operations and processing mode,
this may cause the read to fail (sometimes it is necessary
to read the input table more than once).
It is not normally necessary to set this flag;
in most cases the data will be streamed automatically
if that is the best thing to do.
However it can sometimes result in less resource usage when
processing large files in certain formats (such as VOTable).
[Default: false
]
lineNS = DotToDot|LinearRegression
null
:
No line is plotted.DotToDot
:
Each point is joined to the next one in sequence
by a straight line.LinearRegression
:
A linear regression line is plotted based on all the points
which are visible in the plot.
Note that the regression coefficients take no account of
points out of the visible range.nameNS = <value>
ofmt = png|gif|jpeg|eps|eps-gzip
png
: image/png formatgif
: image/gif formatjpeg
: image/jpeg formateps
: application/postscript formateps-gzip
: application/postscript (gzip) formatout
.
omode = swing|out|cgi|discard|auto
swing
:
Plot will be displayed in a window on the screen.out
:
Plot will be written to a file given by out
using the graphics format given by ofmt
.cgi
:
Plot will be written in a way suitable for CGI use direct from a web server.
The output is in the graphics format given by ofmt
,
preceded by a suitable "Content-type" declaration.discard
:
Plot is drawn, but discarded. There is no output.auto
:
Behaves as swing
or out
mode depending on presence of out
parameter[Default: auto
]
out = <out-file>
phi = <float-value>
[Default: 30.0
]
sequence = <suffix>,<suffix>,...
NS
"
suffixes which appear on the
parameters which apply to subsets.
The sets which are named
will be plotted in order, so the first-named one will be
at the bottom (most likely to be obscured).
Note that if this parameter is supplied, then only those sets
which are named will be plotted,
so this parameter may also be used to restrict which plots appear
(though it may not be the most efficient way of doing this).
If no explicit value is supplied for this parameter,
sets will be plotted in some sequence decided by STILTS
(probably alphabetic by suffix).
shapeNS = filled_circle|open_circle|...
filled_circle
open_circle
cross
x
open_square
open_diamond
open_triangle_up
open_triangle_down
filled_square
filled_diamond
filled_triangle_up
filled_triangle_down
sizeNS = <int-value>
[Default: -1
]
subsetNS = <expr>
NS
".
This is a boolean expression which may be the name of
a boolean-valued column or any other boolean-valued expression.
Rows for which the expression evaluates true will be included
in the subset, and those for which it evaluates false will not.
theta = <float-value>
[Default: 15.0
]
title = <value>
transparencyNS = <int-value>
<n>
means that opacity is only
achieved (the background is only blotted out)
when <n>
pixels of this colour have been
plotted on top of each other.
The minimum value is 1, which means opaque markers.
txtlabelN = <value>
xdataN = <expr>
xerrorN = <expr>|[<lo-expr>],[<hi-expr>]
<expr>
: symmetric error value<lo-expr>,<hi-expr>
:distinct lower and upper error values<lo-expr>,
: lower error value only,<hi-expr>
: upper error value onlynull
: no errorsxflip = true|false
[Default: false
]
xhi = <float-value>
xlabel = <value>
xlo = <float-value>
xlog = true|false
[Default: false
]
xpix = <int-value>
[Default: 300
]
ydataN = <expr>
yerrorN = <expr>|[<lo-expr>],[<hi-expr>]
<expr>
: symmetric error value<lo-expr>,<hi-expr>
:distinct lower and upper error values<lo-expr>,
: lower error value only,<hi-expr>
: upper error value onlynull
: no errorsyflip = true|false
[Default: false
]
yhi = <float-value>
ylabel = <value>
ylo = <float-value>
ylog = true|false
[Default: false
]
ypix = <int-value>
[Default: 300
]
zdataN = <expr>
zerrorN = <expr>|[<lo-expr>],[<hi-expr>]
<expr>
: symmetric error value<lo-expr>,<hi-expr>
:distinct lower and upper error values<lo-expr>,
: lower error value only,<hi-expr>
: upper error value onlynull
: no errorszflip = true|false
[Default: false
]
zhi = <float-value>
zlabel = <value>
zlo = <float-value>
zlog = true|false
[Default: false
]
Here are some examples of plot3d
in use:
stilts plot3d in=cat.xml xdata=RMAG ydata=BMAG zdata=VEL zlog=true
omode
or out
value
has been specified, the plot is posted directly
to the graphics display for inspection.
By adding the parameter
out=xyplot.eps
the plot could be written to an
Encapsulated Postscript file instead.
The generated plot is here.
stilts plot3d in=sim1.fits xdata=x ydata=y zdata=z cmd='addcol vel "sqrt(velx*velx+vely*vely+velz*velz)"' auxdata=vel auxlog=true xpix=500 ypix=400 phi=50 theta=10 out=cube.jpeg
vel
column to the table using the
addcol
filter command, so that the vel
column
can be used as the value for the auxdata
parameter.
Alternatively, the given expression for the velocity
could have been used directly as the value of the
auxdata
parameter.
Additionally, the phi
and
theta
parameters are given
to adjust the orientation of the cube.
The generated plot is here.
plothist
: Histogram
plothist
performs histogram plots,
sending the output to a graphical display or writing it to a file
in some vector or bitmapped graphics format.
You need to supply it with values for one or more sets of X values,
in terms of table columns, and it will bin the data and draw bars
appropriately.
Plot bounds, bin widths etc may be supplied expliicitly, but will be
calculated from the data and set from defaults as appropriate otherwise.
There are many options available to configure the detailed appearance
of the plot, but in its simplest form invocation is quite straightforward.
See Section 7 for more discussion on use of the plotting commands.
The usage of plothist
is
stilts <stilts-flags> plothist xpix=<int-value> ypix=<int-value> font=dialog|serif|... fontsize=<int-value> fontstyle=plain|bold|italic|bold-italic title=<value> omode=swing|out|cgi|discard|auto out=<out-file> ofmt=png|gif|jpeg|eps|eps-gzip inN=<table> ifmtN=<in-format> istreamN=true|false cmdN=<cmds> xdataN=<expr> xlo=<float-value> xhi=<float-value> xlog=true|false xflip=true|false xlabel=<value> subsetNS=<expr> nameNS=<value> colourNS=<rrggbb>|red|blue|... barstyleNS=fill|open|... linewidthNS=<int-value> dashNS=dot|dash|...|<a,b,...> grid=true|false antialias=true|false sequence=<suffix>,<suffix>,... ylo=<float-value> yhi=<float-value> ylog=true|false weightN=<value> binwidth=<float-value> norm=true|false cumulative=true|false binbase=<float-value>If you don't have the
stilts
script installed,
write "java -jar stilts.jar
" instead of
"stilts
" - see Section 3.
The available <stilts-flags>
are listed
in Section 2.1.
Parameter values are assigned on the command line as explained in Section 2.3. They are as follows:
antialias = true|false
[Default: true
]
barstyleNS = fill|open|...
fill
open
tops
spikes
fillover
openover
[Default: fill
]
binbase = <float-value>
[Default: 0
]
binwidth = <float-value>
cmdN = <cmds>
Commands may alteratively be supplied in an external file,
by using the indirection character '@'.
Thus "cmdN=@filename
"
causes the file filename
to be read for a list
of filter commands to execute. The commands in the file
may be separated by newline characters and/or semicolons.
colourNS = <rrggbb>|red|blue|...
ff00ff
" for magenta.
Alternatively it may be the name of one of the
pre-defined colours.
These are currently
red, blue, green, grey, magenta, cyan, orange, pink, yellow, black and white.
For most purposes, either the American or the British spelling is accepted for this parameter name.
cumulative = true|false
[Default: false
]
dashNS = dot|dash|...|<a,b,...>
dot
dash
longdash
dotdash
4,2,8,2
".
A null
value indicates a solid line.
Only certain bar styles are affected by the dash pattern.
font = dialog|serif|...
bitstream_charter
courier
courier_10_pitch
cursor
default
dialog
dialoginput
lucida_bright
lucida_sans
lucida_sans_typewriter
luxi_mono
luxi_sans
luxi_serif
monospaced
sansserif
serif
utopia
[Default: dialog
]
fontsize = <int-value>
[Default: 12
]
fontstyle = plain|bold|italic|bold-italic
plain
,
bold
,
italic
and
bold-italic
.
[Default: plain
]
grid = true|false
[Default: true
]
ifmtN = <in-format>
(auto)
(the default),
then an attempt will be
made to detect the format of the table automatically.
This cannot always be done correctly however, in which case
the program will exit with an error explaining which
formats were attempted.
[Default: (auto)
]
inN = <table>
ifmtN
parameter.
istreamN = true|false
inN
table
will be read as a stream.
It is necessary to give the
ifmtN
parameter
in this case.
Depending on the required operations and processing mode,
this may cause the read to fail (sometimes it is necessary
to read the input table more than once).
It is not normally necessary to set this flag;
in most cases the data will be streamed automatically
if that is the best thing to do.
However it can sometimes result in less resource usage when
processing large files in certain formats (such as VOTable).
[Default: false
]
linewidthNS = <int-value>
[Default: 2
]
nameNS = <value>
norm = true|false
[Default: false
]
ofmt = png|gif|jpeg|eps|eps-gzip
png
: image/png formatgif
: image/gif formatjpeg
: image/jpeg formateps
: application/postscript formateps-gzip
: application/postscript (gzip) formatout
.
omode = swing|out|cgi|discard|auto
swing
:
Plot will be displayed in a window on the screen.out
:
Plot will be written to a file given by out
using the graphics format given by ofmt
.cgi
:
Plot will be written in a way suitable for CGI use direct from a web server.
The output is in the graphics format given by ofmt
,
preceded by a suitable "Content-type" declaration.discard
:
Plot is drawn, but discarded. There is no output.auto
:
Behaves as swing
or out
mode depending on presence of out
parameter[Default: auto
]
out = <out-file>
sequence = <suffix>,<suffix>,...
NS
"
suffixes which appear on the
parameters which apply to subsets.
The sets which are named
will be plotted in order, so the first-named one will be
at the bottom (most likely to be obscured).
Note that if this parameter is supplied, then only those sets
which are named will be plotted,
so this parameter may also be used to restrict which plots appear
(though it may not be the most efficient way of doing this).
If no explicit value is supplied for this parameter,
sets will be plotted in some sequence decided by STILTS
(probably alphabetic by suffix).
subsetNS = <expr>
NS
".
This is a boolean expression which may be the name of
a boolean-valued column or any other boolean-valued expression.
Rows for which the expression evaluates true will be included
in the subset, and those for which it evaluates false will not.
title = <value>
weightN = <value>
When weighted, bars can be of negative height. An anomaly of the plot as currently implemented is that the Y axis never descends below zero, so any such bars are currently invisible. This may be amended in a future release (contact the author to lobby for such an amendment).
[Default: 1
]
xdataN = <expr>
xflip = true|false
[Default: false
]
xhi = <float-value>
xlabel = <value>
xlo = <float-value>
xlog = true|false
[Default: false
]
xpix = <int-value>
[Default: 400
]
yhi = <float-value>
ylo = <float-value>
[Default: 0
]
ylog = true|false
[Default: false
]
ypix = <int-value>
[Default: 300
]
Here are some examples of plothist
in use:
stilts plothist in=cat.xml xdata=RMAG-BMAG
The generated plot is here.
stilts plothist in=cat.xml xdata=RMAG-BMAG ofmt=eps-gzip out=hist.eps.gz
The generated plot is here.
stilts plothist inJ=2mass_xsc.fits xdataJ=j_m_k20fe barstyleJ=tops inH=2mass_xsc.fits xdataH=h_m_k20fe barstyleH=tops inK=2mass_xsc.fits xdataK=k_m_k20fe barstyleK=tops binwidth=0.1 xlo=12 xhi=16 xflip=true xlabel=Magnitude xpix=500 out=2mass.gif
J
",
"H
" and
"K
"
so these suffixes appear on all the dataset-dependent
parameters which are supplied.
The binwidth and X range are specified explicitly
rather than leaving them to be chosen automatically
by examining the data.
The generated plot is here.
regquery
: Queries the VO registry
regquery
submits a query to the Virtual Observatory
registry
and returns the result as a table containing all the records which
match the condition specified. The resulting table can be written out
in any of the supported formats or otherwise processed in the usual ways.
Currently the registry used by default is the SOAP service offered
by the US NVO registry at
http://voservices.net/registry/registry.asmx.
Because VO registries generally harvest from each other, the content of
this one can be expected to be similar to that stored in registries
maintained by other organisations.
The usage of regquery
is
stilts <stilts-flags> regquery query=<value> regurl=<value> ocmd=<cmds> omode=<out-mode> <mode-args> out=<out-table> ofmt=<out-format>If you don't have the
stilts
script installed,
write "java -jar stilts.jar
" instead of
"stilts
" - see Section 3.
The available <stilts-flags>
are listed
in Section 2.1.
Parameter values are assigned on the command line as explained in Section 2.3. They are as follows:
ocmd = <cmds>
The value of this parameter is one or more of the filter commands described in Section 5.1. If more than one is given, they must be separated by semicolon characters (";"). This parameter can be repeated multiple times on the same command line to build up a list of processing steps. The sequence of commands given in this way defines the processing pipeline which is performed on the table.
Commands may alteratively be supplied in an external file,
by using the indirection character '@'.
Thus "ocmd=@filename
"
causes the file filename
to be read for a list
of filter commands to execute. The commands in the file
may be separated by newline characters and/or semicolons.
ofmt = <out-format>
(auto)
"
(the default),
then the output filename will be
examined to try to guess what sort of file is required
usually by looking at the extension.
If it's not obvious from the filename what output format is
intended, an error will result.
This parameter must only be given if
omode
has its default value of "out
".
[Default: (auto)
]
omode = <out-mode> <mode-args>
out
, which means that
the result will be written as a new table to disk or elsewhere,
as determined by the out
and ofmt
parameters.
However, there are other possibilities, which correspond
to uses to which a table can be put other than outputting it,
such as displaying metadata, calculating statistics,
or populating a table in an SQL database.
For some values of this parameter, additional parameters
(<mode-args>
)
are required to determine the exact behaviour.
Possible values are
out
meta
stats
count
cgi
discard
topcat
plastic
tosql
help=omode
flag
or see Section 5.4 for more information.
[Default: out
]
out = <out-table>
This parameter must only be given if
omode
has its default value of "out
".
[Default: -
]
query = <value>
serviceType='CONE'
title like '%2MASS%'
publisher like 'CDS%' and title like '%galax%'
A full description of SQL syntax is beyond the scope of this
documentation, but in general you want to use
<field-name> like '<value>
where '%
' is a wildcard character.
Logical operators and
and or
and
parentheses can be used to group and combine expressions.
You can find the various <field-name>
s
by executing one of the queries above and looking at the
column names in the returned table.
regurl = <value>
[Default: http://voservices.net/registry/registry.asmx
]
Here are some examples of regquery
:
stilts regquery query="identifier like '%astrogrid%'" out=ag.xml
identifier
field contain the string
"astrogrid
". The '%
' characters function
as wildcards for the like
operator.
The output is written to a local VOTable file which can be examined
or further processed later.
stilts regquery query="serviceType = 'SSAP'" omode=count
serviceType
fields equal the string SSAP
(this identifies services which support the Simple Spectral Access
Protocol). These records are not stored, but the
omode=count
output mode counts the rows.
This therefore tells you how many SSAP servers are registered.
stilts regquery query="serviceType = 'CONE' and title like '%Sloan%'" \ ocmd="keepcols 'shortName serviceUrl'" \ ofmt=ascii out=-
keepcols
filter takes the result and throws away
all the columns except for shortName
and
serviceUrl
, and these are written to the terminal
in ASCII format. This may be useful to find the service URL
for a cone search service with particular data for use with
the coneskymatch
command.
server
: Runs an HTTP server to perform STILTS commands
server
runs an HTTP server which makes some or all
of the various STILTS tasks available to
local or remote clients making HTTP requests rather than using the
more usual command line interface.
When you run server
it will start up a server which
runs until it is interrupted, and write to the screen the base URL
at which it can be accessed, for instance
"http://localhost:2112/stilts/
".
If you point your browser here you will see some examples (hyperlinks to
server requests) of how to use the server.
Currently there are two main sets of capabilities:
/task/
task-name)
application/x-www-form-urlencoded
(see e.g. the
HTML FORM specification).
Some examples are given in the
Client Examples subsection below.
Either HTTP GET or POST methods may be used;
since the task invocations will normally be idempotent, GET is more
respectable, but long URLs can cause trouble in some circumstances
(MS IE apparently imposes a limit of about 2000 characters)
so POST may be preferable for lengthy invocations.
/form/
)
You might want to run STILTS in server mode if you are providing
a web service to external users which is able to access files residing
on the server, for instance generating table plots or row selections
on the fly.
This can be done without the server mode, for instance by invoking
the stilts
script or java from a CGI script to serve
each request, but using server mode has two advantages:
first it provides correct HTTP headers such as Content-Types,
and secondly it avoids the Java startup overheads for each invocation.
Note however that in its current form no great attention has been paid
to security, so it may be possible for clients to read and write files
and expend significant system resources by making certain requests to the
server. Anyone exposing the STILTS HTTP server directly to external
clients should bear this in mind.
For more flexibility you can run STILTS in servlet mode.
See the javadocs and sources of the
uk.ac.starlink.ttools.server.TaskServlet
class.
The server
command is a fairly thin wrapper around this,
which simply deploys the servlet in an embedded web application container
(Jetty).
By using the servlet class in your own custom web application instead you
can customise the way it is accessed, for instance providing improved
security.
Note:
The server
command and associated servlet code are
at time of writing (v2.0) experimental, and probably buggy and missing some
features which ought to be present.
If you have requirements which are not currently provided, please
contact the author for discussion.
The usage of server
is
stilts <stilts-flags> server port=<int-value> basepath=<value> tasks=<task-name> ... tablefactory=file|dirs:...|locator:...If you don't have the
stilts
script installed,
write "java -jar stilts.jar
" instead of
"stilts
" - see Section 3.
The available <stilts-flags>
are listed
in Section 2.1.
Parameter values are assigned on the command line as explained in Section 2.3. They are as follows:
basepath = <value>
/stilts
, which means that
for instance requests to execute task plot2d
should be directed to the URL
http://host:portnum/stilts/task/plot2d?name=value&name=value...
[Default: /stilts
]
port = <int-value>
[Default: 2112
]
tablefactory = file|dirs:...|locator:...
in
parameter
of table processing commands)
are used to acquire references to actual table data.
The default behaviour is for input table names to be treated
as filenames, in conjunction with some file type parameter.
While this is usually sensible for local use, in server
situations it may be inappropriate, since you don't want
external users to have read access to your entire filesystem.
This parameter gives options for alternative ways of mapping table names to table data items. The currently available options are:
file
:
default behaviour - names are treated as filenames
dirs:<dir>:<dir>:...
:
following the "dirs:
" prefix
a list of directories is specified which will be
searched for the file named.
Note that the directory separator character differs
between operating systems;
it is a colon (":
") for Unix-like OSs
and a semi-colon (";
") for MS Windows.
If a given name is identical to the path-less filename
in one of the <dir>
directories,
that file is used as the referenced table.
File type information is ignored in this case, so the files
must be one of the types which STILTS can autodetect,
currently FITS or VOTable (FITS is more efficient).
By using this option, clients can be restricted to using
a fixed set of tables in a restricted part of the server's
file system.
locator:<class-name>
:
the <class-name>
must be the name
of a Java class on the classpath which implements
the interface
uk.ac.starlink.ttools.task.TableLocator
and which has a no-arg constructor.
An instance of this class will be used to resolve names
to tables.
The usage and functionality of this parameter is experimental, and may change significantly in future releases.
[Default: file
]
tasks = <task-name> ...
null
then all tasks will be
available. However, some tasks don't make a lot of sense
to run from the server, so the default value is a somewhat
restricted list.
If the server is being exposed to external users, you might
also want to reduce the list for security reasons.
[Default: calc coneskymatch regquery plot2d plot3d plothist sqlclient sqlskymatch sqlupdate tcat tcatn tcopy tcube tjoin tmatch1 tmatch2 tmatchn tpipe tskymatch2 votcopy votlint
]
Here are some examples of running the server
command:
stilts server
http://localhost:2112/stilts/
".
stilts server port=2100 basepath=tableserv
plot2d
task
can be executed will be
"http://
host:2100/tableserv/task/plot2d
"
stilts server tasks="plot2d plothist"
plot2d
and
plothist
will be available for execution by clients.
Here are some examples of URLs which can be retrieved from
a server which is running at the base URL
http://localhost:2112/stilts/
.
All these use the HTTP GET form of request; the POST form could be
used instead with the same effect.
http://localhost:2112/stilts/
http://localhost:2112/stilts/task/tpipe
tpipe
task.
http://localhost:2112/stilts/task/calc?expression=21%2b2
calc
task
to return a document containing the text "23
".
Note that the plus ("+
") sign in the expression has to be
encoded using the sequence "%2b
" since "+
"
has a special significance in query URLs - see for instance sec 2.2 of
RFC 1738.
http://localhost:2112/stilts/task/plot2d?in=/data/table1.vot&xdata=RMAG&ydata=BMAG
plot2d
task
to return a magnitude-magnitude diagram of the named local file as an image
(probably an image/png
).
http://localhost:2112/stilts/task/tcopy?in=/data/cat.fits&ofmt=votable
tcopy
task
to return a translation of the named local FITS file to VOTable format.
sqlclient
:
Executes SQL statements
sqlclient
is a simple command-line client
for use with SQL databases.
One or more SQL statements can be supplied using the sql
parameter.
The result of each statement may be one or more update counts
(for update-type statements) or tables (for query-type statements).
Tables will be written to
standard output in a format given by the ofmt
parameter.
Update results and timing information will be written to standard error.
In most cases, you will find life easier if you use either the
database's own command-line or GUI client, or, if you require STILTS-type
format conversion or post-processing, a jdbc:
-format URL
for the in
parameter of the
tpipe
or
tcopy
commands
(see Section 3.4 for more explanation of the latter).
However, this command enables you to submit multiple queries over the
same JDBC connection, including ones which do not generate a tabular
result. It may be useful if a command-line client is not available to
you for the database you are using.
This command can only be used if you have access to an SQL database via JDBC. The details of how to configure a JDBC connection to a database are discussed in Section 3.4 - obviously you will need a database to connect to and appropriate permissions on it as well as the relevant drivers.
This command is experimental, and it may be enhanced, renamed or withdrawn in future releases.
The usage of sqlclient
is
stilts <stilts-flags> sqlclient db=<jdbc-url> user=<value> password=<value> sql=<sql> ofmt=<out-format>If you don't have the
stilts
script installed,
write "java -jar stilts.jar
" instead of
"stilts
" - see Section 3.
The available <stilts-flags>
are listed
in Section 2.1.
Parameter values are assigned on the command line as explained in Section 2.3. They are as follows:
db = <jdbc-url>
jdbc:<subprotocol>:<subname>
- the details are database- and driver-dependent.
Consult Sun's JDBC documentation and that for the particular
JDBC driver you are using for details.
Note that the relevant driver class will need to be on your
classpath and referenced in the jdbc.drivers
system property as well for the connection to be made.
ofmt = <out-format>
(auto)
"
(the default),
then the output filename will be
examined to try to guess what sort of file is required
usually by looking at the extension.
If it's not obvious from the filename what output format is
intended, an error will result.
[Default: text
]
password = <value>
sql = <sql>
;
") characters.
user = <value>
[Default: mbt
]
Here are some examples of sqlclient
:
stilts -classpath lib/drivers.jtds-1.1.jar \ -Djdbc.drivers=net.sourceforge.jtds.jdbc.Driver \ -Djava.net.preferIPv4Stack=true \ sqlclient \ db='jdbc:jtds:sqlserver://amenhotep:1433/twomass' \ user='guest1' \ ofmt=csv-nohead \ sql='SET SHOWPLAN_TEXT ON' \ sql='SELECT ra,dec FROM twomass_psc WHERE ra BETWEEN 21.7 AND 21.8 \ AND dec BETWEEN 9.1 AND 9.12'
sqlskymatch
:
Crossmatches table on sky position against SQL table
sqlskymatch
resembles
coneskymatch
,
but instead
of sending an HTTP query to a remote cone search service for each
match (i.e. each row of the input table), it executes an SQL query
directly. The query is a SELECT statement with a WHERE clause which
makes restrictions on Right Ascension and Declination columns;
the names of these columns must be given as parameters.
The effect is that of a spatial join between a client-side table and
a table stored in the database.
This command can only be used if you have access to an SQL database via JDBC. The details of how to configure a JDBC connection to a database are discussed in Section 3.4 - obviously you will need a database to connect to and appropriate read permissions on it as well as the relevant drivers.
Note: this task was known as sqlcone
in its experimental
form in STILTS v1.3.
The usage of sqlskymatch
is
stilts <stilts-flags> sqlskymatch ifmt=<in-format> istream=true|false icmd=<cmds> ocmd=<cmds> omode=<out-mode> <mode-args> out=<out-table> ofmt=<out-format> ra=<expr> dec=<expr> sr=<expr> find=best|all copycols=<colid-list> scorecol=<col-name> erract=abort|ignore|retry|retry<n> ostream=true|false fixcols=none|dups|all suffix0=<label> suffix1=<label> db=<jdbc-url> user=<value> password=<value> dbtable=<table-name> dbra=<sql-col> dbdec=<sql-col> dbunit=deg|rad tiling=htm<level>|healpixnest<nside>|healpixring<nside> dbtile=<sql-col> selectcols=<sql-cols> where=<sql-condition> preparesql=true|false [in=]<table>If you don't have the
stilts
script installed,
write "java -jar stilts.jar
" instead of
"stilts
" - see Section 3.
The available <stilts-flags>
are listed
in Section 2.1.
Parameter values are assigned on the command line as explained in Section 2.3. They are as follows:
copycols = <colid-list>
*
", which means that
all columns from the input table are included in the output.
[Default: *
]
db = <jdbc-url>
jdbc:<subprotocol>:<subname>
- the details are database- and driver-dependent.
Consult Sun's JDBC documentation and that for the particular
JDBC driver you are using for details.
Note that the relevant driver class will need to be on your
classpath and referenced in the jdbc.drivers
system property as well for the connection to be made.
dbdec = <sql-col>
dbtable
which gives the declination in degrees.
dbra = <sql-col>
dbtable
which gives the right ascension in degrees.
dbtable = <table-name>
dbtile = <sql-col>
dbtable
which contains a sky tiling pixel index.
The tiling scheme is given by the tiling
parameter.
Use of a tiling column is optional, but if present
(and if the column is indexed in the database table)
it may serve to speed up searches.
Set to null if the database table contains no tiling column
or if you do not wish to use one.
dbunit = deg|rad
[Default: deg
]
dec = <expr>
erract = abort|ignore|retry|retry<n>
abort
:
failure of any query terminates the task
ignore
:
failure of a query is treated the same as a query which
returns no rows
retry
:
failed queries are retried until they succeed;
use with care - if the failure is for some good, or at least
reproducible reason this could prevent the task from ever
completing
retry<n>
:
failed queries are retried at most a fixed number
<n>
of times
If they still fail the task terminates.
[Default: abort
]
find = best|all
best
is selected, then only the query table row
which best matches the row from the input table will be output.
If all
is selected, then any rows in the query table
which match the input table are output.
[Default: all
]
fixcols = none|dups|all
none
: columns are not renameddups
: columns which would otherwise have duplicate names in the output will be renamed to indicate which table they came fromall
: all columns will be renamed to indicate which table they came fromsuffix*
parameters.
[Default: dups
]
icmd = <cmds>
The value of this parameter is one or more of the filter commands described in Section 5.1. If more than one is given, they must be separated by semicolon characters (";"). This parameter can be repeated multiple times on the same command line to build up a list of processing steps. The sequence of commands given in this way defines the processing pipeline which is performed on the table.
Commands may alteratively be supplied in an external file,
by using the indirection character '@'.
Thus "icmd=@filename
"
causes the file filename
to be read for a list
of filter commands to execute. The commands in the file
may be separated by newline characters and/or semicolons.
ifmt = <in-format>
(auto)
(the default),
then an attempt will be
made to detect the format of the table automatically.
This cannot always be done correctly however, in which case
the program will exit with an error explaining which
formats were attempted.
[Default: (auto)
]
in = <table>
ifmt
parameter.
istream = true|false
in
table
will be read as a stream.
It is necessary to give the
ifmt
parameter
in this case.
Depending on the required operations and processing mode,
this may cause the read to fail (sometimes it is necessary
to read the input table more than once).
It is not normally necessary to set this flag;
in most cases the data will be streamed automatically
if that is the best thing to do.
However it can sometimes result in less resource usage when
processing large files in certain formats (such as VOTable).
[Default: false
]
ocmd = <cmds>
The value of this parameter is one or more of the filter commands described in Section 5.1. If more than one is given, they must be separated by semicolon characters (";"). This parameter can be repeated multiple times on the same command line to build up a list of processing steps. The sequence of commands given in this way defines the processing pipeline which is performed on the table.
Commands may alteratively be supplied in an external file,
by using the indirection character '@'.
Thus "ocmd=@filename
"
causes the file filename
to be read for a list
of filter commands to execute. The commands in the file
may be separated by newline characters and/or semicolons.
ofmt = <out-format>
(auto)
"
(the default),
then the output filename will be
examined to try to guess what sort of file is required
usually by looking at the extension.
If it's not obvious from the filename what output format is
intended, an error will result.
This parameter must only be given if
omode
has its default value of "out
".
[Default: (auto)
]
omode = <out-mode> <mode-args>
out
, which means that
the result will be written as a new table to disk or elsewhere,
as determined by the out
and ofmt
parameters.
However, there are other possibilities, which correspond
to uses to which a table can be put other than outputting it,
such as displaying metadata, calculating statistics,
or populating a table in an SQL database.
For some values of this parameter, additional parameters
(<mode-args>
)
are required to determine the exact behaviour.
Possible values are
out
meta
stats
count
cgi
discard
topcat
plastic
tosql
help=omode
flag
or see Section 5.4 for more information.
[Default: out
]
ostream = true|false
[Default: false
]
out = <out-table>
This parameter must only be given if
omode
has its default value of "out
".
[Default: -
]
password = <value>
preparesql = true|false
PreparedStatement
s for the SQL SELECTs
otherwise it will use simple Statement
s.
This is a tuning parameter and affects only performance.
On some database/driver combinations it's a lot faster set
false (the default); on others it may be faster, who knows?
[Default: false
]
ra = <expr>
scorecol = <col-name>
[Default: Separation
]
selectcols = <sql-cols>
*
" retrieves all columns.
[Default: *
]
sr = <expr>
suffix0 = <label>
fixcols
parameter
is set so that input columns are renamed for insertion into
the output table, this parameter determines how the
renaming is done.
It gives a suffix which is appended to all renamed columns
from the input table.
[Default: _0
]
suffix1 = <label>
fixcols
parameter
is set so that input columns are renamed for insertion into
the output table, this parameter determines how the
renaming is done.
It gives a suffix which is appended to all renamed columns
from the cone result table.
[Default: _1
]
tiling = htm<level>|healpixnest<nside>|healpixring<nside>
htm<level>
:
Hierarchical Triangular Mesh with a level value of
level
.healpixnest<nside>
:
HEALPix using the Nest scheme with an nside value of
nside
.healpixring<nside>
:
HEALPix using the Ring scheme with an nside value of
nside
.user = <value>
[Default: mbt
]
where = <sql-condition>
WHERE
keyword.
A null value indicates no additional criteria.
Here are some examples of sqlskymatch
:
stilts -classpath lib/drivers/mysql-connector-java.jar \ -Djdbc.drivers=com.mysql.jdbc.Driver sqlskymatch in=messier.xml ra=RA dec=DEC sr=0.05 \ db='jdbc:mysql://localhost/ASTRO1' user=mbt \ dbtable=FIRST dbra=_RA2000 dbdec=_DE2000 \ out=matches.xml
messier.xml
.
The result, a join between the Messier and FIRST tables, is output
as a VOTable called matches.xml
.
In this case a password has not been supplied on the command line,
so if one is required it will be prompted for on the console.
sqlupdate
: Updates values in an SQL table
sqlupdate
updates values in an existing table in an SQL
database. The rows to update are specified, as a normal
SELECT statement, using the select
parameter.
Each column to update, and the value to write to it, are given using
the assign
parameter.
Why not just use the database's own UPDATE statement? In most cases,
that would be a much better idea. However, using sqlupdate
you can write values using STILTS's
expression language, and hence take
advantage of its various functions, without having to embed them into
the database. SQL column names can be used as variables in these expressions,
in the same way that table column names are used as variables in
other commands such as tpipe
.
This command can only be used if you have access to an SQL database via JDBC. The details of how to configure a JDBC connection to a database are discussed in Section 3.4 - obviously you will need a database to connect to and appropriate write permissions on it as well as the relevant drivers.
This is a somewhat specialised command, and several (database/driver-specific) things can go wrong with it. If you do not have a fairly good understanding of the database with which you are using it then you may run into problems (but then you'd be unlikely to have the permissions to do the updates in any case).
The usage of sqlupdate
is
stilts <stilts-flags> sqlupdate db=<jdbc-url> user=<value> password=<value> select=<select-stmt> assign=<col>=<expr> progress=true|falseIf you don't have the
stilts
script installed,
write "java -jar stilts.jar
" instead of
"stilts
" - see Section 3.
The available <stilts-flags>
are listed
in Section 2.1.
Parameter values are assigned on the command line as explained in Section 2.3. They are as follows:
assign = <col>=<expr>
<colname>=<expr>
where <colname>
is the name of a column
in the SQL table and
<expr>
is the text of an expression using
STILTS's expression language, as described in Section 8.
SQL table column names or $ID
identifiers may be used as variables in the usual way.
This parameter may be supplied more than once to effect multiple assignments, or multiple assignments may be made by separating them with semicolons in the value of this parameter.
db = <jdbc-url>
jdbc:<subprotocol>:<subname>
- the details are database- and driver-dependent.
Consult Sun's JDBC documentation and that for the particular
JDBC driver you are using for details.
Note that the relevant driver class will need to be on your
classpath and referenced in the jdbc.drivers
system property as well for the connection to be made.
password = <value>
progress = true|false
[Default: true
]
select = <select-stmt>
SELECT
")
of the SELECT statement to identify which rows undergo
updates.
user = <value>
[Default: mbt
]
Here are some examples of sqlupdate
:
stilts -classpath lib/drivers/mysql-connector-java.jar \ -Djdbc.drivers=com.mysql.jdbc.Driver \ sqlupdate db='jdbc:mysql://localhost/RADIO' user=root select='SELECT * from FIRST" \ assign='HTMID=htmIndex(20,POS_EQ_RA,POS_EQ_DEC)'
tcat
: Concatenates multiple similar tables
tcat
is a tool for concatenating any number of
similar tables one after the other.
The tables must be of similar form to each other (same number and
types of columns). Preprocessing of the tables may be done using
the icmd
parameter, which will operate in the same
way on all the input tables. Table parameters of the output table
will be taken from the first of the input tables.
Subject to some constraints on the details of the input and output
formats and processing, tcat
is capable of joining
an unlimited number of tables together to produce an output table of
unlimited length, without large memory requirements.
If you have heterogeneous tables, in different formats of
requiring different preprocessing steps from each other before
they can be concatenated, use tcatn
.
The usage of tcat
is
stilts <stilts-flags> tcat in=<table> [<table> ...] ifmt=<in-format> istream=true|false icmd=<cmds> ocmd=<cmds> omode=<out-mode> <mode-args> out=<out-table> ofmt=<out-format> seqcol=<colname> loccol=<colname> uloccol=<colname> lazy=true|false countrows=true|falseIf you don't have the
stilts
script installed,
write "java -jar stilts.jar
" instead of
"stilts
" - see Section 3.
The available <stilts-flags>
are listed
in Section 2.1.
Parameter values are assigned on the command line as explained in Section 2.3. They are as follows:
countrows = true|false
[Default: false
]
icmd = <cmds>
The value of this parameter is one or more of the filter commands described in Section 5.1. If more than one is given, they must be separated by semicolon characters (";"). This parameter can be repeated multiple times on the same command line to build up a list of processing steps. The sequence of commands given in this way defines the processing pipeline which is performed on the table.
Commands may alteratively be supplied in an external file,
by using the indirection character '@'.
Thus "icmd=@filename
"
causes the file filename
to be read for a list
of filter commands to execute. The commands in the file
may be separated by newline characters and/or semicolons.
ifmt = <in-format>
(auto)
(the default),
then an attempt will be
made to detect the format of the table automatically.
This cannot always be done correctly however, in which case
the program will exit with an error explaining which
formats were attempted.
The same format parameter applies to all the tables
specified by in
.
[Default: (auto)
]
in = <table> [<table> ...]
A list of input table locations may be given in an external
file by using the indirction character '@'.
Thus "in=@filename
"
causes the file filename
to be read for a list
of input table locations. The locations in the file should
each be on a separate line.
istream = true|false
in
table
will be read as a stream.
It is necessary to give the
ifmt
parameter
in this case.
Depending on the required operations and processing mode,
this may cause the read to fail (sometimes it is necessary
to read the input table more than once).
It is not normally necessary to set this flag;
in most cases the data will be streamed automatically
if that is the best thing to do.
However it can sometimes result in less resource usage when
processing large files in certain formats (such as VOTable).
The same streaming flag applies to all the tables specified by
in
.
[Default: false
]
lazy = true|false
[Default: false
]
loccol = <colname>
ocmd = <cmds>
The value of this parameter is one or more of the filter commands described in Section 5.1. If more than one is given, they must be separated by semicolon characters (";"). This parameter can be repeated multiple times on the same command line to build up a list of processing steps. The sequence of commands given in this way defines the processing pipeline which is performed on the table.
Commands may alteratively be supplied in an external file,
by using the indirection character '@'.
Thus "ocmd=@filename
"
causes the file filename
to be read for a list
of filter commands to execute. The commands in the file
may be separated by newline characters and/or semicolons.
ofmt = <out-format>
(auto)
"
(the default),
then the output filename will be
examined to try to guess what sort of file is required
usually by looking at the extension.
If it's not obvious from the filename what output format is
intended, an error will result.
This parameter must only be given if
omode
has its default value of "out
".
[Default: (auto)
]
omode = <out-mode> <mode-args>
out
, which means that
the result will be written as a new table to disk or elsewhere,
as determined by the out
and ofmt
parameters.
However, there are other possibilities, which correspond
to uses to which a table can be put other than outputting it,
such as displaying metadata, calculating statistics,
or populating a table in an SQL database.
For some values of this parameter, additional parameters
(<mode-args>
)
are required to determine the exact behaviour.
Possible values are
out
meta
stats
count
cgi
discard
topcat
plastic
tosql
help=omode
flag
or see Section 5.4 for more information.
[Default: out
]
out = <out-table>
This parameter must only be given if
omode
has its default value of "out
".
[Default: -
]
seqcol = <colname>
uloccol = <colname>
Here are some examples of tcat
:
stilts tcat ifmt=ascii in=t1.txt in=t2.txt in=t3.txt out=table.txt
stilts tcat ifmt=ascii in="t1.txt t2.txt t3.txt" out=table.txt
stilts tcat ifmt=ascii in=@inlist out=table.txt
stilts tcat in=r368776.fits#1 in=r368776#2 in=r368776.fits#3 in=r368776.fits#4 out=r368776_all.fits icmd=progress seqcol=ID
in=r368776.fits#{1,2,3,4}
".
stilts tcat in='rA.csv rB.csv rC.csv' ifmt=csv \ icmd='keepcols "RA DEC FLUX"' icmd='sorthead 10 FLUX' \ ocmd='sort FLUX'
tcatn
: Concatenates multiple tables
tcatn
is a tool for concatenating a number of tables
one after the other. Each table can be manipulated separately
prior to the concatenatation.
If you have two tables T1 and T2 which contain similar columns, and you
want to treat them as a single table, you can use tcatn
to produce a new table whose metadata (row headings etc) comes from T1
and whose data consists of all the rows of T1 followed by all the rows of T2.
For this concatenation to make sense, each column of T1 must be
compatible with the corresponding column of T2 - they must have
compatible types and, presumably, meanings.
If this is not the case for the tables that you wish to concatenate,
for instance the columns are in different orders, or the units
differ between a column in T1 and its opposite number in T2,
you can use the icmd1
and/or icmd2
parameters to manipulate the input tables so that the
column sequences are compatible. See Appendix B.13.2 for
some examples.
If the tables are similar to each other
(same format, same columns, same preprocessing stages required if any),
you may find it easier to use tcat
instead.
The usage of tcatn
is
stilts <stilts-flags> tcatn nin=<count> ifmtN=<in-format> inN=<tableN> icmdN=<cmds> ocmd=<cmds> omode=<out-mode> <mode-args> out=<out-table> ofmt=<out-format> seqcol=<colname> loccol=<colname> uloccol=<colname> countrows=true|falseIf you don't have the
stilts
script installed,
write "java -jar stilts.jar
" instead of
"stilts
" - see Section 3.
The available <stilts-flags>
are listed
in Section 2.1.
Parameter values are assigned on the command line as explained in Section 2.3. They are as follows:
countrows = true|false
[Default: false
]
icmdN = <cmds>
The value of this parameter is one or more of the filter commands described in Section 5.1. If more than one is given, they must be separated by semicolon characters (";"). This parameter can be repeated multiple times on the same command line to build up a list of processing steps. The sequence of commands given in this way defines the processing pipeline which is performed on the table.
Commands may alteratively be supplied in an external file,
by using the indirection character '@'.
Thus "icmdN=@filename
"
causes the file filename
to be read for a list
of filter commands to execute. The commands in the file
may be separated by newline characters and/or semicolons.
ifmtN = <in-format>
(auto)
(the default),
then an attempt will be
made to detect the format of the table automatically.
This cannot always be done correctly however, in which case
the program will exit with an error explaining which
formats were attempted.
[Default: (auto)
]
inN = <tableN>
ifmtN
parameter.
loccol = <colname>
nin = <count>
ifmtN
, inN
and icmdN
.
ocmd = <cmds>
The value of this parameter is one or more of the filter commands described in Section 5.1. If more than one is given, they must be separated by semicolon characters (";"). This parameter can be repeated multiple times on the same command line to build up a list of processing steps. The sequence of commands given in this way defines the processing pipeline which is performed on the table.
Commands may alteratively be supplied in an external file,
by using the indirection character '@'.
Thus "ocmd=@filename
"
causes the file filename
to be read for a list
of filter commands to execute. The commands in the file
may be separated by newline characters and/or semicolons.
ofmt = <out-format>
(auto)
"
(the default),
then the output filename will be
examined to try to guess what sort of file is required
usually by looking at the extension.
If it's not obvious from the filename what output format is
intended, an error will result.
This parameter must only be given if
omode
has its default value of "out
".
[Default: (auto)
]
omode = <out-mode> <mode-args>
out
, which means that
the result will be written as a new table to disk or elsewhere,
as determined by the out
and ofmt
parameters.
However, there are other possibilities, which correspond
to uses to which a table can be put other than outputting it,
such as displaying metadata, calculating statistics,
or populating a table in an SQL database.
For some values of this parameter, additional parameters
(<mode-args>
)
are required to determine the exact behaviour.
Possible values are
out
meta
stats
count
cgi
discard
topcat
plastic
tosql
help=omode
flag
or see Section 5.4 for more information.
[Default: out
]
out = <out-table>
This parameter must only be given if
omode
has its default value of "out
".
[Default: -
]
seqcol = <colname>
uloccol = <colname>
Here are some examples of tcatn
:
stilts tcatn nin=2 in1=obs1.fits in2=obs2.fits out=combined.fits
stilts tcatn nin=3 omode=stats in1=obs1.txt ifmt1=ascii in2=obs2.xml ifmt2=votable in3=obs3.fit ifmt3=fits
stilts tcatn nin=2 in1=survey.vot.gz ifmt2=csv in2=more_data.csv icmd1='addskycoords fk5 galactic RA2000 DEC2000 GLON GLAT' \ icmd1='keepcols "OBJ_ID GLON GLAT"' \ icmd2='keepcols "ident gal_long gal_lat"' \ loccol=FILENAME omode=topcat
ifmt1
parameter is required since
VOTables can be detected automatically), and the other is a
comma-separated-values file (for which the ifmt2=csv
parameter must be given).
In the second place, the column structure of the two tables may be
quite different. By pre-processing the two tables using the
icmd1
& icmd2
parameters, we produce
in each case an input table which consists of three columns of
compatible types and meanings: an integer identifier and floating point
galactic longitude and latitude coordinates.
The second table contains such columns to start with,
but the first table requires an initial step to convert
FK5 J2000.0 coordinates to galactic ones.
tcatn
joins the two doctored tables together, to produce
a table which contains only these three columns, with all the rows
from both input tables, and sends the result directly
to a new or running instance of TOPCAT.
An additional column named FILENAME is appended to the table
before sending it; this contains "survey.vot.gz" for all the columns
from the first table and "more_data.csv" for all the columns from
the second one.
tcopy
: Converts between table formats
tcopy
is a table copying tool.
It simply copies a table from one place to another, but since
you can specify the input and output formats as desired, it works
as a converter from any of the supported
input formats
to any of the supported
output formats.
tcopy
is just a stripped-down version of
tpipe
- it doesn't do anything
that tpipe
can't, but the usage is slightly
simplified.
It is provided as a drop-in replacement for the old
tablecopy
(uk.ac.starlink.table.TableCopy
)
tool which was supplied with earlier versions of STIL and TOPCAT -
it has the same arguments and behaviour as tablecopy
,
but is implemented somewhat differently
and will in some cases be more efficient.
The usage of tcopy
is
stilts <stilts-flags> tcopy ifmt=<in-format> ofmt=<out-format> [in=]<table> [out=]<out-table>If you don't have the
stilts
script installed,
write "java -jar stilts.jar
" instead of
"stilts
" - see Section 3.
The available <stilts-flags>
are listed
in Section 2.1.
Parameter values are assigned on the command line as explained in Section 2.3. They are as follows:
ifmt = <in-format>
(auto)
(the default),
then an attempt will be
made to detect the format of the table automatically.
This cannot always be done correctly however, in which case
the program will exit with an error explaining which
formats were attempted.
[Default: (auto)
]
in = <table>
ifmt
parameter.
ofmt = <out-format>
(auto)
"
(the default),
then the output filename will be
examined to try to guess what sort of file is required
usually by looking at the extension.
If it's not obvious from the filename what output format is
intended, an error will result.
[Default: (auto)
]
out = <out-table>
[Default: -
]
Here are some examples of tcopy
in use:
stilts tcopy stars.fits stars.xml
stars.xml
filename is examined to make a guess at the kind of output to write:
the .xml
ending is taken to mean a TABLEDATA-encoded
VOTable.
stilts tcopy stars.fits stars.xml ifmt=fits ofmt=votable
stilts tcopy ofmt=text http://remote.host/data/vizer.xml.gz#4 -
#4
at the end of the URL
indicates that the data from the fifth TABLE
element
in the remote document are to be used. The gzip compression of
the table is taken care of automatically.
stilts tcopy ifmt=csv ofmt=latex spec.csv
stilts -classpath /usr/local/jars/pg73jdbc3.jar \ -Djdbc.drivers=org.postgresql.Driver \ tcopy in="jdbc:postgresql://localhost/imsim#SELECT ra, dec, Imag FROM dqc" \ ofmt=fits wfslist.cat
jdbc.drivers
system property.
As you can see, using SQL from Java is a bit fiddly,
and there are other ways to perform this
setup than on the command line - see Section 3.4
and tpipe
's
omode=tosql
output mode.
tcube
: Calculates N-dimensional histograms
tcube
constructs an N-dimensional histogram, or density map,
from N columns of an input table, and writes it out as an
N-dimensional data cube. The parameters you supply define which N
numeric columns of the input table you want to use and the dimensions
(bounds and pixel sizes) of the output grid.
Each table row then defines a point in N-dimensional space.
The program goes through each row, and if the point that row
defines falls within the bounds of the output grid you have defined,
increments the value associated with the corresponding pixel.
The resulting N-dimensional array, whose pixel values represent a
count of the rows associated with that region of the N-dimensional space,
is then written out as a FITS file.
In one dimension, this gives you a normal histogram of a given variable.
In two dimensions it might typically be used to plot the density on
the sky of objects from a catalogue.
As with some of the other generic table commands,
you can perform extensive pre-processing on the input table by
use of the icmd
parameter before the actual cube
counts are calculated.
The usage of tcube
is
stilts <stilts-flags> tcube cols=<col-id> ... ifmt=<in-format> istream=true|false icmd=<cmds> bounds=[<lo>]:[<hi>] ... binsizes=<size> ... nbins=<num> ... out=<out-file> otype=byte|short|int|long|float|double scale=<col-id> [in=]<table>If you don't have the
stilts
script installed,
write "java -jar stilts.jar
" instead of
"stilts
" - see Section 3.
The available <stilts-flags>
are listed
in Section 2.1.
Parameter values are assigned on the command line as explained in Section 2.3. They are as follows:
binsizes = <size> ...
nbins
parameter
must be supplied.
bounds = [<lo>]:[<hi>] ...
If any of the bounds need to be determined automatically in this way, two passes through the data will be required, the first to determine bounds and the second to populate the cube.
cols = <col-id> ...
<col-id>
elements,
separated by spaces, should be given.
Each one represents a column in the table, using either its
name or index.
The number of columns listed in the value of this parameter defines the dimensionality of the output data cube.
icmd = <cmds>
The value of this parameter is one or more of the filter commands described in Section 5.1. If more than one is given, they must be separated by semicolon characters (";"). This parameter can be repeated multiple times on the same command line to build up a list of processing steps. The sequence of commands given in this way defines the processing pipeline which is performed on the table.
Commands may alteratively be supplied in an external file,
by using the indirection character '@'.
Thus "icmd=@filename
"
causes the file filename
to be read for a list
of filter commands to execute. The commands in the file
may be separated by newline characters and/or semicolons.
ifmt = <in-format>
(auto)
(the default),
then an attempt will be
made to detect the format of the table automatically.
This cannot always be done correctly however, in which case
the program will exit with an error explaining which
formats were attempted.
[Default: (auto)
]
in = <table>
ifmt
parameter.
istream = true|false
in
table
will be read as a stream.
It is necessary to give the
ifmt
parameter
in this case.
Depending on the required operations and processing mode,
this may cause the read to fail (sometimes it is necessary
to read the input table more than once).
It is not normally necessary to set this flag;
in most cases the data will be streamed automatically
if that is the best thing to do.
However it can sometimes result in less resource usage when
processing large files in certain formats (such as VOTable).
[Default: false
]
nbins = <num> ...
binsizes
parameter
must be supplied.
otype = byte|short|int|long|float|double
out = <out-file>
The output cube is currently written as a single-HDU FITS file.
[Default: -
]
scale = <col-id>
null
(the default) then for each
row that falls within the bounds of a pixel, the pixel value
will be incremented by 1.
If a column ID is given, then instead of 1 being added,
the value of that column for the row in question is added.
The effect of this is that the output image contains the mean
of the given column for the rows corresponding to each pixel
rather than just a count of them.
stilts tcube in=2QZ_6QZ_pubcat.fits out=ccm.fits \ cols='Bj_R U_Bj Bj' binsizes='0.05 0.05 0.5' bounds='-2:1 -3:2 :'
stilts tcube in=iras_psc.vot out=iras_psc_map.fits \ icmd='addskycoords fk5 galactic ra dec glat glon' \ cols='glat glon' nbins='400 200'
addskycoords
filter is used to preprocess the data before the cube generation
step (see Section 5.1).
tjoin
: Joins multiple tables side-to-side
tjoin
performs a trivial side-by-side join of multiple tables.
The N'th row of the output table consists of the N'th row of the
first input table, followed by the N'th row of the second input table, ...
and so on. It is suitable if you want to amalgamate two or more
tables whose row orderings correspond exactly to each other.
For the (more usual) case in which the rows of the tables to be joined are not already in the right order, use one of the crossmatching commands.
The usage of tjoin
is
stilts <stilts-flags> tjoin nin=<count> ifmtN=<in-format> inN=<tableN> icmdN=<cmds> ocmd=<cmds> omode=<out-mode> <mode-args> out=<out-table> ofmt=<out-format> fixcols=none|dups|all suffixN=<label>If you don't have the
stilts
script installed,
write "java -jar stilts.jar
" instead of
"stilts
" - see Section 3.
The available <stilts-flags>
are listed
in Section 2.1.
Parameter values are assigned on the command line as explained in Section 2.3. They are as follows:
fixcols = none|dups|all
none
: columns are not renameddups
: columns which would otherwise have duplicate names in the output will be renamed to indicate which table they came fromall
: all columns will be renamed to indicate which table they came fromsuffix*
parameters.
[Default: dups
]
icmdN = <cmds>
The value of this parameter is one or more of the filter commands described in Section 5.1. If more than one is given, they must be separated by semicolon characters (";"). This parameter can be repeated multiple times on the same command line to build up a list of processing steps. The sequence of commands given in this way defines the processing pipeline which is performed on the table.
Commands may alteratively be supplied in an external file,
by using the indirection character '@'.
Thus "icmdN=@filename
"
causes the file filename
to be read for a list
of filter commands to execute. The commands in the file
may be separated by newline characters and/or semicolons.
ifmtN = <in-format>
(auto)
(the default),
then an attempt will be
made to detect the format of the table automatically.
This cannot always be done correctly however, in which case
the program will exit with an error explaining which
formats were attempted.
[Default: (auto)
]
inN = <tableN>
ifmtN
parameter.
nin = <count>
ifmtN
, inN
and icmdN
.
ocmd = <cmds>
The value of this parameter is one or more of the filter commands described in Section 5.1. If more than one is given, they must be separated by semicolon characters (";"). This parameter can be repeated multiple times on the same command line to build up a list of processing steps. The sequence of commands given in this way defines the processing pipeline which is performed on the table.
Commands may alteratively be supplied in an external file,
by using the indirection character '@'.
Thus "ocmd=@filename
"
causes the file filename
to be read for a list
of filter commands to execute. The commands in the file
may be separated by newline characters and/or semicolons.
ofmt = <out-format>
(auto)
"
(the default),
then the output filename will be
examined to try to guess what sort of file is required
usually by looking at the extension.
If it's not obvious from the filename what output format is
intended, an error will result.
This parameter must only be given if
omode
has its default value of "out
".
[Default: (auto)
]
omode = <out-mode> <mode-args>
out
, which means that
the result will be written as a new table to disk or elsewhere,
as determined by the out
and ofmt
parameters.
However, there are other possibilities, which correspond
to uses to which a table can be put other than outputting it,
such as displaying metadata, calculating statistics,
or populating a table in an SQL database.
For some values of this parameter, additional parameters
(<mode-args>
)
are required to determine the exact behaviour.
Possible values are
out
meta
stats
count
cgi
discard
topcat
plastic
tosql
help=omode
flag
or see Section 5.4 for more information.
[Default: out
]
out = <out-table>
This parameter must only be given if
omode
has its default value of "out
".
[Default: -
]
suffixN = <label>
fixcols
parameter
is set so that input columns are renamed for insertion into
the output table, this parameter determines how the
renaming is done.
It gives a suffix which is appended to all renamed columns
from table N.
[Default: _N
]
Here are some examples of using tjoin
stilts tjoin nin=2 in1=positions.fit in2=fluxes.fits out=combined.fits
stilts tjoin nin=3 fixcols=all \ ifmt1=ascii in1=t1.txt suffix1=_T1 \ ifmt2=ascii in2=t2.txt suffix2=_T2 \ ifmt3=ascii in3=t3.txt suffix3=_T3 \ ocmd='select FLAG_T1==0' \ omode=stats
tmatch1
: Performs a crossmatch internal to a single table
tmatch1
performs efficient and flexible
crossmatching between the rows of a single table.
It can match rows on the basis of their relative position in the sky,
or alternatively using many other criteria such as separation in
in some isotropic or anisotropic Cartesian space,
identity of a key value, or some combination of these;
the full range of match criteria is dicussed in Section 6.1.
The basic task performed by the intra-table matcher is to identify
groups of rows within the table which match each other.
See Section 6.2 for an explanation of exactly what
consitutes a match group.
The result of identifying these groups is expressed as an output table
in one of a variety of ways, specified by the action
parameter.
These options include marking group membership in added columns
and eliminating some or all rows which form part of a match group.
The usage of tmatch1
is
stilts <stilts-flags> tmatch1 matcher=<matcher-name> params=<match-params> values=<expr-list> action=identify|keep0|keep1|wide2|wideN progress=true|false ifmt=<in-format> istream=true|false icmd=<cmds> ocmd=<cmds> omode=<out-mode> <mode-args> out=<out-table> ofmt=<out-format> [in=]<table>If you don't have the
stilts
script installed,
write "java -jar stilts.jar
" instead of
"stilts
" - see Section 3.
The available <stilts-flags>
are listed
in Section 2.1.
Parameter values are assigned on the command line as explained in Section 2.3. They are as follows:
action = identify|keep0|keep1|wide2|wideN
identify
:
The output table is the same as the input table except that
it contains two additional columns,
GroupID
and
GroupSize
,
following the input columns.
Each group of rows which matched is assigned a unique integer,
recorded in the GroupID column,
and the size of each group is recorded in the GroupSize
column.
Rows which don't match any others (singles) have null values in
both these columns.
keep0
:
The result is a new table containing only "single" rows,
that is ones which don't match any other rows in the table.
Any other rows are thrown out.
keep1
:
The result is a new table in which only one row
(the first in the input table order)
from each group of matching ones is retained.
A subsequent intra-table match with the same criteria
would therefore show no matches.
wideN
:
The result is a new "wide" table consisting of matched rows in
the input table stacked next to each other.
Only groups of exactly N rows in the input table are used to
form the output table; each row of the output table consists of
the columns of the first group member, followed by the columns of
the second group member and so on.
The output table therefore has
N times as many columns as the input table.
The column names in the new table have
_1
, _2
, ...
appended to them to avoid duplication.
[Default: identify
]
icmd = <cmds>
The value of this parameter is one or more of the filter commands described in Section 5.1. If more than one is given, they must be separated by semicolon characters (";"). This parameter can be repeated multiple times on the same command line to build up a list of processing steps. The sequence of commands given in this way defines the processing pipeline which is performed on the table.
Commands may alteratively be supplied in an external file,
by using the indirection character '@'.
Thus "icmd=@filename
"
causes the file filename
to be read for a list
of filter commands to execute. The commands in the file
may be separated by newline characters and/or semicolons.
ifmt = <in-format>
(auto)
(the default),
then an attempt will be
made to detect the format of the table automatically.
This cannot always be done correctly however, in which case
the program will exit with an error explaining which
formats were attempted.
[Default: (auto)
]
in = <table>
ifmt
parameter.
istream = true|false
in
table
will be read as a stream.
It is necessary to give the
ifmt
parameter
in this case.
Depending on the required operations and processing mode,
this may cause the read to fail (sometimes it is necessary
to read the input table more than once).
It is not normally necessary to set this flag;
in most cases the data will be streamed automatically
if that is the best thing to do.
However it can sometimes result in less resource usage when
processing large files in certain formats (such as VOTable).
[Default: false
]
matcher = <matcher-name>
params
and
values*
parameter(s).
[Default: sky
]
ocmd = <cmds>
The value of this parameter is one or more of the filter commands described in Section 5.1. If more than one is given, they must be separated by semicolon characters (";"). This parameter can be repeated multiple times on the same command line to build up a list of processing steps. The sequence of commands given in this way defines the processing pipeline which is performed on the table.
Commands may alteratively be supplied in an external file,
by using the indirection character '@'.
Thus "ocmd=@filename
"
causes the file filename
to be read for a list
of filter commands to execute. The commands in the file
may be separated by newline characters and/or semicolons.
ofmt = <out-format>
(auto)
"
(the default),
then the output filename will be
examined to try to guess what sort of file is required
usually by looking at the extension.
If it's not obvious from the filename what output format is
intended, an error will result.
This parameter must only be given if
omode
has its default value of "out
".
[Default: (auto)
]
omode = <out-mode> <mode-args>
out
, which means that
the result will be written as a new table to disk or elsewhere,
as determined by the out
and ofmt
parameters.
However, there are other possibilities, which correspond
to uses to which a table can be put other than outputting it,
such as displaying metadata, calculating statistics,
or populating a table in an SQL database.
For some values of this parameter, additional parameters
(<mode-args>
)
are required to determine the exact behaviour.
Possible values are
out
meta
stats
count
cgi
discard
topcat
plastic
tosql
help=omode
flag
or see Section 5.4 for more information.
[Default: out
]
out = <out-table>
This parameter must only be given if
omode
has its default value of "out
".
[Default: -
]
params = <match-params>
matcher
parameter.
If it contains multiple values, they must be separated by spaces;
values which contain a space can be 'quoted' or "quoted".
progress = true|false
[Default: true
]
values = <expr-list>
matcher
.
Depending on the kind of match, the number and type of
the values required will be different.
Multiple values should be separated by whitespace;
if whitespace occurs within a single value it must be
'quoted' or "quoted".
Elements of the expression list are commonly just column
names, but may be algebraic expressions calculated from
zero or more columns as explained in Section 8.
Here are some examples of using tmatch1
stilts tmatch1 matcher=sky values="RA2000 DE2000" params=20 \ action=keep0 in=crowded.vot out=sparse.vot
stilts tmatch1 matcher=skyerr values="RA2000 DE2000 RADIUS*4" params=100 \ action=keep0 in=crowded.vot out=sparse.vot
skyerr
matcher which determines the proximity threshold
on a row-by-row basis from values in the table - in this case 4 times
the value of the RADIUS column (this value must be in arc seconds).
It is still necessary to specify params=60
, but here it
just gives an upper limit for 4*RADIUS for all rows - this value is
required by the algorithm, and should not be too large for performance
reasons, but as long as it's set high enough it will
not affect the result.
stilts tmatch1 matcher=3d values="XPIX YPIX ZPIX" params=10 action=identify \ in=state.fit ocmd='select GroupSize>3' out=groups3+.fit
action=identify
parameter means that the input table
is written out with the same rows, but with additional columns indicating
which rows are associated with each other. One of these columns,
"GroupSize" gives the number of objects in each group.
The postprocessing filter ocmd='select GroupSize>3'
selects only those rows which are part of groups of three objects or
larger; singletons and pairs are discarded before writing the output file.
stilts tmatch1 matcher=sky values="ra dec" params=3 action=wide2 \ ocmd='keepcols "id_1 ra_1 dec_1 id_2 ra_2 dec_2"' in=galaxy.fits out=binaries.txt ofmt=ascii
ocmd
post-processing filter discards all of the
columns except the identifiers and sky positions for each object.
The output is to a text file.
tmatch2
: Crossmatches 2 tables using flexible criteria
tmatch2
is an efficient and highly configurable
tool for crossmatching pairs of tables.
It can match rows between tables on the basis of their relative position
in the sky, or alternatively using many other criteria such as
separation in some isotropic or anisotropic Cartesian space,
identity of a key value, or some combination of these;
the full range of match criteria is discussed in Section 6.1.
You can choose whether you want to identify all the matches or
only the closest,
and what form the output table takes, for instance matched rows only,
or all rows from one or both tables, or only the unmatched rows.
If you simply want to match two tables based on sky position with
a fixed maximum separation, you may find the
tskymatch2
command easier to use.
Note: the duptag1
and duptag2
parameters have been replaced at version 1.4 by
suffix1
and suffix2
for consistency with other table join tasks.
The usage of tmatch2
is
stilts <stilts-flags> tmatch2 ifmt1=<in-format> ifmt2=<in-format> icmd1=<cmds> icmd2=<cmds> ocmd=<cmds> omode=<out-mode> <mode-args> out=<out-table> ofmt=<out-format> matcher=<matcher-name> values1=<expr-list> values2=<expr-list> params=<match-params> join=1and2|1or2|all1|all2|1not2|2not1|1xor2 find=best|all fixcols=none|dups|all suffix1=<label> suffix2=<label> scorecol=<col-name> progress=true|false [in1=]<table1> [in2=]<table2>If you don't have the
stilts
script installed,
write "java -jar stilts.jar
" instead of
"stilts
" - see Section 3.
The available <stilts-flags>
are listed
in Section 2.1.
Parameter values are assigned on the command line as explained in Section 2.3. They are as follows:
find = best|all
best
is selected,
then only the best match
between the two tables will be retained; in this case
the data from a row of either input table will appear in
at most one row of the output table.
If all
is selected, then all pairs of rows
from the two input tables which match the input criteria
will be represented in the output table.
[Default: best
]
fixcols = none|dups|all
none
: columns are not renameddups
: columns which would otherwise have duplicate names in the output will be renamed to indicate which table they came fromall
: all columns will be renamed to indicate which table they came fromsuffix*
parameters.
[Default: dups
]
icmd1 = <cmds>
The value of this parameter is one or more of the filter commands described in Section 5.1. If more than one is given, they must be separated by semicolon characters (";"). This parameter can be repeated multiple times on the same command line to build up a list of processing steps. The sequence of commands given in this way defines the processing pipeline which is performed on the table.
Commands may alteratively be supplied in an external file,
by using the indirection character '@'.
Thus "icmd1=@filename
"
causes the file filename
to be read for a list
of filter commands to execute. The commands in the file
may be separated by newline characters and/or semicolons.
icmd2 = <cmds>
The value of this parameter is one or more of the filter commands described in Section 5.1. If more than one is given, they must be separated by semicolon characters (";"). This parameter can be repeated multiple times on the same command line to build up a list of processing steps. The sequence of commands given in this way defines the processing pipeline which is performed on the table.
Commands may alteratively be supplied in an external file,
by using the indirection character '@'.
Thus "icmd2=@filename
"
causes the file filename
to be read for a list
of filter commands to execute. The commands in the file
may be separated by newline characters and/or semicolons.
ifmt1 = <in-format>
(auto)
(the default),
then an attempt will be
made to detect the format of the table automatically.
This cannot always be done correctly however, in which case
the program will exit with an error explaining which
formats were attempted.
[Default: (auto)
]
ifmt2 = <in-format>
(auto)
(the default),
then an attempt will be
made to detect the format of the table automatically.
This cannot always be done correctly however, in which case
the program will exit with an error explaining which
formats were attempted.
[Default: (auto)
]
in1 = <table1>
ifmt1
parameter.
in2 = <table2>
ifmt2
parameter.
join = 1and2|1or2|all1|all2|1not2|2not1|1xor2
1and2
: An output row for each row represented in both input tables1or2
: An output row for each row represented in either or both of the input tablesall1
: An output row for each matched or unmatched row in table 1all2
: An output row for each matched or unmatched row in table 21not2
: An output row only for rows which appear in the first table but are not matched in the second table2not1
: An output row only for rows which appear in the second table but are not matched in the first table1xor2
: An output row only for rows represented in one of the input tables but not the other one[Default: 1and2
]
matcher = <matcher-name>
params
and
values*
parameter(s).
[Default: sky
]
ocmd = <cmds>
The value of this parameter is one or more of the filter commands described in Section 5.1. If more than one is given, they must be separated by semicolon characters (";"). This parameter can be repeated multiple times on the same command line to build up a list of processing steps. The sequence of commands given in this way defines the processing pipeline which is performed on the table.
Commands may alteratively be supplied in an external file,
by using the indirection character '@'.
Thus "ocmd=@filename
"
causes the file filename
to be read for a list
of filter commands to execute. The commands in the file
may be separated by newline characters and/or semicolons.
ofmt = <out-format>
(auto)
"
(the default),
then the output filename will be
examined to try to guess what sort of file is required
usually by looking at the extension.
If it's not obvious from the filename what output format is
intended, an error will result.
This parameter must only be given if
omode
has its default value of "out
".
[Default: (auto)
]
omode = <out-mode> <mode-args>
out
, which means that
the result will be written as a new table to disk or elsewhere,
as determined by the out
and ofmt
parameters.
However, there are other possibilities, which correspond
to uses to which a table can be put other than outputting it,
such as displaying metadata, calculating statistics,
or populating a table in an SQL database.
For some values of this parameter, additional parameters
(<mode-args>
)
are required to determine the exact behaviour.
Possible values are
out
meta
stats
count
cgi
discard
topcat
plastic
tosql
help=omode
flag
or see Section 5.4 for more information.
[Default: out
]
out = <out-table>
This parameter must only be given if
omode
has its default value of "out
".
[Default: -
]
params = <match-params>
matcher
parameter.
If it contains multiple values, they must be separated by spaces;
values which contain a space can be 'quoted' or "quoted".
progress = true|false
[Default: true
]
scorecol = <col-name>
matcher
,
but it typically represents a distance of some kind between
the two matching points.
If a null value is chosen, no score column will be inserted
in the output table.
The default value of this parameter depends on
matcher
.
[Default: Score
]
suffix1 = <label>
fixcols
parameter
is set so that input columns are renamed for insertion into
the output table, this parameter determines how the
renaming is done.
It gives a suffix which is appended to all renamed columns
from table 1.
[Default: _1
]
suffix2 = <label>
fixcols
parameter
is set so that input columns are renamed for insertion into
the output table, this parameter determines how the
renaming is done.
It gives a suffix which is appended to all renamed columns
from table 2.
[Default: _2
]
values1 = <expr-list>
matcher
.
Depending on the kind of match, the number and type of
the values required will be different.
Multiple values should be separated by whitespace;
if whitespace occurs within a single value it must be
'quoted' or "quoted".
Elements of the expression list are commonly just column
names, but may be algebraic expressions calculated from
zero or more columns as explained in Section 8.
values2 = <expr-list>
matcher
.
Depending on the kind of match, the number and type of
the values required will be different.
Multiple values should be separated by whitespace;
if whitespace occurs within a single value it must be
'quoted' or "quoted".
Elements of the expression list are commonly just column
names, but may be algebraic expressions calculated from
zero or more columns as explained in Section 8.
Here are some examples of using tmatch2
stilts tmatch2 in1=obs_v.xml in2=obs_i.xml out=obs_iv.xml \ matcher=sky values1="ra dec" values2="ra dec" params="2"
stilts tmatch2 survey.fits ifmt2=csv mycat.csv \ icmd1='addskycoords fk4 fk5 RA1950 DEC1950 RA2000 DEC2000' \ matcher=skyerr \ params=10 values1="RA2000 DEC2000 POS_ERR" values2="RA DEC 0" \ join=2not1 omode=count
skyerr
matcher is
used, which takes account of this; the third entry in the
values1
parameter is the POS_ERR column (in arcsec).
Since the second input table has no positional uncertainty information,
0 is used as the third entry in values2
.
The params
still has to contain a value which gives the
maximum error for matching (i.e. >= the largest value in the
POS_ERR column).
The join type is 2not1
, which means the output table
will only contain those entries which are in the second input table
but not in the first one.
The output table is not stored, but the number of rows it contains
(the number of objects represented in the CSV file but not the survey)
is written to the screen.
stilts tmatch2 ifmt1=ascii ifmt2=ascii in1=cat-a.txt in2=cat-b.txt \ matcher=2d values1='X Y' values2='X Y' params=5 join=1and2 \ suffix1=_a suffix2=_b \ ocmd='addcol XDIFF X_a-X_b; addcol YDIFF Y_a-Y_b' \ ocmd'keepcols "XDIFF YDIFF"' omode=stats
suffix*
parameters describe
how the input X and Y columns are to be renamed to avoid duplicate
column names in the output table.
To this result are added two new columns,
representing the X and Y positional
difference between the rows from one input table and those from the other.
The keepcols
filter then throws all the other columns away,
retaining only these difference columns.
The final two-column table is not stored anywhere,
but (omode=stats
)
statistics including mean and standard deviation
are calculated on its columns and displayed to the screen.
Having done all this, you can examine the average X and Y differences
between the two input tables for matched rows, and if they differ
significantly from zero, you can conclude that there is a systematic
error between the positions in the two input files.
stilts tmatch2 in1=mgc.fits in2=6dfgs.xml join=1and2 find=all \ matcher=sky+1d params='3 0.5' \ values1='ra dec bmag' values2='RA2000 DEC2000 B_MAG" \ out=pairs.fits
sky
and 1d
match criteria. This means that the only
rows which match are those which are
both within 3 arcsec of each other on the sky
and and within 0.5 blue magnitudes.
Note that for both the params
and the
values1
and values2
parameters,
the items for the sky
matcher (RA and DEC)
are listed first,
followed by those for the 1d
matcher (in this case,
blue magnitude).
tmatchn
: Crossmatches multiple tables using flexible criteria
tmatchn
performs efficient and flexible crossmatching
between multiple tables.
It can match rows on the basis of their relative position in the sky,
or alternatively using many other criteria such as separation in
in some isotropic or anisotropic Cartesian space,
identity of a key value, or some combination of these;
the full range of match criteria is dicussed in Section 6.1.
Since the match criteria define what counts as a match between two
objects, it is not immediately obvious what is meant by a multi-table
match. In fact the command can work in one of two distinct modes,
controlled by the multimode
parameter.
In pairs
mode, one table (by default the first input table)
is designated the reference table, and pair matches between each of the
other tables and that one are identified.
In group
mode groups of objects from all the input tables are
identified, as discussed in Section 6.2.
Currently, in both cases an output matched row cannot contain more
than one object from each input table.
Options for output of multiple rows per input table per match may be
forthcoming in future releases if there is demand.
tmatchn
is intended for use with more than two input tables -
see tmatch1
and tmatch2
for 1- and 2-table crossmatching respectively.
The usage of tmatchn
is
stilts <stilts-flags> tmatchn nin=<count> ifmtN=<in-format> inN=<tableN> icmdN=<cmds> ocmd=<cmds> omode=<out-mode> <mode-args> out=<out-table> ofmt=<out-format> multimode=pairs|group iref=<table-index> matcher=<matcher-name> params=<match-params> valuesN=<expr-list> joinN=default|match|nomatch|always fixcols=none|dups|all suffixN=<label> progress=true|falseIf you don't have the
stilts
script installed,
write "java -jar stilts.jar
" instead of
"stilts
" - see Section 3.
The available <stilts-flags>
are listed
in Section 2.1.
Parameter values are assigned on the command line as explained in Section 2.3. They are as follows:
fixcols = none|dups|all
none
: columns are not renameddups
: columns which would otherwise have duplicate names in the output will be renamed to indicate which table they came fromall
: all columns will be renamed to indicate which table they came fromsuffix*
parameters.
[Default: dups
]
icmdN = <cmds>
The value of this parameter is one or more of the filter commands described in Section 5.1. If more than one is given, they must be separated by semicolon characters (";"). This parameter can be repeated multiple times on the same command line to build up a list of processing steps. The sequence of commands given in this way defines the processing pipeline which is performed on the table.
Commands may alteratively be supplied in an external file,
by using the indirection character '@'.
Thus "icmdN=@filename
"
causes the file filename
to be read for a list
of filter commands to execute. The commands in the file
may be separated by newline characters and/or semicolons.
ifmtN = <in-format>
(auto)
(the default),
then an attempt will be
made to detect the format of the table automatically.
This cannot always be done correctly however, in which case
the program will exit with an error explaining which
formats were attempted.
[Default: (auto)
]
inN = <tableN>
ifmtN
parameter.
iref = <table-index>
multimode
=pairs
this parameter gives the index of the table in the input table
list which is to serve as the reference table
(the one which must be matched by other tables).
Ignored in other modes.
Row ordering in the output table is usually tidiest if the default setting of 1 is used (i.e. if the first input table is used as the reference table).
[Default: 1
]
joinN = default|match|nomatch|always
The default behaviour is that a row will appear in the output table if it represents a match of rows from two or more of the input tables. This can be altered on a per-input-table basis however by choosing one of the non-default options below:
match
:
Rows are included only if they contain an entry from
input table N.
nomatch
:
Rows are included only if they do not contain an entry from
input table N.
always
:
Rows are included if they contain an entry from
input table N
(overrides any match and nomatch
settings of other tables).
default
:
Input table N has no special effect on
whether rows are included.
[Default: default
]
matcher = <matcher-name>
params
and
values*
parameter(s).
[Default: sky
]
multimode = pairs|group
pairs
:
Each output row corresponds to a single row of the
reference table
(see parameter iref
)
and contains entries from other tables which are pair matches
to that.
If a reference table row matches multiple rows from one of
the other tables, only the best one is included.
group
:
Each output row corresponds to a group of entries from the
input tables which are
mutually linked by pair matches between them.
This means that although you can get from any entry to any
other entry via one or more pair matches,
there is no guarantee that any entry
is a pair match with any other entry.
No table has privileged status in this case.
If there are multiple entries from a given table in the
match group, an arbitrary one is chosen for inclusion
(there is no unique way to select the best).
See Section 6.2 for more discussion.
[Default: pairs
]
nin = <count>
ifmtN
, inN
and icmdN
.
ocmd = <cmds>
The value of this parameter is one or more of the filter commands described in Section 5.1. If more than one is given, they must be separated by semicolon characters (";"). This parameter can be repeated multiple times on the same command line to build up a list of processing steps. The sequence of commands given in this way defines the processing pipeline which is performed on the table.
Commands may alteratively be supplied in an external file,
by using the indirection character '@'.
Thus "ocmd=@filename
"
causes the file filename
to be read for a list
of filter commands to execute. The commands in the file
may be separated by newline characters and/or semicolons.
ofmt = <out-format>
(auto)
"
(the default),
then the output filename will be
examined to try to guess what sort of file is required
usually by looking at the extension.
If it's not obvious from the filename what output format is
intended, an error will result.
This parameter must only be given if
omode
has its default value of "out
".
[Default: (auto)
]
omode = <out-mode> <mode-args>
out
, which means that
the result will be written as a new table to disk or elsewhere,
as determined by the out
and ofmt
parameters.
However, there are other possibilities, which correspond
to uses to which a table can be put other than outputting it,
such as displaying metadata, calculating statistics,
or populating a table in an SQL database.
For some values of this parameter, additional parameters
(<mode-args>
)
are required to determine the exact behaviour.
Possible values are
out
meta
stats
count
cgi
discard
topcat
plastic
tosql
help=omode
flag
or see Section 5.4 for more information.
[Default: out
]
out = <out-table>
This parameter must only be given if
omode
has its default value of "out
".
[Default: -
]
params = <match-params>
matcher
parameter.
If it contains multiple values, they must be separated by spaces;
values which contain a space can be 'quoted' or "quoted".
progress = true|false
[Default: true
]
suffixN = <label>
fixcols
parameter
is set so that input columns are renamed for insertion into
the output table, this parameter determines how the
renaming is done.
It gives a suffix which is appended to all renamed columns
from table N.
[Default: _N
]
valuesN = <expr-list>
matcher
.
Depending on the kind of match, the number and type of
the values required will be different.
Multiple values should be separated by whitespace;
if whitespace occurs within a single value it must be
'quoted' or "quoted".
Elements of the expression list are commonly just column
names, but may be algebraic expressions calculated from
zero or more columns as explained in Section 8.
Here are some examples of using tmatchn
stilts tmatchn multimode=pairs nin=4 matcher=sky params=5 \ in1=transients.txt ifmt1=ascii values1='alpha delta' \ in2=2mass_virgo.fits values2='ra2000 dec2000' \ in3=sdss_virgo.fits values3='ra dec' \ in4=first_virgo.fits values4='pos_eq_ra pos_eq_dec' \ out=matches.xml ofmt=votable-binary
stilts tmatchn multimode=pairs nin=4 matcher=sky params=5 \ in1=transients.txt ifmt1=ascii suffix1='_t' values1='alpha delta' \ in2=2mass_virgo.fits suffix2='_2mass' values2='ra2000 dec2000' \ in3=sdss_virgo.fits suffix3='_sdss' values3='ra dec' \ in4=first_virgo.fits suffix4='_first' values4='pos_eq_ra pos_eq_dec' \ fixcols=all join1=all \ ocmd='keepcols "*_t designation_2mass SDSSName_sdss id_field_first"' \ out=matches.xml ofmt=votable-binary
fixcols=all
and
suffixN
assignments mean that all the columns from the
input tables will be renamed for output by adding the given suffixes.
The keepcols
filter applied to the output table throws out
all the columns except the ones from the reference table (*_t
)
and one column from each of the other table giving object identifiers.
This output table will probably be easier to read (though contain less
information) than that from the previous example).
Additionally, the join1=all
assignment means that the output
table will have one row for each row of the reference table
(transients.txt), even if no matches are found for it.
stilts tmatchn multimode=group nin=3 matcher=skyerr params=8 \ in1=Hband.fits values='RA DEC SEEING*2' \ in2=Jband.fits values='RA DEC SEEING*2' \ in3=Kband.fits values='RA DEC SEEING*2' \ omode=topcat
tpipe
: Performs pipeline processing on a table
tpipe
performs all kinds of general purpose manipulations
which take one table as input.
It is extremely flexible, and can do the following things
amongst others:
The basic operation of tpipe
is that it reads an
input table, performs zero or more processing steps on it,
and then does something with the output. There are therefore
three classes of things you need to tell it when it runs:
in
, ifmt
and
istream
parameters.
cmd
parameters, or the name of a file
containing the steps using the script
parameter.
The steps that you can perform are described in
Section 5.1.
omode
parameter.
By default, omode=out
,
in which case the table is written to a new table file in a format
determined by ofmt
. However, you can do other things
with the result such as
calculate the per-column statistics (omode=stats
),
view only the table and column metadata (omode=meta
),
display it directly in TOPCAT (omode=topcat
) etc.
The parameters mentioned above are listed in detail in the next section.
The usage of tpipe
is
stilts <stilts-flags> tpipe ifmt=<in-format> istream=true|false cmd=<cmds> omode=<out-mode> <mode-args> out=<out-table> ofmt=<out-format> [in=]<table>If you don't have the
stilts
script installed,
write "java -jar stilts.jar
" instead of
"stilts
" - see Section 3.
The available <stilts-flags>
are listed
in Section 2.1.
Parameter values are assigned on the command line as explained in Section 2.3. They are as follows:
cmd = <cmds>
The value of this parameter is one or more of the filter commands described in Section 5.1. If more than one is given, they must be separated by semicolon characters (";"). This parameter can be repeated multiple times on the same command line to build up a list of processing steps. The sequence of commands given in this way defines the processing pipeline which is performed on the table.
Commands may alteratively be supplied in an external file,
by using the indirection character '@'.
Thus "icmd=@filename
"
causes the file filename
to be read for a list
of filter commands to execute. The commands in the file
may be separated by newline characters and/or semicolons.
ifmt = <in-format>
(auto)
(the default),
then an attempt will be
made to detect the format of the table automatically.
This cannot always be done correctly however, in which case
the program will exit with an error explaining which
formats were attempted.
[Default: (auto)
]
in = <table>
ifmt
parameter.
istream = true|false
in
table
will be read as a stream.
It is necessary to give the
ifmt
parameter
in this case.
Depending on the required operations and processing mode,
this may cause the read to fail (sometimes it is necessary
to read the input table more than once).
It is not normally necessary to set this flag;
in most cases the data will be streamed automatically
if that is the best thing to do.
However it can sometimes result in less resource usage when
processing large files in certain formats (such as VOTable).
[Default: false
]
ofmt = <out-format>
(auto)
"
(the default),
then the output filename will be
examined to try to guess what sort of file is required
usually by looking at the extension.
If it's not obvious from the filename what output format is
intended, an error will result.
This parameter must only be given if
omode
has its default value of "out
".
[Default: (auto)
]
omode = <out-mode> <mode-args>
out
, which means that
the result will be written as a new table to disk or elsewhere,
as determined by the out
and ofmt
parameters.
However, there are other possibilities, which correspond
to uses to which a table can be put other than outputting it,
such as displaying metadata, calculating statistics,
or populating a table in an SQL database.
For some values of this parameter, additional parameters
(<mode-args>
)
are required to determine the exact behaviour.
Possible values are
out
meta
stats
count
cgi
discard
topcat
plastic
tosql
help=omode
flag
or see Section 5.4 for more information.
[Default: out
]
out = <out-table>
This parameter must only be given if
omode
has its default value of "out
".
[Default: -
]
Here are some examples of tpipe
in use with explanations
of what's going on. For simplicity these examples assume that you have the
stilts
script installed and are using a Unix-like shell;
see Section 3 for an explanation of how to invoke the command
if you just have the Java classes.
stilts tpipe cat.fits
omode=out
is assumed,
and output is to standard output in text
format.
stilts tpipe cmd='head 5' cat.fits.gz
stilts tpipe ifmt=csv xxx.csv \ cmd='keepcols "index ra dec"' \ omode=out ofmt=fits xxx.fits
cmd
argument: the outer quotes
are so that the argument of the cmd
parameter itself
(keepcols "index ra dec"
)
is not split up by spaces (to protect it from the shell),
and the inner quotes are to keep the
colid-list
argument of the
keepcols
command together.
stilts tpipe ifmt=votable \ cmd='addcol IV_SUM "(IMAG+VMAG)"' \ cmd='addcol IV_DIFF "(IMAG-VMAG)"' \ cmd='delcols "IMAG VMAG"' \ omode=out ofmt=votable \ < tab1.vot \ > tab2.vot
in
nor out
parameters
have been specified, the input and output are actually byte
streams on standard input and standard output of the
tpipe
command in this case.
The processing steps first add a column representing the sum,
then add a column representing the difference, then delete the
original columns.
stilts tpipe cmd='addskycoords -inunit sex fk5 gal \ RA2000 DEC2000 GAL_LONG GAL_LAT' \ 6dfgs.fits 6dfgs+gal.fits
stilts -disk tpipe 2dfgrs_ngp.fits \ cmd='keepcols "SEQNUM AREA ECCENT"' \ cmd='sort -down AREA' \ cmd='head 20'
-disk
flag is supplied, which means that
temporary disk files rather than memory
will be used for caching table data.
stilts tpipe 2dfgrs_ngp.fits \ cmd='keepcols "SEQNUM AREA ECCENT"' \ cmd='sorthead -down 20 AREA'
sorthead
filter is
in most cases faster and cheaper on memory (only 20 rows ever have
to be stored in this case), so this is generally a better approach
than combining the sort
and head
filters.
stilts tpipe omode=meta cmd=@commands.lis http://archive.org/data/survey.vot.Z
stilts tpipe in=survey.fits cmd='select "skyDistance(hmsToRadians(RA),dmsToRadians(DEC), \ hmsToRadians(2,28,11),dmsToRadians(-6,49,45) \ < 5 * ARC_MINUTE"' \ omode=count
skyDistance
function is an expression which
calculates the distance between the position specified in a row
(as given by its RA and DEC columns) and a given point on the sky
(here, 02:28:11,-06:49:45).
Since skyDistance
's arguments and return value are in
radians, some conversions are required: the RA and DEC columns
are sexagesimal strings which are converted using the
hmsToRadians
and dmsToRadians
functions
respectively. Different versions of these functions (ones which take
numeric arguments) are used to convert the coordinates of the fixed
point to radians.
The result is compared to a multiple of the
ARC_MINUTE
constant, which is the size of an arcminute
in radians. Any rows of the input table for which this comparison
is true are included in the output.
An alternative function, skyDistanceDegrees
which works
in degrees, is also available.
The functions and constants used here are described in detail
in Section 8.5.1.
stilts tpipe ifmt=ascii survey.txt \ cmd='select "OBJTYPE == 3 && Z > 0.15"' \ cmd='keepcols "IMAG JMAG KMAG"' \ omode=stats
stilts -classpath lib/drivers/mysql-connector-java.jar \ -Djdbc.drivers=com.mysql.jdbc.Driver \ tpipe in=x.fits cmd="explodeall" omode=tosql \ protocol=mysql host=localhost database=ASTRO1 newtable=TABLEX \ user=mbt
jdbc.drivers
system property is set to the JDBC driver
class name. The output will be written as a new table named TABLEX
in the MySQL database named ASTRO1 on a MySQL server on the
local host. The password, if required, will be prompted for,
as would any of the other required parameters if they had not been
given on the command line.
Any existing table in ASTRO1 with the name TABLEX is overwritten.
The only processing done here is by the explodeall
command,
which takes any columns which have fixed-size array values and
replaces them in the output with multiple scalar columns.
java -classpath stilts.jar:lib/drivers/mysql-connector-java.jar -Djdbc.drivers=com.mysql.jdbc.Driver \ uk.ac.starlink.ttools.Stilts \ tpipe in=x.fits \ cmd=explodeall \ omode=out \ out="jdbc:mysql://localhost/ASTRO1?user=mbt#TABLEX"
stilts
script to do it. Note that you cannot use java's
-jar
flag in this case, because doing it like that
would not permit access to the additional classes that contain
the JDBC driver.
In the second place we use omode=out
rather than
omode=tosql
. For this we need to supply an out
value which encodes the information about the SQL connection and
table in a special URL-like format. As you can see, this is a bit
arcane, which is why the omode=tosql
mode can be a help.
stilts tpipe USNOB.FITS cmd='every 1000000' omode=stats
tskymatch2
: Crossmatches 2 tables on sky position
tskymatch2
performs a crossmatch of two tables based
on the proximity of sky positions.
You specify the columns or expressions giving
right ascension and declination in degrees for each input table,
and a maximum permissible separation in arcseconds, and the resulting
joined table is output.
If you omit expressions for the RA and Dec, an attempt is made to
identify the columns to use using column
Unified Content Descriptors
(UCDs)
or names. First columns bearing appropriate UCD1 or UCD1+ values
(POS_EQ_RA
, POS_EQ_RA_MAIN
,
pos.eq.ra
or pos.eq.ra;meta.main
and
their equivalents for declination) are sought. If these cannot be found,
columns named something like "RA" or "RA2000" are sought.
If either is found, the column units are consulted and radian->degree
conversions are performed if necessary (degrees are assumed if no
unit value is given). If nothing likely can be found, then the
command will fail with an error message.
This search logic is intended as a convenience only; it is somewhat
ad hoc and subject to change. To make sure that the correct
angle values are being used, specify the ra
and dec
position parameters explicitly.
tskymatch2
is simply a cut-down version, provided for
convenience, of the more general matching task
tmatch2
.
If you want more match options or otherwise more configurability,
you can probably find it by using tmatch2
.
The usage of tskymatch2
is
stilts <stilts-flags> tskymatch2 ifmt1=<in-format> ifmt2=<in-format> omode=<out-mode> <mode-args> out=<out-table> ofmt=<out-format> ra1=<expr/degs> dec1=<expr/degs> ra2=<expr/degs> dec2=<expr/degs> error=<value/arcsec> join=1and2|1or2|all1|all2|1not2|2not1|1xor2 find=best|all [in1=]<table1> [in2=]<table2>If you don't have the
stilts
script installed,
write "java -jar stilts.jar
" instead of
"stilts
" - see Section 3.
The available <stilts-flags>
are listed
in Section 2.1.
Parameter values are assigned on the command line as explained in Section 2.3. They are as follows:
dec1 = <expr/degs>
dec2 = <expr/degs>
error = <value/arcsec>
find = best|all
best
is selected,
then only the best match
between the two tables will be retained; in this case
the data from a row of either input table will appear in
at most one row of the output table.
If all
is selected, then all pairs of rows
from the two input tables which match the input criteria
will be represented in the output table.
[Default: best
]
ifmt1 = <in-format>
(auto)
(the default),
then an attempt will be
made to detect the format of the table automatically.
This cannot always be done correctly however, in which case
the program will exit with an error explaining which
formats were attempted.
[Default: (auto)
]
ifmt2 = <in-format>
(auto)
(the default),
then an attempt will be
made to detect the format of the table automatically.
This cannot always be done correctly however, in which case
the program will exit with an error explaining which
formats were attempted.
[Default: (auto)
]
in1 = <table1>
ifmt1
parameter.
in2 = <table2>
ifmt2
parameter.
join = 1and2|1or2|all1|all2|1not2|2not1|1xor2
1and2
: An output row for each row represented in both input tables1or2
: An output row for each row represented in either or both of the input tablesall1
: An output row for each matched or unmatched row in table 1all2
: An output row for each matched or unmatched row in table 21not2
: An output row only for rows which appear in the first table but are not matched in the second table2not1
: An output row only for rows which appear in the second table but are not matched in the first table1xor2
: An output row only for rows represented in one of the input tables but not the other one[Default: 1and2
]
ofmt = <out-format>
(auto)
"
(the default),
then the output filename will be
examined to try to guess what sort of file is required
usually by looking at the extension.
If it's not obvious from the filename what output format is
intended, an error will result.
This parameter must only be given if
omode
has its default value of "out
".
[Default: (auto)
]
omode = <out-mode> <mode-args>
out
, which means that
the result will be written as a new table to disk or elsewhere,
as determined by the out
and ofmt
parameters.
However, there are other possibilities, which correspond
to uses to which a table can be put other than outputting it,
such as displaying metadata, calculating statistics,
or populating a table in an SQL database.
For some values of this parameter, additional parameters
(<mode-args>
)
are required to determine the exact behaviour.
Possible values are
out
meta
stats
count
cgi
discard
topcat
plastic
tosql
help=omode
flag
or see Section 5.4 for more information.
[Default: out
]
out = <out-table>
This parameter must only be given if
omode
has its default value of "out
".
[Default: -
]
ra1 = <expr/degs>
ra2 = <expr/degs>
Here are some examples of using tmatch2
stilts tskymatch2 in1=obs_v.xml in2=obs_i.xml out=obs_iv.xml \ ra1=OBS_RA dec1=OBS_DEC ra2=OBS_RA dec2=OBS_DEC error=2
stilts tskymatch2 in1=obs_v.xml in2=obs_i.xml out=obs_iv.xml \ error=2
pos.eq.ra;meta.main
and
pos.eq.dec:meta.main
.
If no suitable UCDs are in place this invocation will fail with an error.
stilts tskymatch2 in1=virgo1.txt ifmt1=ascii in2=mgc.fits \ ra1='radiansToDegrees(raRad)' dec1='radiansToDegrees(deRad)' \ ra2=MGC_ALPHA_J2000 dec2=MGC_DELTA_J2000 \ error=10 join=2not1 omode=count
join=2not1
parameter causes the command to
identify all the objects in the first list
which do not have counterparts within 10 arcsec in the second list.
The number of such objects found is simply output to the terminal.
votcopy
: Transforms between VOTable encodings
The VOTable standard provides for three basic encodings
of the actual data within each table: TABLEDATA, BINARY and FITS.
TABLEDATA is a pure-XML encoding, which is relatively easy for humans
to read and write.
However, it is verbose and not very efficient for transmission
and processing,
for which reason the more compact BINARY format has been defined.
FITS format shares the advantages of BINARY, but is more likely to
be used where a VOTable is providing metadata 'decoration' for
an existing FITS table.
In addition, the BINARY and FITS encodings may carry their data
either inline
(as the base64-encoded text content of a STREAM
element)
or externally
(referenced by a STREAM
element's href
attribute).
These different formats have their different advantages and disadvantages. Since, to some extent, programmers are humans too, much existing VOTable software deals in TABLEDATA format even though it may not be the most efficient way to proceed. Conversely, you might wish to examine the contents of a BINARY-encoded table without use of any software more specialised than a text editor. So there are times when it is desirable to convert from one of these encodings to another.
votcopy
is a tool which translates between these
encodings while
making a minimum of other changes to the VOTable document.
The processing may result in some changes to lexical details
such as whitespace in start tags, but the element structure is not
modified. Unlike tpipe
it does not impose
STIL's model of what constitutes a table on the data between
reading it in and writing it out, so subtleties dependent on
the exact structure of the VOTable document will not be mangled.
The only important changes should be the contents of
DATA
elements in the document.
The usage of votcopy
is
stilts <stilts-flags> votcopy charset=<xml-encoding> cache=true|false href=true|false base=<location> [in=]<location> [out=]<location> [format=]TABLEDATA|BINARY|FITSIf you don't have the
stilts
script installed,
write "java -jar stilts.jar
" instead of
"stilts
" - see Section 3.
The available <stilts-flags>
are listed
in Section 2.1.
Parameter values are assigned on the command line as explained in Section 2.3. They are as follows:
base = <location>
href
flag is true.
Normally these are given names based on the name of the
output file.
But if this flag is given, the names will be based on the
<location>
string.
This flag is compulsory if
href
is true
and out=-
(output is to standard out),
since in this case there is no default base name to use.
cache = true|false
charset = <xml-encoding>
help=charset
for a full listing.
format = TABLEDATA|BINARY|FITS
null
is selected, then the tables will be
data-less (will contain no DATA element), leaving only
the document structure.
Data-less tables are legal VOTable elements.
[Default: tabledata
]
href = true|false
base
flag.
in = <location>
[Default: -
]
out = <location>
[Default: -
]
Normal use of votcopy
is pretty straightforward.
We give here a couple of examples of its input and output.
Here is an example VOTable document, cat.vot
:
<VOTABLE> <RESOURCE> <TABLE name="Authors"> <FIELD name="AuthorName" datatype="char" arraysize="*"/> <DATA> <TABLEDATA> <TR><TD>Charles Messier</TD></TR> <TR><TD>Mark Taylor</TD></TR> </TABLEDATA> </DATA> </TABLE> <RESOURCE> <COOSYS equinox="J2000.0" epoch="J2000.0" system="eq_FK4"/> <TABLE name="Messier Objects"> <FIELD name="Identifier" datatype="char" arraysize="10"/> <FIELD name="RA" datatype="double" units="degrees"/> <FIELD name="Dec" datatype="double" units="degrees"/> <DATA> <TABLEDATA> <TR> <TD>M51</TD> <TD>202.43</TD> <TD>47.22</TD> </TR> <TR> <TD>M97</TD> <TD>168.63</TD> <TD>55.03</TD> </TR> </TABLEDATA> </DATA> </TABLE> </RESOURCE> </RESOURCE> </VOTABLE>Note that it contains more structure than just a flat table: there are two
TABLE
elements,
the RESOURCE
element of the second one being nested
in the RESOURCE
of the first.
Processing this document using a generic table tool such as
tpipe
or tcopy
would lose this structure.
To convert the data encoding to BINARY format, we simply execute
stilts votcopy format=binary cat.votand the output is
<?xml version="1.0"?> <VOTABLE> <RESOURCE> <TABLE name="Authors"> <FIELD name="AuthorName" datatype="char" arraysize="*"/> <DATA> <BINARY> <STREAM encoding='base64'> AAAAD0NoYXJsZXMgTWVzc2llcgAAAAtNYXJrIFRheWxvcg== </STREAM> </BINARY> </DATA> </TABLE> <RESOURCE> <COOSYS equinox="J2000.0" epoch="J2000.0" system="eq_FK4"/> <TABLE name="Messier Objects"> <FIELD name="Identifier" datatype="char" arraysize="10"/> <FIELD name="RA" datatype="double" units="degrees"/> <FIELD name="Dec" datatype="double" units="degrees"/> <DATA> <BINARY> <STREAM encoding='base64'> TTUxAAAAAAAAAEBpTcKPXCj2QEecKPXCj1xNOTcAAAAAAAAAQGUUKPXCj1xAS4PX Cj1wpA== </STREAM> </BINARY> </DATA> </TABLE> </RESOURCE> </RESOURCE> </VOTABLE>Note that both tables in the document have been translated to BINARY format. The basic structure of the document is unchanged: the only differences are within the
DATA
elements. If we ran
stilts votcopy format=tabledataon either this output or the original input then the output would be identical (apart perhaps from whitespace) to the input table, since the data are originally in TABLEDATA format.
To generate a VOTable document with the data in external files,
the href
parameter is used. We will output in FITS format
this time. Executing:
stilts votcopy format=fits href=true cat.vot fcat.votwrites the following to the file
fcat.vot
:
... <DATA> <FITS> <STREAM href="fcat-1.fits"/> </FITS> </DATA> ... <DATA> <FITS> <STREAM href="fcat-2.fits"/> </FITS> </DATA> ...(the unchanged parts of the document have been skipped here for brevity). The actual data are written in two additional files in the same directory as the output file,
fcat-1.fits
and
fcat-2.fits
. These filenames are based on the
main output filename, but can be altered using the base
flag if required. Note this has also given you FITS binary table
versions of all the tables in the input VOTable document, which can be
operated on by normal FITS-aware software quite separately from the VOTable
if required.
votlint
: Validates VOTable documents
The VOTable standard, while not hugely complicated, has a number of subtleties and it's not difficult to produce VOTable documents which violate it in various ways. In fact it's probably true to say that most VOTable documents out there are not strictly legal. In some cases the errors are small and a parser is likely to process the document without noticing the trouble. In other cases, the errors are so serious that it's hard for any software to make sense of it. In many cases in between, different software will react in different ways, in the worst case appearing to parse a VOTable but in fact understanding the wrong data.
votlint
is a program which can check a VOTable document
and spot places where it does not conform to the VOTable standard,
or places which look like they may not mean what the author intended.
It is meant for use in two main scenarios:
Validating a VOTable document against the VOTable schema or DTD
of course goes a long way towards checking a VOTable document for errors
(though it's clear that many VOTable authors don't even go this far),
but it by no means does the whole job, simply because the schema/DTD
specification languages don't have the facilities
to understand the data structure
of a VOTable document. For instance the VOTable schema
will allow any plain text content in a TD
element, but whether
this makes sense in a VOTable depends on the datatype
attribute of the corresponding FIELD
element. There are many
other examples.
votlint
tackles this by parsing the VOTable document
in a way which understands its structure and assessing the content
as critically as it can. For any incorrect or questionable content
it finds, it will output a short message describing the problem
and giving its location in the document. What you do with this
information is then up to you.
Using votlint
is very straightforward.
The votable
argument
gives the location (filename or URL) of a VOTable document.
Otherwise, the document will be read from standard input.
Error and warning messages will be written on standard error.
Each message is prefixed with the location at which the error was
found (if possible the line and column are shown, though this is
dependent on your JVM's default XML parser).
The processing is SAX-based, so arbitrarily long tables can
be processed without heavy memory use.
votlint
can't guarantee to pick up every possible
error in a VOTable document, but it ought to pick up many of the
most serious errors that are commonly made in authoring VOTables.
The usage of votlint
is
stilts <stilts-flags> votlint validate=true|false version=1.0|1.1 out=<location> [votable=]<location>If you don't have the
stilts
script installed,
write "java -jar stilts.jar
" instead of
"stilts
" - see Section 3.
The available <stilts-flags>
are listed
in Section 2.1.
Parameter values are assigned on the command line as explained in Section 2.3. They are as follows:
out = <location>
[Default: -
]
validate = true|false
votlint
's own checks,
it is validated against an appropriate version of the VOTable
DTD which picks up such things as the presence of
unknown elements and attributes, elements in the wrong place,
and so on.
Sometimes however, particularly when XML namespaces are
involved, the validator can get confused and may produce
a lot of spurious errors. Setting this flag false prevents
this validation step so that only votlint
's
own checks are performed.
In this case many violations of the VOTable standard
concerning document structure will go unnoticed.
[Default: true
]
version = 1.0|1.1
votable = <location>
[Default: -
]
Votlint checks that the XML input is well-formed, and, unless the
valid=false
parameter is supplied, that it validates against the
1.0 or 1.1 (as appropriate) DTD. Although VOTable 1.1 is properly
defined against an XML Schema rather than a DTD, in conjunction with
the other checks done, the DTD validation turns out to be pretty comprehensive.
Some of the DTD validity checks are also done by
votlint
internally, so that some validity-type
errors may give rise to more than one warning.
In general, the program errs on the side of verbosity.
In addition to these checks, the following checks are carried out, and lead to ERROR reports if violations are found:
TD
contents incompatible
datatype
/arraysize
attributes declared
in FIELD
FIELD
PARAM
values incompatible with declared
datatype
/arraysize
arraysize
declarationsTD
elements with the wrong number of elementsPARAM
values with the wrong number of
elementsnrows
attribute on TABLE
element different
from the number of rows actually in the tableVOTABLE
version
attribute is unknownref
attributes without matching ID
elements
elsewhere in the documentID
attribute value on multiple elements.Additionally, the following conditions, which are not actually forbidden by the VOTable standard, will generate WARNING reports. Some of these may result from harmless constructions, but it is wise at least to take a look at the input which caused them:
TD
elements in row of TABLEDATA
tableTABLE
with no FIELD
elementsFIELD
or PARAM
elements with
datatype
of either
char
or unicodeChar
and undeclared arraysize
-
this is a common error which can result in
ignoring all but the first character in TD
elements from
a columnref
attributes which reference other elements by
ID
where the reference makes no, or questionable sense
(e.g. FIELDref
references FIELD
in a
different table)FIELD
s) with the
same name
attributesHere is a brief example of running votlint
against
a (very short) imperfect VOTable document. If the document looks like
this:
<VOTABLE version="1.1"> <RESOURCE> <TABLE nrows="2"> <FIELD name="Identifier" datatype="char"/> <FIELD name="RA" datatype="double"/> <FIELD name="Dec" datatype="double"/> <DESCRIPTION>A very small table</DESCRIPTION> <DATA> <TABLEDATA> <TR> <TD>Fomalhaut</TD> <TD>344.48</TD> <TD>-29.618</TD> <TD>HD 216956</TD> </TR> </TABLEDATA> </DATA> </TABLE> </RESOURCE> </VOTABLE>then the output of a
votlint
run looks like this:
INFO (l.4): No arraysize for character, FIELD implies single character ERROR (l.7): Element "TABLE" does not allow "DESCRIPTION" here. WARNING (l.11): Characters after first in char scalar ignored (missing arraysize?) WARNING (l.15): Wrong number of TDs in row (expecting 3 found 4) ERROR (l.18): Row count (1) not equal to nrows attribute (2)Note the warning at line 11 has resulted from the same error as the one at line 4 - because the
FIELD
element has no
arraysize
attribute, arraysize="1"
(single character) is assumed,
while the author almost certainly intended arraysize="*"
(unknown length string).
By examining these warnings you can see what needs to be done to fix this table up. Here is what it should look like:
<VOTABLE version="1.1"> <RESOURCE> <TABLE nrows="1"> <!-- change row count --> <DESCRIPTION>A very small table</DESCRIPTION> <!-- move DESCRIPTION --> <FIELD name="Identifier" datatype="char" arraysize="*"/> <!-- add arraysize --> <FIELD name="RA" datatype="double"/> <FIELD name="Dec" datatype="double"/> <DATA> <TABLEDATA> <TR> <TD>Fomalhaut</TD> <TD>344.48</TD> <TD>-29.618</TD> </TR> <!-- remove extra TD --> </TABLEDATA> </DATA> </TABLE> </RESOURCE> </VOTABLE>When fed this version,
votlint
gives no warnings.
This is STILTS, Starlink Tables Infrastructure Library Tool Set. It is a collection of non-graphical utilites for general purpose table and VOTable manipulation developed by Starlink.
The initial development of STILTS was done under the UK's Starlink project (1980-2005, R.I.P.), without which it would not have been written. From July 2005 until June 2006, it was supported by grant PP/D002486/1 from the UK's Particle Physics and Astronomy Research Council. Maintenance and development has been funded from July 2006 until December 2007 by the European VOTech project within the UK's AstroGrid, and directly from AstroGrid funding beyond that.
Apart from the excellent Java 2 Standard Edition itself, the following external libraries provide important parts of STILTS's functionality:
Many people have contributed ideas and advice to the development of STILTS and its related products. I can't list all of them here, but my thanks are especially due to the following:
Releases to date have been as follows:
stilts
",
invoked using the stilts
script or the
stilts.jar
jar file, and the various tasks are
named as subsequent arguments on the command line.
Command arguments are supplied after that.
The new invocation syntax is described in detail elsewhere in
this document. As well as invocation features such
as improved on-line help, optional prompting,
parameter defaulting, and more uniform access to common features,
this will make it more straightforward to wrap these tasks
for use in non-command-line environments, such as behind a
SOAP or CORBA interface, or in a CEA-like execution environment.
tmatch2
has been introduced.
This provides flexible and efficient crossmatching between
two input tables. Future releases will provide commands for
intra-table and multi-table matching.
tcat
has been introduced, which
allows two tables to be glued together top-to-bottom.
This is currently working but very rudimentary - improvements
will be forthcoming in future releases.
calc
has been introduced,
which performs one-line expression evaluations from the
command line.
tpipe
and other commands have been introduced:
addskycoords
: calculates new
celestial coordinate pair from existing ones
(FK4, FK5, ecliptic, galactic, supergalactic)replacecol
: replaces column data,
using existing metadatabadval
: replaces given 'magic'
value with nullreplaceval
: replaces given 'magic'
value with any specified valuetablename
: edits table nameexplodecols
and explodecols
commands
replace explode
The new stream
parameter of tpipe
now
allows you to write filter commands in an external file, to
facilitate more manageable command lines.
Wildarding for column specification is now allowed for some filter commands.
tcube
Command
stats
filter provides the same information as
the old stats
output mode, but allows much more
flexible use of the results. It can also calculate many new
quantities, including quantiles, skew and kurtosis.meta
filter provides the same information as
the old meta
output mode, but allows much more
flexible use of the results.assert
filter provides in-pipeline logical
assertions.uniq
filter collapses multiple adjacent identical
or similar rows.sorthead
filter provides a (usually) more
efficient method of doing what you could previously do
by combining sort
and head
filters.colmeta
filter adds/modifies metadata for selected
columns.check
filter checks table in stream - for debugging
purposes only.Additionally usage of the sort
filter has been changed
so that it can now do everything that sortexpr
used to
be able to do; sortexpr
is now withdrawn.
plastic
mode broadcasts the table to
one or all registered PLASTIC listeners.cgi
mode writes the table to standard output in a
form suitable for output from a CGI script.discard
mode throws away the table.topcat
mode now attempts to use PLASTIC
(amongst other methods) to contact TOPCAT.stats
and meta
modes are mildly
deprecated in favour of the corresponding new filters
(see above).csv-noheader
format variant output handler
added.roundDecimal
and formatDecimal
functions introduced for more control over visual appearance
of numeric values.mark.workaround
system property which can
optionally work around a bug in some input streams
("Resetting to invalid mark" errors).ucd
and
utype
attributes of TABLE element in
votlint
.istream=true
is now less likely to cause a
"Can't re-read stream" error.colfits
format, new startable.storage
policy "sideways
") have been introduced.
These can provide considerable efficiency improvements for
certain tasks when working with very large (and especially wide)
tables.
multicone
- Makes multiple cone search queries to the same serviceregquery
- Queries the VO registrytranspose
filter added.setparam
and clearparams
filters.addskycoords
.tcat
command has been replaced by more capable
tcat
and
tcatn
commands.
Between them these provide concatenation of an unlimited number of
homogeneous or heterogeneous input tables.
Additional columns may be added to indicate which of the input tables
given output rows originated from.
in
in tcat
,
cmd
and friends)
may now be specified in the form "@filename".
This indicates that the value for the parameter is to be obtained
by reading it from the named file.
This is useful if a very long value is required for the parameter
in question. The script
parameter of
tpipe
has therefore been
withdrawn, since it did just the same thing.
ivo:
- or myspace:
-type URLs.
toHex
and fromHex
numeric conversion
functions have been added
(Section 8.5.11).commands
etc.-J
flag to stilts
script
for passing flags directly to Java.out
parameter to votlint
.-ifndim
and -ifshape
flags to
explodeall filter.exact
match mode in
tmatch2
now copes with array-valued columns.force
parameter to multicone
task
as a workaround for some broken services.stats
filter.votlint
about
utype
attribute on RESOURCE elements.tjoin
introduced.votlint
and votcopy
.-bench
flag to stilts
command.text
and ascii
output formats (now one-pass not two-pass).duptag
parameters to tmatch2
task for customised renaming of columns with duplicated names.sinh
, cosh
, tanh
and inverses).funcs
task, a browser for
expression language function documentation.-checkversion
to list of stilts
flags.table
parameter to
calc
command
(for access to table parameters).param$
notation (Section 8.2).ucd$
notation
in expressions (Section 8.1)
and as column identifiers (Section 5.2).TNULL
n header
cards - write them as numeric not string values.gcj
).formatDecimalLocal
functions in
class Formats.fluxToLuminosity
and luminosityToFlux
functions in class
Fluxes.transpose
filter.sqlcone
task introduced, along with some classes in package
uk.ac.starlink.ttools.cone
designed for library use by
AstroGrid DSA code.
tmatch2
params
parameter now has minoccurs=0
, since that can be true
for exact matches.-stdout
and -stderr
flags to
stilts
command.app-description.xml
file.sqlcone
command.fluxToLuminosity
function.tmatch2
command are new tasks:
tskymatch2
:
stripped down version of tmatch2
for ease of use when
matching with sky coordinates.tmatch1
:
internal matcher, finds groups of objects within a table.tmatchn
:
finds group or multiple-pair matches between multiple (>2)
tables.Two tasks have been renamed for improved clarity and consistency:
multicone
is now named
coneskymatch
sqlcone
is now named
sqlskymatch
There has also been some enhancement and rationalisation of
parameters for all table join tools (tmatch*
as well as
tjoin
,
coneskymatch
and sqlskymatch
):
fixcols
and
suffix*
parameters to control renaming of duplicated
columns in output tables (note this replaces the old
duptag*
parameters in tmatch2
).progress
parameter
which allows you to configure whether progress is reported to the
console.copycols
parameter of coneskymatch
and sqlskymatch
now defaults to "*
"
(include all columns from input table in the output).Section 6 of the manual has been somewhat rearranged and improved.
votlint
handling of TABLEDATA-type
multi-dimensional char
/unicodeChar
arrays.
These are now split up into strings by counting characters rather
than using whitespace delimiters.
I think it's doing the right thing now.sqlclient
,
which is a general JDBC-based SQL command-line client.sqlupdate
,
which allows updates to existing rows in SQL tables.tosql
output mode:
write
(can be create
, dropcreate
or append
)newtable
renamed
dbtable
database
renamed
db
for consistency with other commandsscorecol
added to
tmatch2
,
coneskymatch
and
sqlskymatch
commands,
which controls adding a new column to match output tables
containing a goodness-of-match value.parallel
added to
coneskymatch
task
which allows multiple cone searches to be carried out in
parallel.erract
added to
coneskymatch
which controls response to
isolated failures in individual cone search queries.-debug
flag).help='*'
which prints help for
all parameters of a task at once.+verbose
flag
for reducing verbosity level.datatype
attribute.progress
parameter to tmatchn
.emptyok
parameter to
coneskymatch
.calc
task.stats
filter cardinality value
calculation.fixcols
.stats
filter
usage message.server
is
provided which allows STILTS commands to be executed via HTTP.
One purpose of this is to facilitate server-side use of the
plotting commands co-located with data to generate on-the-fly
graphical summaries of server-held datasets.
minReal
and maxReal
functions
(max/min ignoring blank values) in class
Arithmetic.
tcat
's
loccol
parameter.