This section lists the filter commands which can be used for
table pipeline processing, in conjunction with cmd
-
or script
-type parameters.
You can string as many of these together as you like.
On the command line, you can repeat the cmd
(or icmd1
, or ocmd
...) parameter
multiple times, or use one cmd
parameter and
separate different filter specifiers with semicolons (";
").
The effect is the same.
It's important to note that each command in the sequence of processing steps acts on the table at that point in the sequence. Thus
stilts tpipe cmd='delcols 1; delcols 1; delcols 1'has the same effect as
stilts tpipe cmd='delcols "1 2 3"'since in the first case the columns are shifted left after each one is deleted, so the table seen by each step has one fewer column than the one before. Note also the use of quotes in the latter of the examples above, which is necessary so that the
<colid-list>
of the delcols
command is interpreted as one argument not
three separate words.
The syntax of some of these arguments is described elsewhere in this document:
<col-id>
: see Section 5.2
<colid-list>
: see Section 5.3
<expr>
: see Section 7
addcol [-after <col-id> | -before <col-id>] [-units <units>] [-ucd <ucd>] [-desc <description>] <col-name> <expr>
<col-name>
defined
by the algebraic expression <expr>
.
By default the new column appears after the last column
of the table, but you can position it either before or
after a specified column using the -before
or -after
flags respectively.
The -units
, -ucd
and
-desc
flags can be used to define
metadata values for the new column.
addskycoords [-epoch <expr>] [-inunit deg|rad|sex] [-outunit deg|rad|sex] <insys> <outsys> <col-id1> <col-id2> <col-name1> <col-name2>
<col-id>
arguments give identifiers for
the two input coordinate columns
in the coordinate system named by
<insys>
, and
the <col-name>
arguments name
the two new columns,
which will be in the coordinate system named by
<outsys>
.
The <insys>
and <outsys>
coordinate system specifiers are one of
fk5
: FK5 J2000.0 (Right Ascension, Declination)fk4
: FK4 B1950.0 (Right Ascension, Declination)galactic
: IAU 1958 Galactic (Longitude, Latitude)supergalactic
: de Vaucouleurs Supergalactic (Longitude, Latitude)ecliptic
: Ecliptic (Longitude, Latitude)The -inunit
and -outunit
flags
may be used to indicate the units of the existing coordinates
and the units for the new coordinates respectively;
use one of
degrees
, radians
or
sexagesimal
(may be abbreviated),
otherwise degrees will be assumed.
For sexagesimal, the two corresponding columns must be
string-valued in forms like hh:mm:ss.s and dd:mm:ss.s
respectively.
For certain conversions, the value specified by the
-epoch
flag is of significance.
Where significant its value defaults to 2000.0.
assert <expr>
<expr>
does not
evaluate true for any row of the table, execution terminates
with an error.
As long as no error occurs, the output table is identical
to the input one.
The exception generated by an assertion violation is of class
uk.ac.starlink.ttools.filter.AssertException
although that is not usually obvious if you are running from
the shell in the usual way.
badval <bad-val> <colid-list>
<colid-list>
any occurrence of the value <bad-val>
is replaced by a blank entry.
cache
check
colmeta [-name <name>] [-units <units>] [-ucd <ucd>] [-desc <descrip>] <colid-list>
<colid-list>
can be set by using some or all of the listed flags.
Typically, <colid-list>
will simply be
the name of a single column.
delcols <colid-list>
every <step>
<step>
'th row in the
result, starting with the first row.
explodeall
explodecols <colid-list>
<colid-list>
must have a fixed-length array type,
though not all the arrays need to have the same number
of elements.
head <nrows>
<nrows>
rows of
the table.
keepcols <colid-list>
<colid-list>
, in that order.
The same column may be listed more than once,
in which case it will appear in the output table more than once.
meta [<item> ...]
By default the output table contains columns for the
items Index, Name, Class, Shape, Units, Description and UCD,
as well as any table-specific column metadata items that
the table contains.
The output may be customised however by supplying one or more
<item>
headings. These may be selected
from the list Index, Name, Class, Shape, Units, Description, UCD and UCD_desc,
as well as any table-specific metadata. It is not an error
to specify an item for which no metadata exists in any of
the columns.
Any table parameters of the input table are propagated to the output one.
progress
random
replacecol [-name <name>] [-units <units>] [-ucd <ucd>] [-desc <descrip>] <col-id> <expr>
<expr>
.
You can specify the metadata for the new column using the
-name
, -units
, -ucd
and -desc
flags; for any of these items which you
do not specify, they will take the values from the column
being replaced.
You can reference the replaced column in the expression,
so for example
"replacecol pixsize pixsize*2
" just multiplies
the values in column pixsize
by 2.
replaceval <old-val> <new-val> <colid-list>
<colid-list>
any instance of <old-val>
is replaced by
<new-val>
.
The value string 'null
' can be used for either
<old-value>
or <new-value>
to indicate a blank value.
select <expr>
<expr>
evaluates to true.
<expr>
must be an expression which
evaluates to a boolean value (true/false).
sequential
sort [-down] [-nullsfirst] <key-list>
<key-list>
; sorting is done on the
first expression first, but if that results in a tie then
the second one is used, and so on.
Each expression must evaluate to a type that
it makes sense to sort, for instance numeric.
If the -down
flag is used, the sort order is
descending rather than ascending.
Blank entries are usually considered to come at the end
of the collation sequence, but if the -nullsfirst
flag is given then they are considered to come at the start
instead.
sorthead [-tail] [-down] [-nullsfirst] <nrows> <key-list>
<nrows>
rows at the head
of the resulting sorted table.
The sort key expressions appear,
as separate (space-separated) words,
in <key-list>
; sorting is done on the
first expression first, but if that results in a tie then
the second one is used, and so on.
If the -tail
flag is used, then the
last <nrows>
rows rather than the first
ones are retained.
If the -down
flag is used the sort order is
descending rather than ascending.
Blank entries are usually considered to come at the end
of the collation sequence, but if the -nullsfirst
flag is given then they are considered to come at the start
instead.
Each expression must evaluate to a type that
it makes sense to sort, for instance numeric.
This filter is functionally equivalent to using
sort
followed by head
,
but it can be done in one pass and is usually cheaper
on memory and faster, as long as <nrows>
is significantly lower than the size of the table.
stats [<item> ...]
By default the output table contains columns for the
items Name, Mean, StDev, Minimum, Maximum and NGood.
The output may be customised however by supplying one or more
<item>
headings. These may be selected
from the list NGood, NBad, Mean, StDev, Variance, Skew, Kurtosis, Minimum, Maximum, Sum, MinPos, MaxPos, Cardinality, Median, Quartile1, Quartile2 and Quartile3,
or have the form "Q.nn" to represent the quantile
corresponding to the proportion 0.nn; for instance
Q.5 is an alias for Median, and Q.25 for Quartile1.
Any parameters of the input table are propagated to the output one.
Note that quantile calculations (including median and quartiles) can be expensive on memory. If you want to calculate quantiles for large tables, it may be wise to reduce the number of columns to only those you need the quantiles for earlier in the pipeline. No interpolation is performed when calculating quantiles.
tablename <name>
tail <nrows>
<nrows>
rows
of the table.
uniq [-count] [<colid-list>]
<colid-list>
parameter is given
then only the values in the specified columns must be equal
in order for the row to be removed.
If the -count
flag is given, then an additional
column with the name DupCount will be
prepended to the table giving a count of the number of duplicated
input rows represented by each output row. A unique row
has a DupCount value of 1.