Next Previous Up Contents
Next: Specifying a single column
Up: Table Pipelines
Previous: Table Pipelines

5.1 Processing Filters

This section lists the filter commands which can be used for table pipeline processing, in conjunction with cmd- or script-type parameters.

You can string as many of these together as you like. On the command line, you can repeat the cmd (or icmd1, or ocmd...) parameter multiple times, or use one cmd parameter and separate different filter specifiers with semicolons (";"). The effect is the same.

It's important to note that each command in the sequence of processing steps acts on the table at that point in the sequence. Thus

   stilts tpipe cmd='delcols 1; delcols 1; delcols 1'
has the same effect as
   stilts tpipe cmd='delcols "1 2 3"'
since in the first case the columns are shifted left after each one is deleted, so the table seen by each step has one fewer column than the one before. Note also the use of quotes in the latter of the examples above, which is necessary so that the <colid-list> of the delcols command is interpreted as one argument not three separate words.

The syntax of some of these arguments is described elsewhere in this document:

addcol [-after <col-id> | -before <col-id>]
       [-units <units>] [-ucd <ucd>] [-desc <description>]
       <col-name> <expr>
Add a new column called <col-name> defined by the algebraic expression <expr>. By default the new column appears after the last column of the table, but you can position it either before or after a specified column using the -before or -after flags respectively. The -units, -ucd and -desc flags can be used to define metadata values for the new column.
addskycoords [-epoch <expr>] [-inunit deg|rad|sex] [-outunit deg|rad|sex]
             <insys> <outsys> <col-id1> <col-id2> <col-name1> <col-name2>
Add new columns to the table representing position on the sky. The values are determined by converting a sky position whose coordinates are contained in existing columns. The <col-id> arguments give identifiers for the two input coordinate columns in the coordinate system named by <insys>, and the <col-name> arguments name the two new columns, which will be in the coordinate system named by <outsys>. The <insys> and <outsys> coordinate system specifiers are one of

The -inunit and -outunit flags may be used to indicate the units of the existing coordinates and the units for the new coordinates respectively; use one of degrees, radians or sexagesimal (may be abbreviated), otherwise degrees will be assumed. For sexagesimal, the two corresponding columns must be string-valued in forms like hh:mm:ss.s and dd:mm:ss.s respectively.

For certain conversions, the value specified by the -epoch flag is of significance. Where significant its value defaults to 2000.0.

badval <bad-val> <colid-list>
For each column specified in <colid-list> any occurrence of the value <bad-val> is replaced by a blank entry.
cache
Stores in memory or on disk a temporary copy of the table at this point in the pipeline. This can provide improvements in efficiency if there is an expensive step upstream and a step which requires more than one read of the data downstream. If you see an error like "Can't re-read data from stream" then adding this step near the start of the filters might help.
delcols <colid-list>
Delete the specified columns. The same column may harmlessly be specified more than once.
every <step>
Include only every <step>'th row in the result, starting with the first row.
explodeall
Replaces any column which is an N-element array with N scalar columns. Only columns with fixed array sizes are affected.
explodecols <colid-list>
Takes a list of specified columns which represent N-element arrays and replaces each one with N scalar columns. Each of the columns specified by <colid-list> must have a fixed-length array type, though not all the arrays need to have the same number of elements.
head <nrows>
Include only the first <nrows> rows of the table.
keepcols <colid-list>
Select the columns from the input table which will be included in the output table. The output table will include only those columns listed in <colid-list>, in that order. The same column may be listed more than once, in which case it will appear in the output table more than once.
progress
Monitors progress by displaying the number of rows processed so far on the terminal (standard error). This number is updated every second or thereabouts; if all the processing is done in under a second you may not see any output. If the total number of rows in the table is known, an ASCII-art progress bar is updated, otherwise just the number of rows seen so far is written.
random
Ensures that steps downstream see the table as random access. Only useful for debugging.
replacecol [-name <name>] [-units <units>] [-ucd <ucd>] [-desc <descrip>]
           <col-id> <expr>
Replaces the content of a column with the value of an algebraic expression. The old values are discarded in favour of the result of evaluating <expr>. You can specify the metadata for the new column using the -name, -units, -ucd and -desc flags; for any of these items which you do not specify, they will take the values from the column being replaced. You can reference the replaced column in the expression, so for example "replacecol pixsize pixsize*2" just multiplies the values in column pixsize by 2.
replaceval <old-val> <new-val> <colid-list>
For each column specified in <colid-list> any instance of <old-val> is replaced by <new-val>. The value string 'null' can be used for either <old-value> or <new-value> to indicate a blank value.
select <expr>
Include in the output table only rows for which the expression <expr> evaluates to true. <expr> must be an expression which evaluates to a boolean value (true/false).
sequential
Ensures that steps downstream see the table as sequential access. Only useful for debugging.
sort [-down] [-nullsfirst] <colid-list>
Sorts the table according to the columns named in <colid-list>. One or more columns may be specified; sorting is done on the values in the first-specified field., but if they are equal the tie is resolved by looking at the second-specified field, and so on. If the -down flag is used, the sort order is descending rather than ascending. Blank entries are usually considered to come at the end of the collation sequence, but if the -nullsfirst flag is given then they are considered to come at the start instead.
sortexpr [-down] [-nullsfirst] <expr>
Sorts the table according to the value of an algebraic expression. <expr> must evaluate to a type that it makes sense to sort, for instance numeric. If the -down flag is used, the sort order is descending rather than ascending. Blank entries are usually considered to come at the end of the collation sequence, but if the -nullsfirst flag is given then they are considered to come at the start instead.
tablename <name>
Sets the table's name attribute to the given string.
tail <nrows>
Include only the last <nrows> rows of the table.


Next Previous Up Contents
Next: Specifying a single column
Up: Table Pipelines
Previous: Table Pipelines

STILTS - Starlink Tables Infrastructure Library Tool Set
Starlink User Note 256
STILTS web page: http://www.starlink.ac.uk/stilts/
Author email: m.b.taylor@bristol.ac.uk