Next Previous Up Contents
Up: Release Notes
Previous: Acknowledgements


C.2 Version History

Releases to date have been as follows:

Version 0.1b (29 April 2005)
First public release
Version 0.2b (30 June 2005)
Version 1.0b (30 September 2005)
This is the first non-experimental release of STILTS, and it incorporates major changes and backward incompatibilities since version 0.2b.

Parameter system
The parameter system has undergone a complete rewrite; there is now only a single command "stilts", invoked using the stilts script or the stilts.jar jar file, and the various tasks are named as subsequent arguments on the command line. Command arguments are supplied after that. The new invocation syntax is described in detail elsewhere in this document. As well as invocation features such as improved on-line help, optional prompting, parameter defaulting, and more uniform access to common features, this will make it more straightforward to wrap these tasks for use in non-command-line environments, such as behind a SOAP or CORBA interface, or in a CEA-like execution environment.
Crossmatching
A new command tmatch2 has been introduced. This provides flexible and efficient crossmatching between two input tables. Future releases will provide commands for intra-table and multi-table matching.
Concatentation
A new command tcat has been introduced, which allows two tables to be glued together top-to-bottom. This is currently working but very rudimentary - improvements will be forthcoming in future releases.
Calculator
A new utility command calc has been introduced, which performs one-line expression evaluations from the command line.
Pipeline filters
The following new filter commands for use in tpipe and other commands have been introduced:
  • addskycoords: calculates new celestial coordinate pair from existing ones (FK4, FK5, ecliptic, galactic, supergalactic)
  • replacecol: replaces column data, using existing metadata
  • badval: replaces given 'magic' value with null
  • replaceval: replaces given 'magic' value with any specified value
  • tablename: edits table name
  • explodecols and explodecols commands replace explode

The new stream parameter of tpipe now allows you to write filter commands in an external file, to facilitate more manageable command lines.

Wildarding for column specification is now allowed for some filter commands.

Algebraic functions
  • New functions for converting time values between different coordinate systems (Modified Julian Date, ISO-8601, Julian Epoch and Besselian Epoch).
  • New RANDOM special function.
Documentation
SUN/256 has undergone many changes. Much of the tool documentation is now automatically generated from the code itself, which goes a long way to ensuring that the documentation is correct with respect to the current state of the code.

Version 1.0-1b (7 October 2005)
Fixed jar file manifest bug which prevented working on Java 1.5
Version 1.1 (10 May 2006)
A number of new features and capabilities have been introduced:
tcube Command
The new tcube command calculates N-dimensional histograms (density maps) from N columns of an input table and writes the result to a FITS file.
Processing Filters
The following new filters have been added:
  • stats filter provides the same information as the old stats output mode, but allows much more flexible use of the results. It can also calculate many new quantities, including quantiles, skew and kurtosis.
  • meta filter provides the same information as the old meta output mode, but allows much more flexible use of the results.
  • assert filter provides in-pipeline logical assertions.
  • uniq filter collapses multiple adjacent identical or similar rows.
  • sorthead filter provides a (usually) more efficient method of doing what you could previously do by combining sort and head filters.
  • colmeta filter adds/modifies metadata for selected columns.
  • check filter checks table in stream - for debugging purposes only.

Additionally usage of the sort filter has been changed so that it can now do everything that sortexpr used to be able to do; sortexpr is now withdrawn.

Output Modes
The following new output modes have been introduced:
  • plastic mode broadcasts the table to one or all registered PLASTIC listeners.
  • cgi mode writes the table to standard output in a form suitable for output from a CGI script.
  • discard mode throws away the table.
and usage of the following has been modified:
  • topcat mode now attempts to use PLASTIC (amongst other methods) to contact TOPCAT.
  • stats and meta modes are mildly deprecated in favour of the corresponding new filters (see above).
Other new features
  • New IPAC table format input handler added.
  • New csv-noheader format variant output handler added.
  • roundDecimal and formatDecimal functions introduced for more control over visual appearance of numeric values.
  • Experimental facilities for automatically generating a CEA application description file.
Bug fixes and minor improvements
  • Now copes with 'K'-format FITS binary table columns (64-bit integers).
  • Improved, though still imperfect, retention of table-wide metadata in VOTables.
  • Distinctions between null and false values in boolean columns are handled more carefully for FITS and VOTable files.
  • Efficiency improvement when writing FITS-plus format (now only requires a maximum of two passes rather than three of the input rows).
  • Added the mark.workaround system property which can optionally work around a bug in some input streams ("Resetting to invalid mark" errors).
  • Fixed a bug in Cartesian matching which failed to match if the required error in any dimension was zero.
  • Fixed erroneous reports about unknown ucd and utype attributes of TABLE element in votlint.
  • When joining tables, column name comparison to determine whether deduplication is required is now case-insensitive.
  • Error message improved when no automatic format detection is attempted for streamed tables.
  • Setting istream=true is now less likely to cause a "Can't re-read stream" error.
Version 1.2 (7 July 2006)
Column-oriented Storage
New features for permitting column-oriented storage (colfits format, new startable.storage policy "sideways") have been introduced. These can provide considerable efficiency improvements for certain tasks when working with very large (and especially wide) tables.
New VO commands
Added two new commands for querying Virtual Observatory services:
  • multicone - Makes multiple cone search queries to the same service
  • regquery - Queries the VO registry
These tasks are experimental and may be modified or renamed in future releases.
Other items
  • transpose filter added.
  • Added flux conversion functions (Jansky<->magnitude).
  • ISO-8601 strings now permit times of 24:00:00 as they should.
Version 1.2-1 (3 August 2006)
Version 1.3 (5 October 2006)
Table Concatenation
The old tcat command has been replaced by more capable tcat and tcatn commands. Between them these provide concatenation of an unlimited number of homogeneous or heterogeneous input tables. Additional columns may be added to indicate which of the input tables given output rows originated from.
Parameter value indirection
Certain parameters (in in tcat, cmd and friends) may now be specified in the form "@filename". This indicates that the value for the parameter is to be obtained by reading it from the named file. This is useful if a very long value is required for the parameter in question. The script parameter of tpipe has therefore been withdrawn, since it did just the same thing.
MySpace access
Direct access to the MySpace virtual file system is now provided by use of ivo:- or myspace:-type URLs.
Conversion functions
  • Time conversion functions between MJD and Decimal Year have been added (Section 9.5.1).
  • toHex and fromHex numeric conversion functions have been added (Section 9.5.3).
Documentation improvements
  • The HTML version of SUN/256 now uses CSS to provide better highlighting of commands etc.
  • The Output Modes and Processing Filter sections are now split into subsections to make the table of contents clearer.
  • The Command Reference section now has only one level of subsection listed in the table of contents to make it clearer.
Other new features and improvements
  • Added -J flag to stilts script for passing flags directly to Java.
  • Added new out parameter to votlint.
  • Added -ifndim and -ifshape flags to explodeall filter.
  • The exact match mode in tmatch2 now copes with array-valued columns.
  • Added force parameter to multicone task as a workaround for some broken services.
  • Added Sample (as opposed to Population) Standard Deviation/Variance calculation options to the stats filter.
  • Improved CEA description file output - now contains details of all tasks rather than just a few, as well as various improvements in documentation etc.
Bug fixes
  • Fixed erroneous complaints from votlint about utype attribute on RESOURCE elements.
  • Fixed a couple of minor crossmatching bugs (which wouldn't have affected results).
Version 1.3-1 (Starlink Hokulei release)
Version 1.3-2 (6 July 2007)
Version 1.3-3 (4 Sep 2007)
Version 1.3-4 (10 Sep 2007)
Version 1.3-5 (30 Oct 2007)
Version 1.4 (6 December 2007)
Table joins
This version provides more cross matching functionality. Added to the existing tmatch2 command are new tasks:
  • tskymatch2: stripped down version of tmatch2 for ease of use when matching with sky coordinates.
  • tmatch1: internal matcher, finds groups of objects within a table.
  • tmatchn: finds group or multiple-pair matches between multiple (>2) tables.

Two tasks have been renamed for improved clarity and consistency:

There has also been some enhancement and rationalisation of parameters for all table join tools (tmatch* as well as tjoin, coneskymatch and sqlskymatch):

  • All table join commands now use similar fixcols and suffix* parameters to control renaming of duplicated columns in output tables (note this replaces the old duptag* parameters in tmatch2).
  • Crossmatching tasks have a new progress parameter which allows you to configure whether progress is reported to the console.
  • The copycols parameter of coneskymatch and sqlskymatch now defaults to "*" (include all columns from input table in the output).

Section 7 of the manual has been somewhat rearranged and improved.

Other enhancements
  • FITS reader now imports table HDU header cards as table parameters.
  • CeaWriter can now output CEA service definition XML config file as well as app-description file (experimental - may be withdrawn).
Bug fixes
  • Embedded spaces in output ASCII format table column names are now substituted with underscores.
  • Fix a bug which caused an infinite number of dots to be printed when attempting a crossmatch with an empty input table.
  • Corrected votlint handling of TABLEDATA-type multi-dimensional char/unicodeChar arrays. These are now split up into strings by counting characters rather than using whitespace delimiters. I think it's doing the right thing now.
Version 1.4-1 (28 January 2008)
New RDBMS-related features
  • New command sqlclient, which is a general JDBC-based SQL command-line client.
  • New command sqlupdate, which allows updates to existing rows in SQL tables.
  • Some changes to tosql output mode:
    • choice of options for how to write to the database output table, controlled by new associated parameter write (can be create, dropcreate or append)
    • associated parameter newtable renamed dbtable
    • associated parameter database renamed db for consistency with other commands
Local and service-based matching command enhancements
  • New parameter scorecol added to tmatch2, coneskymatch and sqlskymatch commands, which controls adding a new column to match output tables containing a goodness-of-match value.
  • New parameter parallel added to coneskymatch task which allows multiple cone searches to be carried out in parallel.
  • New parameter erract added to coneskymatch which controls response to isolated failures in individual cone search queries.
General improvements
  • Improved error reporting (reasons for errors are now reported even without the -debug flag).
  • Add new help option help='*' which prints help for all parameters of a task at once.
  • Added (mostly undocumented) +verbose flag for reducing verbosity level.
  • Minor improvements to CEA app-description.
  • Downgraded from WARNING to INFO log messages about the (extremely common) VOTable syntax error of omitting a FIELD/PARAM element's datatype attribute.
Version 1.4-2 (26 March 2008)
Minor enhancements:
  • Add progress parameter to tmatchn.
  • Add emptyok parameter to coneskymatch.
Bugfixes:
  • Fixed pair matching performance bug (slower if tables were not given in the right order) introduced at v1.4.
  • Fixed null handling error in calc task.
  • Fixed error in stats filter cardinality value calculation.
  • Fixed minor bugs in suffix addition for matching commands fixcols.
  • Removed unformatted XML output in stats filter usage message.
  • Try to avoid exponential format in cone search URLs (some endpoints seem to require fixed point format).
  • Minor CEA fixes.
Version 2.0b (23 October 2008)
This version contains two new major items, plotting and server mode. Both work, but are missing desirable features and have not had extensive testing in the field, so should be considered experimental at this stage.
Plotting
Two table plotting commands are now provided: See also the new Plotting section in the manual.
Server/Servlet Mode
A new command server is provided which allows STILTS commands to be executed via HTTP. One purpose of this is to facilitate server-side use of the plotting commands co-located with data to generate on-the-fly graphical summaries of server-held datasets.
Smaller enhancements and bugfixes
  • Efficiency improvements (~25%? in both CPU time and memory usage) for HEALPix-based sky crossmatching (thanks to Nikolay Kouropatkine at Fermilab for a new version of the PixTools library).
  • New class Arrays added to algebraic functions.
  • New Appendix Commands by Category added to manual.
  • Add minReal and maxReal functions (max/min ignoring blank values) in class Arithmetic.
  • Sexagesimal field identification for ASCII input files is less stringent (now permits minutes or seconds equal to 60).
  • Minor CEA fixes.
  • HEALPix bug fix (PixTools bug fix update).
  • Fix bug in use of tcat's loccol parameter.
Version 2.0-1 (23 December 2008)
Version 2.0-2 (9 January 2009)
Version 2.0-3 (27 March 2009)
Version 2.0-4 (17 July 2009)
Version 2.0-5 (2 Oct 2009)
Version 2.1 (6 November 2009)
Note that the STIL API used by this release has changed in some backwardly incompatible ways, and may change further. If you're using STILTS as a library rather than an application you might want to wait for a later release when the API has settled down.
Version 2.1-1 (21 December 2009)
Version 2.1-2 (24 March 2010)
Version 2.2 (6 August 2010)
New capabilities for multi-table I/O have been introduced: There are some additional enhancements: And a number of bug fixes:

Finally, from this release STILTS requires version 1.5 (a.k.a. 5.0) of the Java J2SE Runtime Environment; it will no longer run on version 1.4, which is now very old. I don't expect this to cause compatibility issues for anyone, but I'm interested to hear if that's not the case.

Version 2.2-1 (23 December 2010)
Version 2.3 (9 May 2011)
TAP
The new commands tapquery and tapresume have been introduced. These provide support for the Table Access Protocol (TAP), and allow freeform queries in an SQL-like language to be made to remote databases.
Minor enhancements
  • Random Groups HDUs are now tolerated, though not interpreted, within FITS files.
  • Added soapout parameter to regquery command.
  • Added count, variance and stdev functions to Arrays.
  • Upgrade to JSAMP v1.2.
  • Improve text rendering in funcs window display.
  • Attempt case-sensitive matching before case-insensitive for column names.
  • Fix replaceval filter to work with Infinities.
Bug fixes and workarounds
  • JDBC table input handler now effectively downcasts BigInteger/BigDecimal types to Long/Double. The PostgreSQL JDBC driver seems to use the Big* types routinely for numeric values (which I don't think it used to do).
  • Add workaround for J2SE bug #4795134, which could cause errors when reading compressed FITS files.
  • Fix FITS character handling bug which could cause corrupted FITS files on output in presence of non-ASCII characters.
  • Fix (some) JDBC connection leaks.
  • Add missing parameters dashNS and linewidthNS to plot2d task.


Next Previous Up Contents
Up: Release Notes
Previous: Acknowledgements

STILTS - Starlink Tables Infrastructure Library Tool Set
Starlink User Note256
STILTS web page: http://www.starlink.ac.uk/stilts/
Author email: m.b.taylor@bristol.ac.uk
Mailing list: topcat-user@bristol.ac.uk