Next Previous Up Contents
Up: Release Notes
Previous: Acknowledgements


C.2 Version History

Releases to date have been as follows:

Version 0.1b (29 April 2005)
First public release
Version 0.2b (30 June 2005)
Version 1.0b (30 September 2005)
This is the first non-experimental release of STILTS, and it incorporates major changes and backward incompatibilities since version 0.2b.

Parameter system
The parameter system has undergone a complete rewrite; there is now only a single command "stilts", invoked using the stilts script or the stilts.jar jar file, and the various tasks are named as subsequent arguments on the command line. Command arguments are supplied after that. The new invocation syntax is described in detail elsewhere in this document. As well as invocation features such as improved on-line help, optional prompting, parameter defaulting, and more uniform access to common features, this will make it more straightforward to wrap these tasks for use in non-command-line environments, such as behind a SOAP or CORBA interface, or in a CEA-like execution environment.
Crossmatching
A new command tmatch2 has been introduced. This provides flexible and efficient crossmatching between two input tables. Future releases will provide commands for intra-table and multi-table matching.
Concatentation
A new command tcat has been introduced, which allows two tables to be glued together top-to-bottom. This is currently working but very rudimentary - improvements will be forthcoming in future releases.
Calculator
A new utility command calc has been introduced, which performs one-line expression evaluations from the command line.
Pipeline filters
The following new filter commands for use in tpipe and other commands have been introduced:
  • addskycoords: calculates new celestial coordinate pair from existing ones (FK4, FK5, ecliptic, galactic, supergalactic)
  • replacecol: replaces column data, using existing metadata
  • badval: replaces given 'magic' value with null
  • replaceval: replaces given 'magic' value with any specified value
  • tablename: edits table name
  • explodecols and explodecols commands replace explode

The new stream parameter of tpipe now allows you to write filter commands in an external file, to facilitate more manageable command lines.

Wildarding for column specification is now allowed for some filter commands.

Algebraic functions
  • New functions for converting time values between different coordinate systems (Modified Julian Date, ISO-8601, Julian Epoch and Besselian Epoch).
  • New RANDOM special function.
Documentation
SUN/256 has undergone many changes. Much of the tool documentation is now automatically generated from the code itself, which goes a long way to ensuring that the documentation is correct with respect to the current state of the code.

Version 1.0-1b (7 October 2005)
Fixed jar file manifest bug which prevented working on Java 1.5
Version 1.1 (10 May 2006)
A number of new features and capabilities have been introduced:
tcube Command
The new tcube command calculates N-dimensional histograms (density maps) from N columns of an input table and writes the result to a FITS file.
Processing Filters
The following new filters have been added:
  • stats filter provides the same information as the old stats output mode, but allows much more flexible use of the results. It can also calculate many new quantities, including quantiles, skew and kurtosis.
  • meta filter provides the same information as the old meta output mode, but allows much more flexible use of the results.
  • assert filter provides in-pipeline logical assertions.
  • uniq filter collapses multiple adjacent identical or similar rows.
  • sorthead filter provides a (usually) more efficient method of doing what you could previously do by combining sort and head filters.
  • colmeta filter adds/modifies metadata for selected columns.
  • check filter checks table in stream - for debugging purposes only.

Additionally usage of the sort filter has been changed so that it can now do everything that sortexpr used to be able to do; sortexpr is now withdrawn.

Output Modes
The following new output modes have been introduced:
  • plastic mode broadcasts the table to one or all registered PLASTIC listeners.
  • cgi mode writes the table to standard output in a form suitable for output from a CGI script.
  • discard mode throws away the table.
and usage of the following has been modified:
  • topcat mode now attempts to use PLASTIC (amongst other methods) to contact TOPCAT.
  • stats and meta modes are mildly deprecated in favour of the corresponding new filters (see above).
Other new features
  • New IPAC table format input handler added.
  • New csv-noheader format variant output handler added.
  • roundDecimal and formatDecimal functions introduced for more control over visual appearance of numeric values.
  • Experimental facilities for automatically generating a CEA application description file.
Bug fixes and minor improvements
  • Now copes with 'K'-format FITS binary table columns (64-bit integers).
  • Improved, though still imperfect, retention of table-wide metadata in VOTables.
  • Distinctions between null and false values in boolean columns are handled more carefully for FITS and VOTable files.
  • Efficiency improvement when writing FITS-plus format (now only requires a maximum of two passes rather than three of the input rows).
  • Added the mark.workaround system property which can optionally work around a bug in some input streams ("Resetting to invalid mark" errors).
  • Fixed a bug in Cartesian matching which failed to match if the required error in any dimension was zero.
  • Fixed erroneous reports about unknown ucd and utype attributes of TABLE element in votlint.
  • When joining tables, column name comparison to determine whether deduplication is required is now case-insensitive.
  • Error message improved when no automatic format detection is attempted for streamed tables.
  • Setting istream=true is now less likely to cause a "Can't re-read stream" error.
Version 1.2 (7 July 2006)
Column-oriented Storage
New features for permitting column-oriented storage (colfits format, new startable.storage policy "sideways") have been introduced. These can provide considerable efficiency improvements for certain tasks when working with very large (and especially wide) tables.
New VO commands
Added two new commands for querying Virtual Observatory services:
  • multicone - Makes multiple cone search queries to the same service
  • regquery - Queries the VO registry
These tasks are experimental and may be modified or renamed in future releases.
Other items
  • transpose filter added.
  • Added flux conversion functions (Jansky<->magnitude).
  • ISO-8601 strings now permit times of 24:00:00 as they should.
Version 1.2-1 (3 August 2006)
Version 1.3 (5 October 2006)
Table Concatenation
The old tcat command has been replaced by more capable tcat and tcatn commands. Between them these provide concatenation of an unlimited number of homogeneous or heterogeneous input tables. Additional columns may be added to indicate which of the input tables given output rows originated from.
Parameter value indirection
Certain parameters (in in tcat, cmd and friends) may now be specified in the form "@filename". This indicates that the value for the parameter is to be obtained by reading it from the named file. This is useful if a very long value is required for the parameter in question. The script parameter of tpipe has therefore been withdrawn, since it did just the same thing.
MySpace access
Direct access to the MySpace virtual file system is now provided by use of ivo:- or myspace:-type URLs.
Conversion functions
  • Time conversion functions between MJD and Decimal Year have been added (Section 9.5.2).
  • toHex and fromHex numeric conversion functions have been added (Section 9.5.4).
Documentation improvements
  • The HTML version of SUN/256 now uses CSS to provide better highlighting of commands etc.
  • The Output Modes and Processing Filter sections are now split into subsections to make the table of contents clearer.
  • The Command Reference section now has only one level of subsection listed in the table of contents to make it clearer.
Other new features and improvements
  • Added -J flag to stilts script for passing flags directly to Java.
  • Added new out parameter to votlint.
  • Added -ifndim and -ifshape flags to explodeall filter.
  • The exact match mode in tmatch2 now copes with array-valued columns.
  • Added force parameter to multicone task as a workaround for some broken services.
  • Added Sample (as opposed to Population) Standard Deviation/Variance calculation options to the stats filter.
  • Improved CEA description file output - now contains details of all tasks rather than just a few, as well as various improvements in documentation etc.
Bug fixes
  • Fixed erroneous complaints from votlint about utype attribute on RESOURCE elements.
  • Fixed a couple of minor crossmatching bugs (which wouldn't have affected results).
Version 1.3-1 (Starlink Hokulei release)
Version 1.3-2 (6 July 2007)
Version 1.3-3 (4 Sep 2007)
Version 1.3-4 (10 Sep 2007)
Version 1.3-5 (30 Oct 2007)
Version 1.4 (6 December 2007)
Table joins
This version provides more cross matching functionality. Added to the existing tmatch2 command are new tasks:
  • tskymatch2: stripped down version of tmatch2 for ease of use when matching with sky coordinates.
  • tmatch1: internal matcher, finds groups of objects within a table.
  • tmatchn: finds group or multiple-pair matches between multiple (>2) tables.

Two tasks have been renamed for improved clarity and consistency:

There has also been some enhancement and rationalisation of parameters for all table join tools (tmatch* as well as tjoin, coneskymatch and sqlskymatch):

  • All table join commands now use similar fixcols and suffix* parameters to control renaming of duplicated columns in output tables (note this replaces the old duptag* parameters in tmatch2).
  • Crossmatching tasks have a new progress parameter which allows you to configure whether progress is reported to the console.
  • The copycols parameter of coneskymatch and sqlskymatch now defaults to "*" (include all columns from input table in the output).

Section 7 of the manual has been somewhat rearranged and improved.

Other enhancements
  • FITS reader now imports table HDU header cards as table parameters.
  • CeaWriter can now output CEA service definition XML config file as well as app-description file (experimental - may be withdrawn).
Bug fixes
  • Embedded spaces in output ASCII format table column names are now substituted with underscores.
  • Fix a bug which caused an infinite number of dots to be printed when attempting a crossmatch with an empty input table.
  • Corrected votlint handling of TABLEDATA-type multi-dimensional char/unicodeChar arrays. These are now split up into strings by counting characters rather than using whitespace delimiters. I think it's doing the right thing now.
Version 1.4-1 (28 January 2008)
New RDBMS-related features
  • New command sqlclient, which is a general JDBC-based SQL command-line client.
  • New command sqlupdate, which allows updates to existing rows in SQL tables.
  • Some changes to tosql output mode:
    • choice of options for how to write to the database output table, controlled by new associated parameter write (can be create, dropcreate or append)
    • associated parameter newtable renamed dbtable
    • associated parameter database renamed db for consistency with other commands
Local and service-based matching command enhancements
  • New parameter scorecol added to tmatch2, coneskymatch and sqlskymatch commands, which controls adding a new column to match output tables containing a goodness-of-match value.
  • New parameter parallel added to coneskymatch task which allows multiple cone searches to be carried out in parallel.
  • New parameter erract added to coneskymatch which controls response to isolated failures in individual cone search queries.
General improvements
  • Improved error reporting (reasons for errors are now reported even without the -debug flag).
  • Add new help option help='*' which prints help for all parameters of a task at once.
  • Added (mostly undocumented) +verbose flag for reducing verbosity level.
  • Minor improvements to CEA app-description.
  • Downgraded from WARNING to INFO log messages about the (extremely common) VOTable syntax error of omitting a FIELD/PARAM element's datatype attribute.
Version 1.4-2 (26 March 2008)
Minor enhancements:
  • Add progress parameter to tmatchn.
  • Add emptyok parameter to coneskymatch.
Bugfixes:
  • Fixed pair matching performance bug (slower if tables were not given in the right order) introduced at v1.4.
  • Fixed null handling error in calc task.
  • Fixed error in stats filter cardinality value calculation.
  • Fixed minor bugs in suffix addition for matching commands fixcols.
  • Removed unformatted XML output in stats filter usage message.
  • Try to avoid exponential format in cone search URLs (some endpoints seem to require fixed point format).
  • Minor CEA fixes.
Version 2.0b (23 October 2008)
This version contains two new major items, plotting and server mode. Both work, but are missing desirable features and have not had extensive testing in the field, so should be considered experimental at this stage.
Plotting
Two table plotting commands are now provided: See also the new Plotting section in the manual.
Server/Servlet Mode
A new command server is provided which allows STILTS commands to be executed via HTTP. One purpose of this is to facilitate server-side use of the plotting commands co-located with data to generate on-the-fly graphical summaries of server-held datasets.
Smaller enhancements and bugfixes
  • Efficiency improvements (~25%? in both CPU time and memory usage) for HEALPix-based sky crossmatching (thanks to Nikolay Kouropatkine at Fermilab for a new version of the PixTools library).
  • New class Arrays added to algebraic functions.
  • New Appendix Commands by Category added to manual.
  • Add minReal and maxReal functions (max/min ignoring blank values) in class Arithmetic.
  • Sexagesimal field identification for ASCII input files is less stringent (now permits minutes or seconds equal to 60).
  • Minor CEA fixes.
  • HEALPix bug fix (PixTools bug fix update).
  • Fix bug in use of tcat's loccol parameter.
Version 2.0-1 (23 December 2008)
Version 2.0-2 (9 January 2009)
Version 2.0-3 (27 March 2009)
Version 2.0-4 (17 July 2009)
Version 2.0-5 (2 Oct 2009)
Version 2.1 (6 November 2009)
Note that the STIL API used by this release has changed in some backwardly incompatible ways, and may change further. If you're using STILTS as a library rather than an application you might want to wait for a later release when the API has settled down.
Version 2.1-1 (21 December 2009)
Version 2.1-2 (24 March 2010)
Version 2.2 (6 August 2010)
New capabilities for multi-table I/O have been introduced: There are some additional enhancements: And a number of bug fixes:

Finally, from this release STILTS requires version 1.5 (a.k.a. 5.0) of the Java J2SE Runtime Environment; it will no longer run on version 1.4, which is now very old. I don't expect this to cause compatibility issues for anyone, but I'm interested to hear if that's not the case.

Version 2.2-1 (23 December 2010)
Version 2.3 (9 May 2011)
TAP
The new commands tapquery and tapresume have been introduced. These provide support for the Table Access Protocol (TAP), and allow freeform queries in an SQL-like language to be made to remote databases.
Minor enhancements
  • Random Groups HDUs are now tolerated, though not interpreted, within FITS files.
  • Added soapout parameter to regquery command.
  • Added count, variance and stdev functions to Arrays.
  • Upgrade to JSAMP v1.2.
  • Improve text rendering in funcs window display.
  • Attempt case-sensitive matching before case-insensitive for column names.
  • Fix replaceval filter to work with Infinities.
Bug fixes and workarounds
  • JDBC table input handler now effectively downcasts BigInteger/BigDecimal types to Long/Double. The PostgreSQL JDBC driver seems to use the Big* types routinely for numeric values (which I don't think it used to do).
  • Add workaround for J2SE bug #4795134, which could cause errors when reading compressed FITS files.
  • Fix FITS character handling bug which could cause corrupted FITS files on output in presence of non-ASCII characters.
  • Fix (some) JDBC connection leaks.
  • Add missing parameters dashNS and linewidthNS to plot2d task.
Version 2.3-1 (30 June 2011)
Version 2.4 (27 October 2011)
Crossmatching:
  • Two new asymmetric match options best1 and best2 have been added for the find parameter in the pair matching commands tmatch2 and tskymatch2. They correspond to finding the best match in table B for each row in table A, and in crowded fields often provide more intuitive semantics than the previous symmetric best option (in non-crowded fields there is generally no difference). This replicates the matching performed by some other tools, including Aladin.
  • New matchers have been added to permit matching of general elliptical, rather than just circular, regions in both planar and sky coordinates; see 2d_ellipse, and skyellipse.
  • Another new matcher is available for dealing with per-object errors in Cartesian coordinates (previously per-object errors could only be handled in sky coords); see Nd_err.
  • Semantics of the skyerr matcher have changed slightly.
Expression language functions:
  • Algebraic functions involving angles are now mostly available using degrees as well as radians. The Coords class has been replaced by CoordsDegrees and CoordsRadians classes providing sky coordinate functions, and a new class TrigDegrees provides normal degree-based trigonometric functions alongside the radian-based versions in Maths. Some of the old function names have changed to make clear that they use radians and not degrees. This change should be much more convenient in most cases; sorry it's taken so long to get round to.
  • Add new join function is added to the Arrays class to combine all the elements of an array into a string.
taplint:
There are several bugfixes and changes related to the TAP validator tool taplint, mostly thanks to bug reports etc from the TAP community:
  • Improve test logic for record limiting queries.
  • Errors no longer reported (e.g. E-Qxx-CNAM) for unexpected TAP_SCHEMA table column ordering (when running query stage but no metadata acquisition stages).
  • Add new stage MDQ, which checks query result columns for all tables against declared metadata.
  • Add check of versioned and unversioned LANG variants.
  • Now uses corrected upload ID (ivo://ivoa.net/std/TAPRegExt#upload-*) as per most recent TAPRegExt draft.
Bug fixes and minor enhancements:
  • Add parameter parse to tapquery command, allowing pre-send syntax checking of submitted ADQL.
  • Add experimental system properties star.basicauth.user and star.basicauth.password.
  • Improve resilience of coneskymatch in the presence of unreliable or inconsistent DAL services.
  • A PARAMref element with no referent in a VOTable no longer causes an uncaught NullPointerException.
Version 2.5 (28 March 2013)
New coverage-related functionality:
  • Add new command pixsample which can sample pixel data from HEALPix table files (useful for things like Schlegel dust extinction). Also addpixsample filter, which does the same job.
  • Add new command pixfoot which can generate MOC (Multi-Order Coverage) maps.
  • Add MOC-based coverage filter to coneskymatch when using some Cone Search services (mostly VizieR). This uses the Multi-Order Coverage map service operated by CDS. It can make VizieR multi-cone queries much faster by not doing cone searches that are outside the coverage region of the catalogue in question.
  • Add new class Coverage to the expression language containing MOC-related functions (currently, just inMoc).
Other new capabilities:
  • Add IPAC table output format.
  • Add new class KCorrections to the expression language, containing a method for calculating K-corrections following the method of Chilingarian and Zolotukhin.
  • You can now reference tables in multi-extension FITS files by name (EXTNAME or EXTNAME-EXTVER) as an alternative to by HDU index.
VOTable enhancements:
  • VOTable input, output and validation are now supported for version 1.3 of the VOTable standard.
  • The version of the VOTable format used for VOTable output can now be selected, by using the system property votable.version. Output version is VOTable 1.2 by default.
  • votlint has been changed so that it handles different VOTable versions more capably. Versions 1.1+ are now validated against a schema (which is how those versions are defined) rather than against a DTD hacked to do the same job as the schema. VOTable 1.3 validation is now provided.
  • The votcopy command has a new version parameter to control output version, and a new nomagic parameter to control whether VALUES/null attributes are removed where appropriate.
  • Infinite floating point values are now correctly encoded in VOTable output ("+Inf"/"-Inf", not "Infinity"/"-Infinity" as in previous versions).
  • votlint is now stricter about floating point TD element contents.
  • VOTable output no longer writes the schemaLocation attribute by default.
Other enhancements:
  • Add new function hypot (=sqrt(x*x+y*y)) to the Maths class in expression language.
  • Add new split functions for string splitting to the Strings class in expression language.
  • Add -utype flags for addcol, replacecol, colmeta and setparam filters, and utype option for meta filter.
  • Some changes to the toString function: it now works on non-numeric values, gives the right answer for Long integers and character values, and returns a blank value rather than the string "null" or "NaN" for blank inputs.
  • Sexagesimal to numeric angle conversion functions now permit the seconds part of the sexagesimal string to be missing.
  • Changes to the IPAC format definition are accommodated: the "long"/"l" type, which is apparently now official, no longer generates a warning, and headers may now use minus signs instead of whitespace.
  • Add OBS stage (ObsTAP validation) to taplint.
  • Add more checks to CAP stage of taplint. Declared languages (including features) and output formats are now checked.
  • Tidy up error reporting a bit (fewer duplicate nested messages reported).
  • PNG graphics output no longer has transparent background.
  • Issue a warning for high values of coneskymatch parallel parameter.
  • Upgrade JSAMP library to version 1.3-3.
  • Upgrade Grégory Mantelet's ADQL library to version 1.1.
Bug fixes:
  • Fix serious and long-standing bug (bad TZERO header, causes subsequent reads to fail) for FITS output of boolean array columns.
  • Fix small but genuine sky matching bug. The effect was that near the poles matches near the specified threshold could be missed. The bug was in the PixTools library, fixed at the 2012-07-28 release.
  • Fix bug in tmatchn group mode which could result in output rows with columns from only a single table, i.e. not representing an inter-table match, even when join*=default.
  • Fix bug which failed when attempting to read FITS files with complex array columns (TFORMn=rC/rM).
  • Fix failure when caching very large sequential tables.
  • Fix bug in replacecol and replaceval filters which could cause truncation of strings in FITS and possibly VOTable output when the new value was longer than the previously declared maximum length.
  • Fix tcat, tcatn so that in most cases output column metadata is compatible with all input tables, not just the first one in terms of nullability, array shape etc.
  • Adjust SQL writer to avoid a type error for MySQL.
  • Fix bug in HMS sexagesimal formatting: minus sign was omitted from negative angles. Now the output is forced positive.
  • Cope with 1-column CSV files.
  • Use the correct form "rows"/"bytes" rather than "row"/"byte" for TAP capability unit values.
  • Fix error bar rendering bug which could result in diagonal lines being offset near the edge of plots.
Version 2.5-1 (1 July 2013)
New functionality
  • Add read-only support for CDF (NASA Common Data Format) files.
  • Add Median Absolute Deviation calculation (MedAbsDev and ScMedAbsDev) options to stats filter.
  • Improved handling of HTTP basic authorization. 401s now generate a useful message about the star.basicauth.* system properties if they have not been set up.
Bug fixes and minor enhancements
  • Fix CSV regression bug introduced at v2.5 - CSV files now work again with MSDOS-style line breaks.
  • Fixed FITS output bug which could result in badly-formed string-valued header cards (no closing quote).
  • Source code is now managed by git and not subversion. The format of the "Starjava revision" string reported by the -version flag has changed accordingly.
  • Output mode meta now copes better with array-valued table parameters.
  • Implemented fixes to reduce the chance of users inadvertently overloading external Cone/SIA/SSA services with multicone-like queries. First, fix it so that abandoned queries are properly terminated, rather than continuing to hit the server until completion or JVM shutdown. Second, implement a sensible default maximum value for the parallel parameter of skyconematch (though this may be adjusted with a system property).
  • Quoting behaviour has changed when generating SQL to write to RDBMS tables. This ought to reduce problems related to mixed-case identifiers. However, it is possible that it could lead to unforseen new anomalies.


Next Previous Up Contents
Up: Release Notes
Previous: Acknowledgements

STILTS - Starlink Tables Infrastructure Library Tool Set
Starlink User Note256
STILTS web page: http://www.starlink.ac.uk/stilts/
Author email: m.b.taylor@bristol.ac.uk
Mailing list: topcat-user@bristol.ac.uk