Next Previous Up Contents
Up: Release Notes
Previous: Acknowledgements


C.2 Version History

Releases to date have been as follows:

Version 0.1b (29 April 2005)
First public release
Version 0.2b (30 June 2005)
Version 1.0b (30 September 2005)
This is the first non-experimental release of STILTS, and it incorporates major changes and backward incompatibilities since version 0.2b.

Parameter system
The parameter system has undergone a complete rewrite; there is now only a single command "stilts", invoked using the stilts script or the stilts.jar jar file, and the various tasks are named as subsequent arguments on the command line. Command arguments are supplied after that. The new invocation syntax is described in detail elsewhere in this document. As well as invocation features such as improved on-line help, optional prompting, parameter defaulting, and more uniform access to common features, this will make it more straightforward to wrap these tasks for use in non-command-line environments, such as behind a SOAP or CORBA interface, or in a CEA-like execution environment.
Crossmatching
A new command tmatch2 has been introduced. This provides flexible and efficient crossmatching between two input tables. Future releases will provide commands for intra-table and multi-table matching.
Concatentation
A new command tcat has been introduced, which allows two tables to be glued together top-to-bottom. This is currently working but very rudimentary - improvements will be forthcoming in future releases.
Calculator
A new utility command calc has been introduced, which performs one-line expression evaluations from the command line.
Pipeline filters
The following new filter commands for use in tpipe and other commands have been introduced:
  • addskycoords: calculates new celestial coordinate pair from existing ones (FK4, FK5, ecliptic, galactic, supergalactic)
  • replacecol: replaces column data, using existing metadata
  • badval: replaces given 'magic' value with null
  • replaceval: replaces given 'magic' value with any specified value
  • tablename: edits table name
  • explodecols and explodecols commands replace explode

The new stream parameter of tpipe now allows you to write filter commands in an external file, to facilitate more manageable command lines.

Wildarding for column specification is now allowed for some filter commands.

Algebraic functions
  • New functions for converting time values between different coordinate systems (Modified Julian Date, ISO-8601, Julian Epoch and Besselian Epoch).
  • New RANDOM special function.
Documentation
SUN/256 has undergone many changes. Much of the tool documentation is now automatically generated from the code itself, which goes a long way to ensuring that the documentation is correct with respect to the current state of the code.

Version 1.0-1b (7 October 2005)
Fixed jar file manifest bug which prevented working on Java 1.5
Version 1.1 (10 May 2006)
A number of new features and capabilities have been introduced:
tcube Command
The new tcube command calculates N-dimensional histograms (density maps) from N columns of an input table and writes the result to a FITS file.
Processing Filters
The following new filters have been added:
  • stats filter provides the same information as the old stats output mode, but allows much more flexible use of the results. It can also calculate many new quantities, including quantiles, skew and kurtosis.
  • meta filter provides the same information as the old meta output mode, but allows much more flexible use of the results.
  • assert filter provides in-pipeline logical assertions.
  • uniq filter collapses multiple adjacent identical or similar rows.
  • sorthead filter provides a (usually) more efficient method of doing what you could previously do by combining sort and head filters.
  • colmeta filter adds/modifies metadata for selected columns.
  • check filter checks table in stream - for debugging purposes only.

Additionally usage of the sort filter has been changed so that it can now do everything that sortexpr used to be able to do; sortexpr is now withdrawn.

Output Modes
The following new output modes have been introduced:
  • plastic mode broadcasts the table to one or all registered PLASTIC listeners.
  • cgi mode writes the table to standard output in a form suitable for output from a CGI script.
  • discard mode throws away the table.
and usage of the following has been modified:
  • topcat mode now attempts to use PLASTIC (amongst other methods) to contact TOPCAT.
  • stats and meta modes are mildly deprecated in favour of the corresponding new filters (see above).
Other new features
  • New IPAC table format input handler added.
  • New csv-noheader format variant output handler added.
  • roundDecimal and formatDecimal functions introduced for more control over visual appearance of numeric values.
  • Experimental facilities for automatically generating a CEA application description file.
Bug fixes and minor improvements
  • Now copes with 'K'-format FITS binary table columns (64-bit integers).
  • Improved, though still imperfect, retention of table-wide metadata in VOTables.
  • Distinctions between null and false values in boolean columns are handled more carefully for FITS and VOTable files.
  • Efficiency improvement when writing FITS-plus format (now only requires a maximum of two passes rather than three of the input rows).
  • Added the mark.workaround system property which can optionally work around a bug in some input streams ("Resetting to invalid mark" errors).
  • Fixed a bug in Cartesian matching which failed to match if the required error in any dimension was zero.
  • Fixed erroneous reports about unknown ucd and utype attributes of TABLE element in votlint.
  • When joining tables, column name comparison to determine whether deduplication is required is now case-insensitive.
  • Error message improved when no automatic format detection is attempted for streamed tables.
  • Setting istream=true is now less likely to cause a "Can't re-read stream" error.
Version 1.2 (7 July 2006)
Column-oriented Storage
New features for permitting column-oriented storage (colfits format, new startable.storage policy "sideways") have been introduced. These can provide considerable efficiency improvements for certain tasks when working with very large (and especially wide) tables.
New VO commands
Added two new commands for querying Virtual Observatory services:
  • multicone - Makes multiple cone search queries to the same service
  • regquery - Queries the VO registry
These tasks are experimental and may be modified or renamed in future releases.
Other items
  • transpose filter added.
  • Added flux conversion functions (Jansky<->magnitude).
  • ISO-8601 strings now permit times of 24:00:00 as they should.
Version 1.2-1 (3 August 2006)
Version 1.3 (5 October 2006)
Table Concatenation
The old tcat command has been replaced by more capable tcat and tcatn commands. Between them these provide concatenation of an unlimited number of homogeneous or heterogeneous input tables. Additional columns may be added to indicate which of the input tables given output rows originated from.
Parameter value indirection
Certain parameters (in in tcat, cmd and friends) may now be specified in the form "@filename". This indicates that the value for the parameter is to be obtained by reading it from the named file. This is useful if a very long value is required for the parameter in question. The script parameter of tpipe has therefore been withdrawn, since it did just the same thing.
MySpace access
Direct access to the MySpace virtual file system is now provided by use of ivo:- or myspace:-type URLs.
Conversion functions
  • Time conversion functions between MJD and Decimal Year have been added (Section 10.5.6).
  • toHex and fromHex numeric conversion functions have been added (Section 10.5.3).
Documentation improvements
  • The HTML version of SUN/256 now uses CSS to provide better highlighting of commands etc.
  • The Output Modes and Processing Filter sections are now split into subsections to make the table of contents clearer.
  • The Command Reference section now has only one level of subsection listed in the table of contents to make it clearer.
Other new features and improvements
  • Added -J flag to stilts script for passing flags directly to Java.
  • Added new out parameter to votlint.
  • Added -ifndim and -ifshape flags to explodeall filter.
  • The exact match mode in tmatch2 now copes with array-valued columns.
  • Added force parameter to multicone task as a workaround for some broken services.
  • Added Sample (as opposed to Population) Standard Deviation/Variance calculation options to the stats filter.
  • Improved CEA description file output - now contains details of all tasks rather than just a few, as well as various improvements in documentation etc.
Bug fixes
  • Fixed erroneous complaints from votlint about utype attribute on RESOURCE elements.
  • Fixed a couple of minor crossmatching bugs (which wouldn't have affected results).
Version 1.3-1 (Starlink Hokulei release)
Version 1.3-2 (6 July 2007)
Version 1.3-3 (4 Sep 2007)
Version 1.3-4 (10 Sep 2007)
Version 1.3-5 (30 Oct 2007)
Version 1.4 (6 December 2007)
Table joins
This version provides more cross matching functionality. Added to the existing tmatch2 command are new tasks:
  • tskymatch2: stripped down version of tmatch2 for ease of use when matching with sky coordinates.
  • tmatch1: internal matcher, finds groups of objects within a table.
  • tmatchn: finds group or multiple-pair matches between multiple (>2) tables.

Two tasks have been renamed for improved clarity and consistency:

There has also been some enhancement and rationalisation of parameters for all table join tools (tmatch* as well as tjoin, coneskymatch and sqlskymatch):

  • All table join commands now use similar fixcols and suffix* parameters to control renaming of duplicated columns in output tables (note this replaces the old duptag* parameters in tmatch2).
  • Crossmatching tasks have a new progress parameter which allows you to configure whether progress is reported to the console.
  • The copycols parameter of coneskymatch and sqlskymatch now defaults to "*" (include all columns from input table in the output).

Section 7 of the manual has been somewhat rearranged and improved.

Other enhancements
  • FITS reader now imports table HDU header cards as table parameters.
  • CeaWriter can now output CEA service definition XML config file as well as app-description file (experimental - may be withdrawn).
Bug fixes
  • Embedded spaces in output ASCII format table column names are now substituted with underscores.
  • Fix a bug which caused an infinite number of dots to be printed when attempting a crossmatch with an empty input table.
  • Corrected votlint handling of TABLEDATA-type multi-dimensional char/unicodeChar arrays. These are now split up into strings by counting characters rather than using whitespace delimiters. I think it's doing the right thing now.
Version 1.4-1 (28 January 2008)
New RDBMS-related features
  • New command sqlclient, which is a general JDBC-based SQL command-line client.
  • New command sqlupdate, which allows updates to existing rows in SQL tables.
  • Some changes to tosql output mode:
    • choice of options for how to write to the database output table, controlled by new associated parameter write (can be create, dropcreate or append)
    • associated parameter newtable renamed dbtable
    • associated parameter database renamed db for consistency with other commands
Local and service-based matching command enhancements
  • New parameter scorecol added to tmatch2, coneskymatch and sqlskymatch commands, which controls adding a new column to match output tables containing a goodness-of-match value.
  • New parameter parallel added to coneskymatch task which allows multiple cone searches to be carried out in parallel.
  • New parameter erract added to coneskymatch which controls response to isolated failures in individual cone search queries.
General improvements
  • Improved error reporting (reasons for errors are now reported even without the -debug flag).
  • Add new help option help='*' which prints help for all parameters of a task at once.
  • Added (mostly undocumented) +verbose flag for reducing verbosity level.
  • Minor improvements to CEA app-description.
  • Downgraded from WARNING to INFO log messages about the (extremely common) VOTable syntax error of omitting a FIELD/PARAM element's datatype attribute.
Version 1.4-2 (26 March 2008)
Minor enhancements:
  • Add progress parameter to tmatchn.
  • Add emptyok parameter to coneskymatch.
Bugfixes:
  • Fixed pair matching performance bug (slower if tables were not given in the right order) introduced at v1.4.
  • Fixed null handling error in calc task.
  • Fixed error in stats filter cardinality value calculation.
  • Fixed minor bugs in suffix addition for matching commands fixcols.
  • Removed unformatted XML output in stats filter usage message.
  • Try to avoid exponential format in cone search URLs (some endpoints seem to require fixed point format).
  • Minor CEA fixes.
Version 2.0b (23 October 2008)
This version contains two new major items, plotting and server mode. Both work, but are missing desirable features and have not had extensive testing in the field, so should be considered experimental at this stage.
Plotting
Two table plotting commands are now provided: See also the new Plotting section in the manual.
Server/Servlet Mode
A new command server is provided which allows STILTS commands to be executed via HTTP. One purpose of this is to facilitate server-side use of the plotting commands co-located with data to generate on-the-fly graphical summaries of server-held datasets.
Smaller enhancements and bugfixes
  • Efficiency improvements (~25%? in both CPU time and memory usage) for HEALPix-based sky crossmatching (thanks to Nikolay Kouropatkine at Fermilab for a new version of the PixTools library).
  • New class Arrays added to algebraic functions.
  • New Appendix Commands by Category added to manual.
  • Add minReal and maxReal functions (max/min ignoring blank values) in class Arithmetic.
  • Sexagesimal field identification for ASCII input files is less stringent (now permits minutes or seconds equal to 60).
  • Minor CEA fixes.
  • HEALPix bug fix (PixTools bug fix update).
  • Fix bug in use of tcat's loccol parameter.
Version 2.0-1 (23 December 2008)
Version 2.0-2 (9 January 2009)
Version 2.0-3 (27 March 2009)
Version 2.0-4 (17 July 2009)
Version 2.0-5 (2 Oct 2009)
Version 2.1 (6 November 2009)
Note that the STIL API used by this release has changed in some backwardly incompatible ways, and may change further. If you're using STILTS as a library rather than an application you might want to wait for a later release when the API has settled down.
Version 2.1-1 (21 December 2009)
Version 2.1-2 (24 March 2010)
Version 2.2 (6 August 2010)
New capabilities for multi-table I/O have been introduced: There are some additional enhancements: And a number of bug fixes:

Finally, from this release STILTS requires version 1.5 (a.k.a. 5.0) of the Java J2SE Runtime Environment; it will no longer run on version 1.4, which is now very old. I don't expect this to cause compatibility issues for anyone, but I'm interested to hear if that's not the case.

Version 2.2-1 (23 December 2010)
Version 2.3 (9 May 2011)
TAP
The new commands tapquery and tapresume have been introduced. These provide support for the Table Access Protocol (TAP), and allow freeform queries in an SQL-like language to be made to remote databases.
Minor enhancements
  • Random Groups HDUs are now tolerated, though not interpreted, within FITS files.
  • Added soapout parameter to regquery command.
  • Added count, variance and stdev functions to Arrays.
  • Upgrade to JSAMP v1.2.
  • Improve text rendering in funcs window display.
  • Attempt case-sensitive matching before case-insensitive for column names.
  • Fix replaceval filter to work with Infinities.
Bug fixes and workarounds
  • JDBC table input handler now effectively downcasts BigInteger/BigDecimal types to Long/Double. The PostgreSQL JDBC driver seems to use the Big* types routinely for numeric values (which I don't think it used to do).
  • Add workaround for J2SE bug #4795134, which could cause errors when reading compressed FITS files.
  • Fix FITS character handling bug which could cause corrupted FITS files on output in presence of non-ASCII characters.
  • Fix (some) JDBC connection leaks.
  • Add missing parameters dashNS and linewidthNS to plot2d task.
Version 2.3-1 (30 June 2011)
Version 2.4 (27 October 2011)
Crossmatching:
  • Two new asymmetric match options best1 and best2 have been added for the find parameter in the pair matching commands tmatch2 and tskymatch2. They correspond to finding the best match in table B for each row in table A, and in crowded fields often provide more intuitive semantics than the previous symmetric best option (in non-crowded fields there is generally no difference). This replicates the matching performed by some other tools, including Aladin.
  • New matchers have been added to permit matching of general elliptical, rather than just circular, regions in both planar and sky coordinates; see 2d_ellipse, and skyellipse.
  • Another new matcher is available for dealing with per-object errors in Cartesian coordinates (previously per-object errors could only be handled in sky coords); see Nd_err.
  • Semantics of the skyerr matcher have changed slightly.
Expression language functions:
  • Algebraic functions involving angles are now mostly available using degrees as well as radians. The Coords class has been replaced by CoordsDegrees and CoordsRadians classes providing sky coordinate functions, and a new class TrigDegrees provides normal degree-based trigonometric functions alongside the radian-based versions in Maths. Some of the old function names have changed to make clear that they use radians and not degrees. This change should be much more convenient in most cases; sorry it's taken so long to get round to.
  • Add new join function is added to the Arrays class to combine all the elements of an array into a string.
taplint:
There are several bugfixes and changes related to the TAP validator tool taplint, mostly thanks to bug reports etc from the TAP community:
  • Improve test logic for record limiting queries.
  • Errors no longer reported (e.g. E-Qxx-CNAM) for unexpected TAP_SCHEMA table column ordering (when running query stage but no metadata acquisition stages).
  • Add new stage MDQ, which checks query result columns for all tables against declared metadata.
  • Add check of versioned and unversioned LANG variants.
  • Now uses corrected upload ID (ivo://ivoa.net/std/TAPRegExt#upload-*) as per most recent TAPRegExt draft.
Bug fixes and minor enhancements:
  • Add parameter parse to tapquery command, allowing pre-send syntax checking of submitted ADQL.
  • Add experimental system properties star.basicauth.user and star.basicauth.password.
  • Improve resilience of coneskymatch in the presence of unreliable or inconsistent DAL services.
  • A PARAMref element with no referent in a VOTable no longer causes an uncaught NullPointerException.
Version 2.5 (28 March 2013)
New coverage-related functionality:
  • Add new command pixsample which can sample pixel data from HEALPix table files (useful for things like Schlegel dust extinction). Also addpixsample filter, which does the same job.
  • Add new command pixfoot which can generate MOC (Multi-Order Coverage) maps.
  • Add MOC-based coverage filter to coneskymatch when using some Cone Search services (mostly VizieR). This uses the Multi-Order Coverage map service operated by CDS. It can make VizieR multi-cone queries much faster by not doing cone searches that are outside the coverage region of the catalogue in question.
  • Add new class Coverage to the expression language containing MOC-related functions (currently, just inMoc).
Other new capabilities:
  • Add IPAC table output format.
  • Add new class KCorrections to the expression language, containing a method for calculating K-corrections following the method of Chilingarian and Zolotukhin.
  • You can now reference tables in multi-extension FITS files by name (EXTNAME or EXTNAME-EXTVER) as an alternative to by HDU index.
VOTable enhancements:
  • VOTable input, output and validation are now supported for version 1.3 of the VOTable standard.
  • The version of the VOTable format used for VOTable output can now be selected, by using the system property votable.version. Output version is VOTable 1.2 by default.
  • votlint has been changed so that it handles different VOTable versions more capably. Versions 1.1+ are now validated against a schema (which is how those versions are defined) rather than against a DTD hacked to do the same job as the schema. VOTable 1.3 validation is now provided.
  • The votcopy command has a new version parameter to control output version, and a new nomagic parameter to control whether VALUES/null attributes are removed where appropriate.
  • Infinite floating point values are now correctly encoded in VOTable output ("+Inf"/"-Inf", not "Infinity"/"-Infinity" as in previous versions).
  • votlint is now stricter about floating point TD element contents.
  • VOTable output no longer writes the schemaLocation attribute by default.
Other enhancements:
  • Add new function hypot (=sqrt(x*x+y*y)) to the Maths class in expression language.
  • Add new split functions for string splitting to the Strings class in expression language.
  • Add -utype flags for addcol, replacecol, colmeta and setparam filters, and utype option for meta filter.
  • Some changes to the toString function: it now works on non-numeric values, gives the right answer for Long integers and character values, and returns a blank value rather than the string "null" or "NaN" for blank inputs.
  • Sexagesimal to numeric angle conversion functions now permit the seconds part of the sexagesimal string to be missing.
  • Changes to the IPAC format definition are accommodated: the "long"/"l" type, which is apparently now official, no longer generates a warning, and headers may now use minus signs instead of whitespace.
  • Add OBS stage (ObsTAP validation) to taplint.
  • Add more checks to CAP stage of taplint. Declared languages (including features) and output formats are now checked.
  • Tidy up error reporting a bit (fewer duplicate nested messages reported).
  • PNG graphics output no longer has transparent background.
  • Issue a warning for high values of coneskymatch parallel parameter.
  • Upgrade JSAMP library to version 1.3-3.
  • Upgrade Grégory Mantelet's ADQL library to version 1.1.
Bug fixes:
  • Fix serious and long-standing bug (bad TZERO header, causes subsequent reads to fail) for FITS output of boolean array columns.
  • Fix small but genuine sky matching bug. The effect was that near the poles matches near the specified threshold could be missed. The bug was in the PixTools library, fixed at the 2012-07-28 release.
  • Fix bug in tmatchn group mode which could result in output rows with columns from only a single table, i.e. not representing an inter-table match, even when join*=default.
  • Fix bug which failed when attempting to read FITS files with complex array columns (TFORMn=rC/rM).
  • Fix failure when caching very large sequential tables.
  • Fix bug in replacecol and replaceval filters which could cause truncation of strings in FITS and possibly VOTable output when the new value was longer than the previously declared maximum length.
  • Fix tcat, tcatn so that in most cases output column metadata is compatible with all input tables, not just the first one in terms of nullability, array shape etc.
  • Adjust SQL writer to avoid a type error for MySQL.
  • Fix bug in HMS sexagesimal formatting: minus sign was omitted from negative angles. Now the output is forced positive.
  • Cope with 1-column CSV files.
  • Use the correct form "rows"/"bytes" rather than "row"/"byte" for TAP capability unit values.
  • Fix error bar rendering bug which could result in diagonal lines being offset near the edge of plots.
Version 2.5-1 (1 July 2013)
New functionality
  • Add read-only support for CDF (NASA Common Data Format) files.
  • Add Median Absolute Deviation calculation (MedAbsDev and ScMedAbsDev) options to stats filter.
  • Improved handling of HTTP basic authorization. 401s now generate a useful message about the star.basicauth.* system properties if they have not been set up.
Bug fixes and minor enhancements
  • Fix CSV regression bug introduced at v2.5 - CSV files now work again with MSDOS-style line breaks.
  • Fixed FITS output bug which could result in badly-formed string-valued header cards (no closing quote).
  • Source code is now managed by git and not subversion. The format of the "Starjava revision" string reported by the -version flag has changed accordingly.
  • Output mode meta now copes better with array-valued table parameters.
  • Implemented fixes to reduce the chance of users inadvertently overloading external Cone/SIA/SSA services with multicone-like queries. First, fix it so that abandoned queries are properly terminated, rather than continuing to hit the server until completion or JVM shutdown. Second, implement a sensible default maximum value for the parallel parameter of skyconematch (though this may be adjusted with a system property).
  • Quoting behaviour has changed when generating SQL to write to RDBMS tables. This ought to reduce problems related to mixed-case identifiers. However, it is possible that it could lead to unforseen new anomalies.
  • More toString overloads - now works for byte and boolean values too.
Version 2.5-2 (7 March 2014)
Version 2.5-3 (4 July 2014)
New and improved functionality:
  • Add new command cdsskymatch. In most cases (for querying tables that can be found in VizieR) this can and should be used instead of coneskymatch - it's much faster.
  • Commands coneskymatch, sqlskymatch and pixfoot will now guess RA/Dec columns if relevant parameters are left blank.
  • Added new graphics output format png-transp to generate PNG files with transparent backgrounds.
  • Upgraded Gregory Mantelet's ADQL library to version 1.2. Better ADQL parsing.
Improvements and adjustments to taplint:
  • Rework taplint API to facilitate static acquisition of report codes during programmatic use. A few error codes have changed.
  • Add new "duff query" test to taplint.
  • Avoid taplint MDQ stage data type mismatch error report for BOOLEAN/boolean declared/returned data.
  • taplint now takes steps to ensure that TAP_SCHEMA column list query is not truncated.
  • taplint now flags absence of ObsCore table with I[nfo] not F[ailure] status.
  • Change the implementation of taplint stages which perform validation against XSD schemas. Schemas from external namespaces may now be imported and used. The CPV stage, which was previously broken and disabled by default, is now fixed and enabled by default. Known/expected schemas are stored locally, and a warning is reported if external ones are used. Schema validation seems remarkably complicated, so it's possible there are still errors in this implementation - if you suspect so, please report it.
  • Add missing geometric reserved words to ADQL reserved word list. This fixes some problems with column names like "DISTANCE" in taplint tests.
  • Fixed some bugs related to TAP table uploads. In particular these could cause incorrect table upload error reports in taplint.
Version 3.0 (3 October 2014)
New plotting commands:
A set of new plotting commands are provided which give comprehensive access to all the new-style visualisation capabilities available in TOPCAT v4. These commands are documented in Section 8. These commands, and the underlying visualisation facilities, are considerably more capable than the, now deprecated, old-style plot commands plot2d, plot3d and plothist.
Programmatic invocation:
Programmatic invocation of STILTS tasks from third-party java code is now officially sanctioned and documented in the new Section 11. To support this changes have been made to the parameter system (Parameter class now supports generics) and there are some visible changes to the user documentation as well (parameters now report their data type, and tasks report their classname). Normal (e.g. command-line) usage should not undergo any changes, but a fair bit of UI code has changed, so unexpected problems are possible.
Other items:
  • Add new output mode gui, which displays the table data in a scrollable window on the screen.
  • Add new -allowunused flag to the stilts command. If this is set, then unused parameter settings on the command line just result in a warning, not failure of the command.
  • Attempting to write FITS tables with >999 columns now fails with a more helpful error message.
  • Improved Unicode handling in VOTables. Fixed a serious bug in votcopy that generated unreadable output to BINARY or BINARY2 serialization if any non-empty column had datatype="unicodeChar". Also improved behaviour when copying between tables with unicodeChar columns; these are usually preserved now, rather than being squashed to datatype char. Some lurking Unicode-related issues remain.
  • The TAP client now tolerates whitespace around UWS status codes.
  • taplint: downgrade unknown post-table QUERY_STATUS value message from Error to Warning.
Version 3.0-1 (13 November 2014)
New functionality:
  • Add (experimental) read-only support for Gaia/DPAC GBIN format.
  • Add new task tapskymatch.
  • Functions in class Coverage adjusted: new function nearMoc, and MOC can be identified by VizieR table IDs as well as by filename/URL.
  • For repeat filter, add -row|-table flags to control sequence of output rows.
  • For setparam and repeat filters, allow use of an algebraic expression for values, not just a literal value.
  • Add special values $ncol and $nrow to the expression language to refer to the column and row counts in a table. The special variable index is also deprecated in favour of $index or $0.
Bugfixes and minor improvements:
  • Add some more colour maps for aux/density shading.
  • Fix stilts invocation script to pick up classes from stilts.jar in script directory in preference to other places (e.g. topcat-full.jar).
  • Fix taplint to permit application/xml not just text/xml content-type where appropriate (UWS stage).
  • Fix taplint so it doesn't warn (W-TMV-UNSC) about unknown VOSITables schema.
  • Fix taplint so that unicodeChar matches CHAR/VARCHAR in the same way as char for column type declaration purposes.
  • Fix taplint so that capabilities document can have TAPRegExt dataModel ivo-id elements with xs:anyURI rather than vr:IdentifierURI (only a warning is issued in the latter case), in anticipation of TAPRegExt-1.0 Erratum #1.
  • Adjust taplint to handle adql:TIMESTAMP columns more carefully on upload and retrieval.
  • Update JSAMP to v1.3.5.
Version 3.0-2 (6 February 2015)
Plotting enhancements:
  • Linear fitting of points is now available using the linearfit layer type for plot2plane. Points may be weighted.
  • You can add titles to plots using the new title parameter.
  • New plot layer type sizexy allows plotting (optionally autoscaled) markers with horizontal and vertical extents independently determined by input data.
  • More flexibility when assigning colour maps, in aux and density shading modes, and spectrogram layer. New parameters *func allow assignment of different data->ramp mapping functions (sqrt and square as well as linear and logarithmic), and new parameters *quant allow quantisation of the colour map to discrete levels.
  • Replace maxsizeN parameter with autoscaleN for size plot layer type. You can now optionally turn off autoscaling and specify marker size in pixels instead.
  • Add auxcrowd parameter to plot2 tasks to influence tick crowding on aux axis colour ramp. Also adjust default to use fewer ticks.
  • Add some "dart" options (fixed-base open or filled triangles) for plotting vectors (see arrowN parameter in layers like xyvector).
  • Add some "triangle" options (variable-base open or filled triangles) for plotting ellipses (see ellipseN parameter in layers like xyellipse).
  • Histogram normalisation option adjusted so that total area under bars, rather than total height of bars, is fixed.
  • The PlotDisplay class that forms the result of plot2 commands can now have PointSelectionListeners registered on it. This lets you determine what point a user has clicked on if you're using the plotting classes from third party java code.
FITS I/O:
  • Reworked part of the FITS table input implementation, in particular adjusting the way memory mapping is done to reduce resource requirements on some platforms. If you notice any difference, it should be reduced virtual and perhaps resident memory usage, and some (~10%?) performance improvements, when reading large FITS/colfits files. If you were previously having problems with large memory allocations leading to disk thrashing and system lockup when scanning files larger than RAM (this didn't happen on all OSes), these will hopefully have gone away. However, please report anything that appears to be working worse than before, or continued memory usage issues.
  • Colfits files can now be accessed from streams, not just uncompressed disk files (though that's not necessarily a good idea).
Bugfixes and workarounds:
  • Fixed a query bug (missing REQUEST=queryData parameter) in the multi-SSA mode (servicetype=ssa) of coneskymatch. This long-standing bug would have stopped this command working at all with well-behaved SSA services.
  • Fixed error in fits-var output (PCOUNT header card did not include block alignment gap).
  • Graphics coordinates are now calculated in floating point rather than as integers. This fixes problems that could cause scaled vectors, ellipses etc to be drawn with shapes or orientations badly wrong due to rounding errors. It also improves plotting of analytic functions, especially to vector contexts (PDF/EPS).
  • Fix some problems to do with zooming to very large/small plot axis ranges.
  • Hide error bars (etc) that would extend to negative values on logarithmic axes; previously they were being drawn in anomalous places.
  • Fix NullPointerException bug when null value was supplied to multi-word parameter (e.g. tcube).
  • Fix Aux axis positioning for 3D plots so that the numeric labels don't get snipped off at top and bottom.
  • Add a hack that allows LDAC FITS tables to be treated sensibly in auto-format-detection mode.
  • Make VOTable handling more robust against unknown (illegal) datatypes.
  • Add missing parameters auxmax, auxmin to plotting task documentation.
Version 3.0-3 (14 April 2015)
Version 3.0-4 (17 August 2015)
Bugfixes (some significant):
  • Fix a serious bug in processing of FITS bit vector (TFORMn='rX') columns. Values read from these columns are presented as a boolean[] array. In all previous versions of STIL the bits have appeared in that array in the wrong sequence (LSB..MSB per byte rather than the other way round). Apologies to anyone who may have got incorrect science results from this error in the past, and thanks to Paul Price for helping to diagnose it.
  • Fix a less serious bug with TFORMn='rX' processing; attempting to read a single-element bit vector column (TFORMn=1X or X) previously resulted in an error making the file unreadable. Values read from such columns are now presented as Boolean scalars.
  • Fix a VOTable reading bug relating to bit vector data (datatype="bit") appearing in BINARY/BINARY2 serializations. This one was more obvious, it would usually generate an error when attempting to read the file.
  • Fix serious bug in time conversion for CDF TIME_TT2000 data types.
  • Fix a bug in votcopy that converted columns from datatype unsignedByte to short when transforming. Since v3.0-3 this is no longer necessary. In the case of converting to a binary serialization, since v3.0-3 this was causing it to generate unreadable VOTable output.
  • Fix a bug in votcopy that failed to handle columns with datatype bit. In the case of converting to a binary serialization, these were in all previous versions generating unreadable VOTable output. Now they convert them to columns with datatype boolean (not perfect, but better).
  • Fix skyvector bug: dlat and dlon values were being used the wrong way round.
  • Upgrade JEL to v2.0.2. Fixes problem with evaluating void-typed expressions, and possibly some other obscure bugs.
  • Some taplint bug fixes.
Behaviour changes:
  • Changes to the way that TAP service table/column name reports are interpreted (to conform to original intention of TAP standard). Taplint now checks that table/column names from TAP_SCHEMA and /tables endpoint are regular-or-delimited-identifiers, but no longer submits example queries using supplied column names wrapped in additional quotes.
  • Modify the heuristics that determine whether the first row of a CSV file is a header.
Enhancements (mostly minor):
  • Added some functions to the Arrays class that return array-valued results from array-valued parameters: add, subtract, multiply, divide, reciprocal, condition.
  • Improve error reporting in the face of non-VOTable TAP error responses. In many cases this makes it much easier to see what's going wrong with a TAP query.
  • As a diagnostic tool, when making TAP queries, a log message giving a roughly equivalent curl(1) command is now issued at the CONFIG level (visible using flags -verbose -verbose).
  • New taplint parameter maxtable limits the number of tables tested in the stage that queries data from each individual table (MDQ). May be useful for very large services.
  • New tapquery parameter upvotformat to determine what VOTable serialization variant is used to transmit uploaded tables to the TAP server. Previously uploads were always BINARY which ought to work, but the parameter now defaults to TABLEDATA, since some services (e.g. CADC) currently fail with binary uploads.
  • Minor improvement to version reporting (reports java specification version, no longer issues warning for absent revision string).
  • Update JCDF library to v1.1 (minor changes to do with leap seconds).
Version 3.0-5 (22 October 2015)
Version 3.0-6 (27 November 2015)
Crossmatching bug fix
Fix a long-standing crossmatch bug relating to range restriction during pre-processing. This could have caused missed associations (but not false positives) near the edge of coverage regions when using per-row errors, if the scale of the errors differed (especially differed significantly) between the matched tables. It affected matcher values of <n>-d_err, skyerr, 2d_ellipse and skyellipse only. Thanks to Grant Kennedy (IoA) for reporting this bug.
Density plots
Some more options for making weighted density plots have been added. Since v3.0 the Density shading mode has let you see the density of plotted points, but this lacked some features. Three new ways to do density plots are added; these all give you the option of weighting by an additional coordinate (like the Aux mode), choosing the combination method (mean, median, sum, max, ...), and displaying the quantitative value-colour mapping on the shared colour ramp (previously aux axis) beside the plot. The new density plots are:
  • Weighted shading mode, using shaped marker kernels on the screen pixel grid, available for all plot types
  • SkyDensity layer, using HEALPix bins on the celestial sphere, for the Sky plot
  • Density layer (later obsoleted by Grid layer), using square N*N screen pixel bins, for the Plane plot
The details are somewhat experimental and may undergo some adjustments in future releases (feedback welcome).
Colour maps
There are various changes affecting selection and display of colour maps used for density and aux axis shading:
  • The default colour map for Aux mode, and other layers using the shared colour map, is no longer Rainbow! It's Inferno. Rainbow colour maps are much hated by visualisation experts. Of course you can still choose Rainbow if you like.
  • Add some new colour maps: Viridis, Inferno, Magma and Plasma from Matplotlib 1.5, the SRON rainbow variant developed by Paul Tol, some diverging maps (HotCold, RdBu, PiYG, BrBG) and a qualitative constant chroma/luminance map HueCL.
  • The options for Density and Aux shading are now mostly the same as each other except where there's good reason to differ. Previously they were different in haphazard ways.
  • An attempt is made to give the default form of each colour map a sensible name, without leading minus signs.
  • Fix it so that the whole range of each map is distinguishable from white. This is a good idea when you're plotting symbols on a white background, which is common in stilts. Perhaps there are cases it's not such a good idea; if you think so, complain and I may change it back.
  • Try to fix it so that all the colour maps go in the same direction (light->dark) where applicable.
  • Throw out a couple of particularly useless colour maps.
  • Colour map ramp display is now different for non-absolute maps; their effect is shown on a selection of base colours, not just for one base colour.
Minor items
  • Try harder to identify epoch columns (suitable for time plot), in particular look for VOTable xtype of JD or MJD, and units of year.
  • Add some functions to the Tilings class to do with solid angles (healpixSqdeg, healpixSteradians, steradiansToSqdeg, sqdegToSteradians, SQDEG).
  • Fix plot bug; titles were painted in white for pixel output formats.
  • Rationalise plot report logging. Some more diagnostic information about plots is now logged at the INFO level (visible if topcat is run with the -verbose flag).
Version 3.0-7 (10 June 2016)
Expression Language
The JEL library underlying the expression language parser has been upgraded to v2.1.1 (thanks to Konstantin Metlov), now supporting variable-length argument lists among other things. This allows the following improvements:
  • New functions that support any number of arguments are provided: array, intArray, stringArray in class Arrays; concat, join in class Strings; and sum, mean, variance, stdev, min, max, median, countTrue in new class Lists.
  • Some old lists of similarly-named functions with fixed numbers of arguments have been replaced by single functions that take an arbitrary number of arguments (e.g. array(x1), array(x1,x2), ... array(x1,x2,x3,x4,x5,x6,x7,x8) replaced by array(values...)).
  • The 2-argument min/max functions in class Arithmetic have been renamed minNaN/maxNaN to avoid confusion, but in most cases existing expressions involving min/max will work as before.
  • Some functions that used to require string arguments will now auto-convert numeric types (e.g. concat(toString(RA),";",toString(DEC)) can now be written concat(RA,";",DEC)).
  • You can now implement user-defined functions with variable numbers of arguments.
  • Writing large (>=2**31) literal integers used to fail with an inscrutable error message. Now the message tells you to append the "L" character.
Time plots
The plot2time command has been enhanced so that it can make multi-zone plots - multiple plots stacked vertically that share the same horizontal (time) axis but have independent vertical axes. The time plot itself and this multi-zone feature are currently experimental; in future versions they may be improved or changed, and the multi-zone feature may be extended to other plot types. Some other changes and fixes have gone along with this:
  • A few API changes have been made to support multi-zone plots, including generalising the NavigationListener interface and rearranging some PlotDisplay and AbstractPlot2Task constructor/factory method arguments. For single-zone plots the changes are not very substantial. This only affects you if you are using the STILTS classes as a java plotting library. If that applies to you and you have trouble upgrading, I'm happy to provide assistance.
  • plot2time now supports shading modes (shadingN and associated parameters).
  • The spectrogram layer now uses the (per-plot, or more precisely now per-zone) Aux colour map rather than a layer-specific colour map. This means that the colour ramp is displayed alongside the plot, but also that some parameters have been renamed (e.g. auxmapZ replaces spectromapN, where Z is an optional zone suffix and N is an optional layer suffix).
  • The function layer type now works in the time plot, rather than throwing an error. However, it doesn't work very well, since the time coordinate is in unix seconds rather than something more user-friendly.
  • Fixed a serious bug in ISO-8601 axis labelling. In some cases axis labels were being drawn at positions badly different from the correct position.
Miscellaneous enhancements and changes
  • This and subsequent releases target Java SE 6, so will no longer run under the (now very ancient) Java 5 runtime.
  • Provide more careful documentation of licensing arrangements. The distributed LICENSE.txt file notes that the starjava code is LGPL, and documents licenses for each third-party dependency.
  • Add Fill layer type for Plane and Time plots.
  • tapquery and tapresume now use blocking HTTP requests rather than repeated polls to wait for asynchronous TAP job completion from services that declare themselves UWS 1.1 compliant.
  • Add new parameters executionduration and destruction to tapquery command. These let you request resource limit adjustments when submitting an asynchronous TAP job.
  • Improve sky plot border painting.
  • Clean up noisy Cubehelix colour map.
  • New function countTrue in class Arrays.
  • New stage EXA for taplint checks TAP /examples endpoint. Note the details of the examples format are still under discussion, (this version targets WD-DALI-1.1-20160415 & WD-TAP-1.1-20160428, somewhat informed by TAPNotes-2013-12-13), so the details may change in future.
  • taplint now validates ObsCore 1.1 where declared alongside ObsCore 1.0. Currently uses PR-ObsCore-v1.1-20160330.
  • The taplint API has changed slightly: the class that used to be Reporter is now called TextOutputReporter. If you are using taplint programmatically you may need to make small changes. This results from some refactoring that makes it easier to customise taplint output.
  • Replaced opaque config option with transparency for plane and sky density plots.
  • Changed implementation of GIF exporter for plots, from Acme to ImageIO. Shouldn't be any noticable difference. Acme encoding dependency removed.
Bug fixes
  • Fix bug in cumulative histogram calculation.
  • Fix read failure for FITS files with non-blank TDIM for zero-length columns.
  • Fix bugs that led to timezone-dependent results when reading ISO-8601 or decimal year time columns.
  • Fix numeric field truncation bug in LaTeX table output.
  • Fix some parameter handling errors in coneskymatch.
  • Fix NullPointerException bug for disjoint regions in some cases in tmatchn.
Version 3.0-8 (13 September 2016)
Version 3.0-9 (23 September 2016)
Version 3.1 (8 March 2017)
Plotting improvements
  • Improved documentation of plot layer types and shading modes in the user document - each option now has an example graphic and the text of the command that generates it.
  • New layer type grid, to plot a 2-d weighted histogram. This replaces the density layer, which has been withdrawn (grid can do all the same things and more, except specify bin size in screen pixels).
  • New plot layer type quantile, which can (e.g.) plot median lines through noisy data.
  • New plot layer type gaussian, for Gaussian fits to histograms.
  • Histogram-like layer types are now available from the plot2time command as well as from plot2plane, so you can now plot histograms with a temporal horizontal axis.
  • New normalisation (scaling) option unit for histogram, KDE, and KNN plots.
  • New normalisation (scaling) options per_day etc for histogram, KDE and KNN layers when used from the plot2time command.
  • Colour names recognised by the plot2* command *color parameters now include the (140) CSS-like colours alongside the dozen standard plotting colours.
  • Plot commands insets parameter now lets you specify margins round plots using any combination of <top>,<left>,<bottom>,<right>, rather than requiring all values or none.
  • Add new parameter auxwidth to plot2 commands, to control colour ramp lateral dimension in pixels.
  • Modify plan caching arrangements for STILTS plots. This results in much better interactive performance (navigation etc) for some plot layer types plotted to the screen. There are also some minor API changes to a few of the plot2 utility classes, which now provide more flexibility for caching.
  • Various tweaks to the details of how plots are positioned on the screen or in output graphics files.
Miscellaneous enhancements
  • Fixed the match score (distance measure) for combined matchers. Previously, the score was a linear sum of the unscaled distances for the constituent matchers, which meant a Best match was pretty meaningless. Now, it adds scaled distances in quadrature, so Best matching should give you a somewhat sensible result.
  • The skyerr and nd_err matchers now report output separations as scaled (dimensionless) values rather than in physical units; this means the results are more comparable, so Best matches will make more sense.
  • taplint can now optionally write its output in JSON format (see format parameter).
  • Update taplint OBS stage for PR-ObsCore-v1.1-20160923.
  • The Maths function hypot now takes an arbitrary number of arguments (instead of exactly two).
  • Add blockmaxrec parameter to tapskymatch, and improve warning text in case of result truncations.
  • Improve tapquery/tapresume behaviour when monitoring async jobs; if the service goes down, the application will wait for it to come back rather than bailing out.
  • Options for MOC output format (mocfmt parameter in pixfoot command) have changed; option ascii is replaced by json. The old ASCII format was slightly broken JSON in any case.
  • MOC library upgraded to v4.6 (from v3.3). Improved MOC output.
  • Update JCDF library to v1.2-2 (2017-01-01 leap second).
Bug fixes
  • Fix KDE/KNN plotting bug that could get scaling badly wrong for normalised cumulative plots.
  • Fix some regression bugs relating to plot2 command insets parameter, present since v3.0-6.
  • Remove spurious padding from EPS graphics output.
  • Fix small bug in plot2plane/plot2time label painting; horizontal axis label was sometimes off the bottom of the plot by a few pixels.
  • Fix subpixel offset of colour ramp frame in PDF/PostScript graphics output.


Next Previous Up Contents
Up: Release Notes
Previous: Acknowledgements

STILTS - Starlink Tables Infrastructure Library Tool Set
Starlink User Note256
STILTS web page: http://www.starlink.ac.uk/stilts/
Author email: m.b.taylor@bristol.ac.uk
Mailing list: topcat-user@jiscmail.ac.uk