Next Previous Up Contents
Up: Release Notes
Previous: Package Dependencies


C.3 Version History

Version 1.0 (30 Jan 2004)
Initial public release.
Version 1.0-2 (11 Feb 2004)
Version 1.0-3 (12 Feb 2004)
Version 1.0-4 (17 Mar 2004)
Version 1.1 (29 Mar 2004)
Version 1.1-1 (11 May 2004)
Version 2.0 (20 October 2004)
Version 2.0 is a major revision incorporating some non-backwardly-compatible changes to the public API. The main differences are as follows.
RowSequence interface modified
The RowSequence interface has been modified; a new close method has been introduced, and the old advance() and getRowIndex() methods have been withdrawn (these latter were not very useful and in some cases problematic to implement).
Setter methods added to StarTable interface
The methods setName() and setURL() have been added to the StarTable interface.
Pluggable storage policies
The StoragePolicy class was introduced, which allows you to influence whether cached table data are stored in memory or on disk. This has led to backwardly-incompatible changes to public interfaces and classes: makeStarTable now takes a new StoragePolicy argument, and VOElementFactory's methods are now instance methods rather than static ones.
Input table format now either specified explicitly or detected automatically
The StarTableFactory class's makeStarTable methods now come in two flavours - with and without a format name. This corresponds to two table reading modes: named format mode and automatic format detection mode. In named format mode you specify the format of the table you are trying to read and in automatic format detection mode you rely on the factory to work it out using magic numbers. Although automatic detection works well for VOTable and FITS, it's poor for text-based formats like ASCII and CSV. This has resulted in addition of some new two-argument makeStarTable methods and the withdrawal of the getBuilders method in favour of two new methods getDefaultBuilders and getKnownBuilders (similarly for setter methods) which deal with the handlers used in automatic detection mode and the ones available for named format mode respectively. Note that the ASCII table format is not automatically detected, so to use ASCII tables you now have to specify the format explicitly.
VOTable parsing overhauled
The VOElement class has been rewritten and now implements the DOM Element interface. This means that the hierarchical structure which you can navigate to obtain information about the VOTable document and extract table data actually is a DOM rather than just being sat on top of one. You can therefore now use it just as a normal DOM tree (making use of the methods defined in the org.w3c.dom interface, interoperating with third-party components which require a DOM). This has had a number of additional benefits and consequences:
  • VOTable handling now fully meets version 1.1 of the VOTable standard. This includes full ID/ref crossreferencing (e.g. a TABLE element obtaining its structure by reference to a previously defined one) which was absent in previous versions.
  • VOTable processing is now independent of Java version; in previous versions it failed on J2SE1.5/5.0 due to absence of some Crimson parser classes.
  • The VOElementFactory class now has instance methods rather than static methods.
  • By installing a StoragePolicy.DISCARD into a VOElementFactory it is now possible to obtain a data-less (structure only, hence minimal resource) VOTable DOM.
TableSink interface modified
Some TableSink methods now throw exceptions.
Comma-Separated Value format supported
There are now CSV input and output handlers. The input handler is not by default installed in the StarTableFactory's list for automatic format detection, but CSV-format tables can be loaded using named format mode. The format is intended to match the (widely-used) variety used by Microsoft Excel amongst others (with optional column names).
New 'FITS-plus' format introduced
Handlers are introduced for a variant of FITS called 'FITS-plus'. This is a FITS file with a BINTABLE extension in HDU#1 as usual, but with the VOTable text containing its metadata stored in a byte array in the primary HDU. This means that the rich VOTable metadata are available when reading it with a matching input handler, but it looks like a perfectly normal FITS table without the metadata when read by a normal FITS-aware application. This is now the format in which FITS tables are written by default (unless you choose the format name "basic-fits").
ASCII-format input handler improvements
  • Now runs in limited memory, but requires two passes of stream (data caching as per current StoragePolicy).
  • Now uses Short/Float types in preference to Integer/Double if the input data make this appropriate.
  • Now preserves negative zero values (often important for sexagesimal representations).
  • Now understands d or D as an exponent letter as well as e or E.
  • A '!' character in column 1 is now understood to introduce a comment line.
Table matching
There have been several changes including performance enhancements and improved functionality in the table matching classes in the package uk.ac.starlink.table.join.. These work and have full javadocs, but they are still experimental, subject to substantial change in future releases, and not documented properly in this document.
Null handling improvements
There is now a mechanism for flagging the magic value you would like to use when encoding nulls in an integer output column (NULL_VALUE_INFO) Nulls in FITS and VOTable/FITS tables are now preserved correctly on output.
Miscellaneous
There have been a number of miscellaneous improvements and bugfixes in various parts of the library, including the following:
  • FITS files now store column descriptions in TCOMMx headers.
  • A type-translation bug in the JDBC handler has been fixed, so that it now works with PostgreSQL (and possibly other JDBC implementations).
  • New class EmptyStarTable added.
Version 2.0-1 (October 2004)
Version 2.0-2
Version 2.0-3
Version 2.1 (4 February 2005)
Some of the public interfaces have been modified in backwardly incompatible ways at this release. However, it is not expected that much user code will need to be changed.
RequireRandom flag in StarTableFactory
The wantRandom flag has been changed in name and semantics to requireRandom in StarTableFactory. When set, any table returned from the factory is now guaranteed to have random access.
Table output to streams
StarTableOutput now has a new method writeStarTable which writes a table to an OutputStream as well as the one which writes to a location string (usually filename). This is supported by changes to the writeStarTable methods which StarTableWriter implementations must provide.
Table load dialogue
The uk.ac.starlink.table.gui.StarTableChooser table loader dialogue has been improved in several ways. Loading is now done asynchronously, so that the GUI does not lock up when a long load is taking place (a load cancel button can be pressed). Additionally, custom load dialogues have been made pluggable, so that you can add new load sub-dialogues by implementing TableLoadDialog (most likely subclassing BasicTableLoadDialog) and naming the class in the startable.load.dialogs property. A dialogue for browsing AstroGrid's MySpace remote filestore is available, but for reasons of size STIL is not by default packaged with all the classes required to make it work (AXIS and the CDK are missing).
StarTable parameter method
A new utility method setParameter has been added to the StarTable interface.
BeanStarTable
A new StarTable implementation, BeanStarTable which can store Java Beans has been introduced. This is handy for storing arrays of objects of the same kind without having to write a custom table implementation.
Undeclared character arraysize workaround
A workaround has been introduced to cope with a common error in VOTable documents in which FIELD elements for string values lack the required arraysize attribute; by default it is now assumed to have the value "*" rather than "1" as the standard dictates. See Section 7.3.4.
Minor changes
  • LINK elements can now be added to FIELDs in a VOTable on output by adding a suitable URL-type metadatum to the corresponding ColumnInfo.
  • Temporary files are now deleted by finalizers (may lead to better reclamation of temporary file space during operation).
  • Fixed a bug in VOTable parsing when TD elements were empty.
  • V1.1 VOTable tables written now contain the declaration
      xsi:schemaLocation="http://www.ivoa.net/xml/VOTable/v1.1
                          http://www.ivoa.net/xml/VOTable/v1.1"
                
    instead of
      xsi:noNamespaceSchemaLocation="http://www.ivoa.net/xml/VOTable/v1.1"
                
    (thanks to Paul Harrison for suggesting this correction).
Version 2.2 (17 March 2005)
New tool:
tpipe command introduced
The tpipe command has been tentatively introduced at this release. This useful command-line tool is experimental and may undergo major changes or be moved to a separate package altogether in future releases.

There have been changes to some of the main interfaces:

RowSequence hasNext withdrawn
The hasNext() method has been withdrawn from the RowSequence interface and the next() method, which used to be declared void, now returns boolean indicating whether there is another row. This is quite likely to break existing code, but the fix is easy; simply replace:
              RowSequence rseq = table.getRowSequence();
              while ( rseq.hasNext() ) {
                  rseq.next();
                  ...
              }
          
with
              RowSequence rseq = table.getRowSequence();
              while ( rseq.next() ) {
                  ...
              }
          
TableBuilder streaming
A new method streamStarTable has been added to the TableBuilder interface to provide improved support for table streaming.
GUI table choosers changed
There have been several changes in the uk.ac.starlink.gui package. StarTableChooser and StarTableSaver have been replaced by TableLoadChooser and TableSaveChooser, and these both now use a graphical widget which can view files in remote filestores (such as MySpace and SRB) if the relevant classes are present.

Minor changes:

Version 2.3 (29 April 2005)
Version 2.3-1 (30 June 2005)
Version 2.3-2 (30 September 2005)
Version 2.4 (10 May 2006)
The following API change has taken place: Additionally, there are the following minor improvements and bugfixes:
Version 2.5 (7 July 2006)
Version 2.6 (3 August 2006)
Version 2.6-1 (Starlink Hokulei release)
Version 2.6-2 (23 July 2007)
Version 2.6-3 (4 Sep 2007)
Version 2.6-4 (30 Oct 2007)
Version 2.6-5 (6 Dec 2007)
Version 2.6-6 (28 Jan 2008)
Version 2.6-7 (4 Apr 2008)
Version 2.7 (19 Aug 2008)
Version 2.7-1 (27 Mar 2009)
Version 2.7-2 (17 July 2009)
Version 2.8 (2 Oct 2009)
Versions 2.9*x
STIL versions 2.9x, 2.9-1x, 2.9-2x, 2.9-3x did not get a public release, since the backwardly-incompatible changes they contained were not stable, but were present in some versions of TOPCAT and STILTS. Details of what changed in which of these versions are only available by examining relevant versions of the XML sources for SUN/252 (sun252.xml) in the version control system. All the changes are amalgamated into version 3.0.
Version 3.0 (23 December 2010)
This major release contains some new capabilities and some backward incompatibilities with respect to the previous public release, version 2.8. The major changes are in the following areas: Anyone implementing a table read handler, write handler, load dialogue or save dialogue will need to make some adjustments since the relevant interfaces have changed. Anyone using the GUI load dialogue classes in package uk.ac.starlink.table.gui (as far as I know, nobody apart from me is) will require significant rewriting. Other users of the library will probably find no or only minor issues if compiling against the new version. In most cases significant changes will be marked by compilation errors rather than silent changes in behaviour. The exception is use of the new Adaptive storage policy which is now the default; this change is expected to be beneficial rather than problematic in most cases.

If you experience any difficulties in upgrading from a previous version to this one, please contact the author, who will be happy to advise or assist.

The changes in more detail are as follows:

Multiple table read:
  • New MultiTableBuilder interface which TableBuilders may implement if they know how to load multiple tables from the same source. The FITS and VOTable handlers now implement this.
  • Three new makeStarTables methods added to StarTableFactory. These return a TableSequence which contains multiple tables. Multiple tables are only a possibility if the relevant handler implements the new MultiTableBuilder interface (of the supplied handlers, FITS and VOTable).
  • StarTableFactory methods makeStarTable(URL) and makeStarTable(URL,String) are deprecated. They are very thin utility wrappers around existing methods which may be replaced easily in calling code using the URLDataSource class. These methods may be removed in a future release.
Multiple table write:
  • New interface MultiStarTableWriter, which extends StarTableWriter, for output handlers which can write multiple tables to the same container. Corresponding methods added to StarTableOutput.
  • MultiStarTableWriter is implemented for most variants of the FITS and VOTable output handlers, to generate multi-extension FITS and multi-TABLE VOTable outputs respectively. Implementations for some other output handlers generating output that may not be machine-readable as a multi-table file are provided as well.
GUI load/save dialogues:
There have been substantial changes to the GUI load framework, mainly to support multiple table load and non-modal load dialogues. The main interface is still called TableLoadDialog, but its definition has changed considerably. See the javadocs for details.

The save dialogue framework has also undergone some incompatible changes to support writing of multiple files, though these are less dramatic. There are backwardly incompatible effects on the APIs of SaveWorker, TableSaveChooser and TableSaveDialog and its implementations.

Storage policies:
  • New StoragePolicy ADAPTIVE, which effectively uses memory for relatively small tables and scratch disk files for relatively large ones. The intention is that for most purposes this can be used without the user or the programmer having to guess whether small or large tables are likely to be in use. The implementation makes use in some circumstances of direct byte buffer allocation (=malloc()), which means that the size of the controlling java process can grow beyond the size of the maximum specified java heap. The Adaptive storage policy is the new default.
  • Added method toByteBuffers to ByteStore class.
  • Implementation changes in Disk storage policy to reduce memory footprint.
Other:
  • Library now distributed as zip of jars (stil_jars.zip) as an alternative to monolithic jar file (stil.jar). This may be more appropriate for those using the library in a framework that contains other third party class libraries.
  • VOTableBuilder.makeStarTable now works in a streaming fashion. This should be much faster in the case of a VOTable document containing many TABLE elements. There is a possibility that behaviour will change slightly (some post-positioned INFO/PARAM elements may not get picked up, tables may be successfully loaded from invalid XML documents) - I don't believe these are likely to cause trouble, but please alert me if you disagree.
  • Updated VOTable 1.2 schema to final version (elementFormDefault="qualified").
  • New attribute Utype for ValueInfos. ValueInfo has new method getUtype, DefaultValueInfo has new method setUtype, and Tables.get/setUtype is deprecated.
  • FITS files now store table names in EXTNAME (and possibly EXTVAR) header cards.
  • Added configurable JDBC type conversion framework for reading results from SQL queries. By default JDBC results with Date-type contents will now be turned into String values, but this can be configured by supplying a custom TypeMapper. Previously they were left as Date-type objects, which meant that without special attention they could not be written by general-purpose table output handlers.
  • Better behaviour (warn + failover) when attempting to read large files on 32-bit OS or JVM.
  • VOTable PARAM output now works out nullability and unspecified array and element size values from the data rather than leaving them as unspecified.
  • The wantRandom parameter of TableBuilder.makeStarTable has been deprecated in the documentation.
  • Fixed an obscure bug which could under rare circumstances cause truncation of strings with leading/trailing whitespace read from text-format files.
  • Fixed bug in TST table output.
  • Fixed bug in CSV file parsing that could ignore header row in absence of non-numeric columns.
  • Fixed minor bug in CSV file parsing which ignored first row in no-header CSV file when calculating column Element Size values.

Version 3.0-1 (9 May 2011)
Version 3.0-2 (30 June 2011)
Version 3.0-3 (27 October 2011)
Version 3.0-4 (17 December 2012)
Version 3.0-5 (1 July 2013)
Version 3.0-6 (4 July 2014)
Version 3.0-7 (3 October 2014)
Version 3.0-8 (13 November 2014)
Version 3.0-9 (6 Feb 2015)
Version 3.0-10 (26 Feb 2015)
Version 3.0-11 (14 April 2015)
Version 3.0-12 (10 Jul 2015)
Version 3.0-13 (17 Aug 2015)
Version 3.0-14 (22 Oct 2015)
Version 3.1 (27 Nov 2015)
Version 3.1-1 (10 Jun 2016)
Version 3.1-2 (13 Sep 2016)
Version 3.2 (8 Mar 2017)
Version 3.2-1 (29 Sep 2017)
Version 3.2-2 (7 Nov 2017)
Version 3.2-3 (13 Nov 2017)
Version 3.3 (24 Apr 2018)
Version 3.3-1 (18 May 2018)
Version 3.3-2 (2 Nov 2018)
Version 3.3-3 (9 May 2019)
Version 3.4 (18 November 2019)
Version 3.4-1 (5 June 2020)
Version 4.0 (11 January 2021)
This major release provides support for parallel processing of table data, enhances table I/O specification options, addresses some long-standing issues that require backwardly incompatible API changes, and breaks out crossmatching classes into a separate package. The documentation has been extended accordingly. Library users using existing STIL functionality for table I/O should not need to make too many code changes when upgrading from STIL v3, but those providing StarTable or I/O handler implementations may need to modify their implementations in accordance with the API changes; see below for details.
Notable new functionality
  • Support for parallel table data processing added, see Section 2.3.3.
  • Improved patterns for threadsafe random access using StarTable.getRowAccess method, see Section 2.3.2.
  • I/O handler configuration options may be configured by user-supplied string, see Section 3.5. Config options are provided for several handler implementations; more may be forthcoming in future releases.
  • Table Schemes introduced, which can be used to load tables not serialized as byte streams, see Section 3.9.
  • Auto file format detection now examines filenames to help guess input file format.
  • StarTable.close() can now be called to release global resources such as file descriptors and (where unmapping is supported) mapped files.
API changes
  • New data access methods added to StarTable interface to support multithreaded processing: getRowAccess, getRowSplittable.
  • New default methods added to StarTable interface for convenience in parameter handling: getParameterByName and setParameter.
  • StarTable has a new close method that should relinquish any non-heap-based resources (file descriptors, mapped files). In most cases this can be implemented as a no-op.
  • RowSequence and StarTable now implement java.io.Closeable so they can be used in try-with-resources statements.
  • New method looksLikeFile added to TableBuilder interface, to enable filename-based (file extension-based) input format guessing.
  • Aux data access methods getAuxData, getAuxDatumByName, setAuxDatum are now on the ValueInfo interface, rather than on (ValueInfo's subtype) the ColumnInfo class. This means that table parameters, as well as table columns, can now sport auxiliary metadata. That should really always have been the case.
  • Add ValueInfo.getXtype method.
  • Clarified requirements of class RandomStarTable; implemented data access methods must be thread-safe.
  • RowData introduced as super-interface of RowSequence.
  • Class RandomWrapperStarTable has been withdrawn; the implementation was complicated and hard to upgrade, and it was probably(?) never used.
  • Some other minor API changes.
Package contents
  • The classes in the namespace uk.ac.starlink.table.join (table joins and crossmatches) are no longer part of the STIL library. Those classes are still available elsewhere, but the functionality is distinct from the I/O that STIL mostly provides, there was never corresponding tutorial text in the user document, they required external dependencies that sometimes complicated STIL deployment, and they are not needed by most STIL users. The dependencies cdshealpix.jar and htmIndex.jar are no longer required.
Documentation
  • Descriptions of input handlers (Section 3.6) and output handlers (Section 3.7) updated.
  • Add new documentation section Section 3.8.1 describing the the FITS-plus convention.
  • Add Documented interface to improve auto-documentation of I/O handlers. Handler documentation now resides in the code itself, and is extracted programmatically to generate entries in user documentation.
Miscellaneous enhancements and bugfixes
  • COOSYS and TIMESYS attributes are now preserved during VOTable I/O for table PARAMs (as well as for FIELDs, which was already the case).
  • Add miscellaneous utility methods in Tables class etc.
  • Fix ECSV output bug: encoding was incorrect for metadata scalars with certain non-alphanumeric first characters, leading to invalid YAML.
  • Remove some unhelpful per-column metadata items from ECSV output.
  • Prevented HealpixSkyPixellator.calculateDefaultK from returning -1 for large angles.
  • Improve exception handling in StreamTableSink/OnceRowPipe implementations, make interruption work better.
  • Fix bug that could give unhelpful table load error message for very short non-FITS files in auto-detection mode.
Version 4.0-1 (16 April 2021)
Version 4.0-2 (10 June 2021)
New functionality:
  • Apache Parquet format is now supported for input and output (note not all distributions include Parquet-MR support libraries).
  • AAS Machine-Readable Table (MRT) format is now supported for input.
  • ECSV format input and output handlers are upgraded to version 1.0 of the ECSV format, meaning they can now read and write array-valued columns.
  • AbstractFitsTableWriter and subclasses now have configuration methods setAllowSignedByte, setAllowZeroLengthString, setWide and setPadCharacter. These should be used in preference to the various custom constructors, which have now been deprecated.
  • Add new table scheme test (TestTableScheme).
Minor enhancements and behaviour changes:
  • Space-delimited ECSV files now write empty fields quoted.
  • Unknown or unsupported column datatype values in ECSV files are now treated like string rather than causing table read failure.
  • Empty strings in FITS 1-character columns are now returned as blank values rather than ASCII NUL ('\0').
  • Undersized, including zero-length, strings written to FITS columns are now by default terminated with an ASCII NUL rather than in some cases padded with spaces.
Bugfixes:
  • Fix regression bug since v4.0 that meant table drag'n'drop wasn't working.
  • Fix regression bug since v4.0 that meant jdbc: URLs didn't work.
  • Fix long-standing logic error in ASCII/CSV input handler that could misidentify column types and cause read failures.
  • FITS TZERO headers are now written correctly with numeric values rather than string values.
Version 4.0-3 (2 July 2021)
Version 4.0-4 (15 Oct 2021)


Next Previous Up Contents
Up: Release Notes
Previous: Package Dependencies

STIL - Starlink Tables Infrastructure Library
Starlink User Note252
STIL web page: http://www.starlink.ac.uk/stil/
Author email: m.b.taylor@bristol.ac.uk