java.io.tmpdir is a standard Java system property which is used by the disk-based storage policies. It determines where the JVM writes temporary files, including those written by these storage policies (see and ). The default value is typically "/tmp" on Unix-like platforms.

java.util.concurrent.ForkJoinPool.common.parallelism is a standard Java system property which controls the number of processing cores apparently available from the system. By default it is typically set to one less than the number of cores on the current machine. To inhibit parallelisation you can set this to 1.

The jdbc.drivers property is a standard JDBC property which names JDBC driver classes that can be used to talk to SQL databases. See for more details.

The mark.workaround determines whether a workaround is employed to fix bugs in which certain InputStream implementations lie about their abiilty to do mark/reset operations ( returns true when it should return false). Several classes in various versions of Sun's J2SE do this. It can result in an error with a message like "Resetting to invalid mark". Setting this property true works around it. By default it is set false.

The star.connectors property names additional remote filestore implementations. Its value is a colon-separated list of class names, where each element of the list must be the name of a class on the classpath which implements the interface. It is used in the graphical filestore browsers in and . See for more details.

The startable.readers property provides additional input handlers which can use for loading tables in named format mode. Its value is a colon-separated list of class names, where each element of the list must be the name of a class on the classpath which implements the interface and has a no-arg constructor. When a new is constructed, an instance of each such named class is created and added to its known handler list. Users of the library can therefore read tables in the format that the new handler understands by giving its format name when doing the load.

The startable.schemes property Can be set to a (colon-separated) list of custom table scheme handler classes. Each class must implement the interface, and must have a no-arg constructor. The schemes thus named will be available alongside the standard ones listed in .

The startable.storage property sets the initial value of the default storage policy, which influences where bulk table data will be cached. The recognised values are:

• memory: table data will normally be stored in memory ()
• disk: table data will normally be stored in temporary disk files ()
• adaptive: table data will be stored in memory for small tables and on disk for larger ones ()
• sideways: table data will normally be stored in temporary disk files using a column-oriented arrangement ()
• discard: table data will normally be thrown away, leaving only metadata ()

You may also give the name of a subclass of which has a no-arg constructor, in which case an instance of this class will be used as the default policy. See for further discussion.

See for further discussion of storage policies.

The startable.unmap property controls how memory-mapped buffers (s) that have been allocated by STIL are unmapped when it can be determined that they will no longer be used. Specifically, it controls the implementation of the class that will be used. See the implementation and comments in that class for further discussion. This is currently only used for FITS input, but it may be extended for other purposes in future versions.

Possible values are:

• sun: Best efforts unmapping based on available sun.misc classes, discovered by reflection.
• cleaner: Buffers are unmapped using non-J2SE classes including sun.misc.Cleaner, discovered by reflection. Expected to work for java6 through java8.
• unsafe: Buffers are unmapped using non-J2SE classes including sun.misc.Unsafe, discovered by reflection. Expected to work for java9 and possibly later versions.
• none: No attempt is made to unmap buffers explicitly. They will be unmapped only when garbage collected.

In general you are advised to leave this parameter alone. It is provided because the sun-like unmapping is doing fundamentally inadvisable things, and although I think it's done in a way which will not cause problems, nasty consequences like JVM crashes are possible if I've made mistakes, so it's a good idea to have a switch here that allows the dangerous behaviour to be switched off (startable.unmap=none).

The startable.writers property provides additional output handlers which can use for writing tables. Its value is a colon-separated list of class names, where each element of the list must be the name of a class on the classpath which implements the interface and has a no-arg constructor. When a new is constructed, an instance of each such named class is created and added to its handler list. Users of the library can therefore write tables in the format that the new handler knows how to write to by giving its format name when performing the write.

The votable.namespacing property determines how XML namespacing is handled in VOTable documents. It may take one of the following fixed values:

• none: No namespace handling is done. If the VOTable document contains xmlns declarations, the parser will probably become confused. ()
• lax: Anything that looks like it is probably a VOTable element is treated as a VOTable element, regardless of whether the namespacing has been declared correctly or not. ()
• strict: Only elements declared to be in one of the official VOTable namespaces are treated as VOTable elements. If the VOTable document does not contain appropriate xmlns declarations, the parser may not treat it as a VOTable. ()

The votable.strict property determines whether VOTable parsing is done strictly according to the letter of the standard. See for details.

The votable.version property selects the version of the VOTable standard which output VOTables will conform to by default. May take the values "1.0", "1.1", "1.2" "1.3" or. "1.4". By default, version 1.3 VOTables are written.

My thanks are due to a number of people who have contributed help to me in writing this document and the STIL software, including:

• Alasdair Allan (Starlink, Exeter)
• Malcolm Currie (Starlink, RAL)
• Clive Davenhall (AstroGrid, RoE)
• Pierre Didelon (CEA)
• Peter Draper (Starlink, Durham)
• David Giaretta (Starlink, RAL)
• Paul Harrison (ESO)
• Jonathan Irwin (IoA)
• Nickolai Kouropatkine (Fermilab)
• Clive Page (AstroGrid, Leicester)
• Chris Stoughton (Fermilab)

STIL is written in Java and contains code from the following non-Starlink libraries:

• Ant's Bzip2 compression/decompression code
• JCDF is used for reading CDF files
• JARROW is used for reading Feather files
• Snakeyaml is used for YAML parsing in the ECSV input handler
• JSON-java is used for JSON I/O in Feather and ECSV handlers

STIL is currently available in several forms; you may have the stil.jar file which contains most of the important classes, or a full starjava installation, or a standalone TOPCAT jar file, or the stil_jars.zip file containing various packages in separate jar files, or in some other form. None of these is definitive; different packages are required for different usages. If you are keen to prepare a small class library you can identify functionality you are not going to need and prepare a class library omitting those classes. In most cases, STIL classes will cope with absence of such packages without falling over.

The following is a list of what packages are required for what functions:

uk.ac.starlink.table

uk.ac.starlink.table.formats

uk.ac.starlink.table.jdbc

uk.ac.starlink.table.storage

uk.ac.starlink.table.text

uk.ac.starlink.util

Core table processing.

uk.ac.starlink.fits

FITS table processing.

uk.ac.starlink.votable

uk.ac.starlink.votable.dom

VOTable processing.

uk.ac.starlink.votable.soap

StarTable <-> VOTable serialization/deserialization for use with SOAP RPC methods. Moribund?

uk.ac.starlink.cdf

uk.ac.bristol.star.cdf

CDF table processing.

uk.ac.starlink.feather

uk.ac.bristol.star.feather

uk.ac.bristol.star.fbs.*

Feather table processing.

uk.ac.starlink.ecsv

org.yaml.snakeyaml.*

ECSV table processing

uk.ac.starlink.pds4

gov.nasa.pds.*

PDS4 table processing

uk.ac.starlink.mirage

Mirage-format table output

org.json

JSON library used as part of Feather support.

uk.ac.starlink.table.gui

Graphical components for tables, mainly load/save dialogues.

uk.ac.starlink.connect

org.apache.axis.*

Generic code for browsing remote filespaces in load/save dialogues.

uk.ac.starlink.astrogrid

org.astrogrid.*

Browsing MySpace remote filesystems in load/save dialogues.

uk.ac.starlink.srb

edu.sdsc.grid.*

Browsing SRB remote filesystems in load/save dialogues.

Version 1.0 (30 Jan 2004)

Initial public release.

Version 1.0-2 (11 Feb 2004)

• Added .

Version 1.0-3 (12 Feb 2004)

• Considerably improved performance of inline (base64-encoded) BINARY/FITS table parsing.

Version 1.0-4 (17 Mar 2004)

• VOTable-derived StarTables now pick up parameters from INFO elements as well as PARAM elements.
• Text format output handler now by default outputs table parameters as well as the table data and column metadata.

Version 1.1 (29 Mar 2004)

• New ASCII format output handler can write tables in the same text-based format used by the ASCII input handler.
• can now deduplicate column names.
• New class permits adding the rows of one table after the rows of another.

Version 1.1-1 (11 May 2004)

• Improved PostgreSQL compatibility

Version 2.0 (20 October 2004)

Version 2.0 is a major revision incorporating some non-backwardly-compatible changes to the public API. The main differences are as follows.

RowSequence interface modified

The interface has been modified; a new method has been introduced, and the old advance() and getRowIndex() methods have been withdrawn (these latter were not very useful and in some cases problematic to implement).

Setter methods added to StarTable interface

The methods setName() and setURL() have been added to the interface.

Pluggable storage policies

The class was introduced, which allows you to influence whether cached table data are stored in memory or on disk. This has led to backwardly-incompatible changes to public interfaces and classes: now takes a new StoragePolicy argument, and 's methods are now instance methods rather than static ones.

Input table format now either specified explicitly or detected automatically

The class's makeStarTable methods now come in two flavours - with and without a format name. This corresponds to two table reading modes: named format mode and automatic format detection mode. In named format mode you specify the format of the table you are trying to read and in automatic format detection mode you rely on the factory to work it out using magic numbers. Although automatic detection works well for VOTable and FITS, it's poor for text-based formats like ASCII and CSV. This has resulted in addition of some new two-argument makeStarTable methods and the withdrawal of the getBuilders method in favour of two new methods and (similarly for setter methods) which deal with the handlers used in automatic detection mode and the ones available for named format mode respectively. Note that the ASCII table format is not automatically detected, so to use ASCII tables you now have to specify the format explicitly.

VOTable parsing overhauled

The class has been rewritten and now implements the DOM interface. This means that the hierarchical structure which you can navigate to obtain information about the VOTable document and extract table data actually is a DOM rather than just being sat on top of one. You can therefore now use it just as a normal DOM tree (making use of the methods defined in the interface, interoperating with third-party components which require a DOM). This has had a number of additional benefits and consequences:

• VOTable handling now fully meets version 1.1 of the VOTable standard. This includes full ID/ref crossreferencing (e.g. a TABLE element obtaining its structure by reference to a previously defined one) which was absent in previous versions.
• VOTable processing is now independent of Java version; in previous versions it failed on J2SE1.5/5.0 due to absence of some Crimson parser classes.
• The class now has instance methods rather than static methods.
• By installing a into a it is now possible to obtain a data-less (structure only, hence minimal resource) VOTable DOM.

TableSink interface modified

Some methods now throw exceptions.

Comma-Separated Value format supported

There are now CSV input and output handlers. The input handler is not by default installed in the StarTableFactory's list for automatic format detection, but CSV-format tables can be loaded using named format mode. The format is intended to match the (widely-used) variety used by Microsoft Excel amongst others (with optional column names).

New 'FITS-plus' format introduced

Handlers are introduced for a variant of FITS called 'FITS-plus'. This is a FITS file with a BINTABLE extension in HDU#1 as usual, but with the VOTable text containing its metadata stored in a byte array in the primary HDU. This means that the rich VOTable metadata are available when reading it with a matching input handler, but it looks like a perfectly normal FITS table without the metadata when read by a normal FITS-aware application. This is now the format in which FITS tables are written by default (unless you choose the format name "basic-fits").

ASCII-format input handler improvements

• Now runs in limited memory, but requires two passes of stream (data caching as per current ).
• Now uses Short/Float types in preference to Integer/Double if the input data make this appropriate.
• Now preserves negative zero values (often important for sexagesimal representations).
• Now understands d or D as an exponent letter as well as e or E.
• A '!' character in column 1 is now understood to introduce a comment line.

Table matching

There have been several changes including performance enhancements and improved functionality in the table matching classes in the package uk.ac.starlink.table.join.. These work and have full javadocs, but they are still experimental, subject to substantial change in future releases, and not documented properly in this document.

Null handling improvements

There is now a mechanism for flagging the magic value you would like to use when encoding nulls in an integer output column () Nulls in FITS and VOTable/FITS tables are now preserved correctly on output.

Miscellaneous

There have been a number of miscellaneous improvements and bugfixes in various parts of the library, including the following:

• FITS files now store column descriptions in TCOMMx headers.
• A type-translation bug in the JDBC handler has been fixed, so that it now works with PostgreSQL (and possibly other JDBC implementations).
• New class added.

Version 2.0-1 (October 2004)

• Fixed bugs related to reading streamed (rather than mapped) FITS tables
• Fixed a bug in VOTable 1.1 schema namespace declaration on output

Version 2.0-2

• Better documentation () and facilities for manipulation of VOTable FIELD attributes from StarTable object

Version 2.0-3

• Fixed two more bugs in VOTable 1.1 namespace declaration on output; output elements were being declared in the unnamed namespace rather than the VOTable 1.1 one, and the VOTable schema location was wrong. Both of these errors arose from the fact that the example VOTable in the recommendation document was declared in a wrong/misleading fashion.
• Added architecture cartoon to SUN/252.

Version 2.1 (4 February 2005)

Some of the public interfaces have been modified in backwardly incompatible ways at this release. However, it is not expected that much user code will need to be changed.

RequireRandom flag in StarTableFactory

The wantRandom flag has been changed in name and semantics to requireRandom in . When set, any table returned from the factory is now guaranteed to have random access.

Table output to streams

StarTableOutput now has a new method which writes a table to an OutputStream as well as the one which writes to a location string (usually filename). This is supported by changes to the writeStarTable methods which implementations must provide.

Table load dialogue

The uk.ac.starlink.table.gui.StarTableChooser table loader dialogue has been improved in several ways. Loading is now done asynchronously, so that the GUI does not lock up when a long load is taking place (a load cancel button can be pressed). Additionally, custom load dialogues have been made pluggable, so that you can add new load sub-dialogues by implementing (most likely subclassing BasicTableLoadDialog) and naming the class in the startable.load.dialogs property. A dialogue for browsing AstroGrid's MySpace remote filestore is available, but for reasons of size STIL is not by default packaged with all the classes required to make it work (AXIS and the CDK are missing).

StarTable parameter method

A new utility method has been added to the StarTable interface.

BeanStarTable

A new StarTable implementation, which can store Java Beans has been introduced. This is handy for storing arrays of objects of the same kind without having to write a custom table implementation.

Undeclared character arraysize workaround

A workaround has been introduced to cope with a common error in VOTable documents in which FIELD elements for string values lack the required arraysize attribute; by default it is now assumed to have the value "*" rather than "1" as the standard dictates. See .

Minor changes

• LINK elements can now be added to FIELDs in a VOTable on output by adding a suitable URL-type metadatum to the corresponding ColumnInfo.
• Temporary files are now deleted by finalizers (may lead to better reclamation of temporary file space during operation).
• Fixed a bug in VOTable parsing when TD elements were empty.
• V1.1 VOTable tables written now contain the declaration xsi:schemaLocation="http://www.ivoa.net/xml/VOTable/v1.1 http://www.ivoa.net/xml/VOTable/v1.1" instead of xsi:noNamespaceSchemaLocation="http://www.ivoa.net/xml/VOTable/v1.1" (thanks to Paul Harrison for suggesting this correction).

Version 2.2 (17 March 2005)

New tool:

tpipe command introduced

The tpipe command has been tentatively introduced at this release. This useful command-line tool is experimental and may undergo major changes or be moved to a separate package altogether in future releases.

There have been changes to some of the main interfaces:

RowSequence hasNext withdrawn

The hasNext() method has been withdrawn from the interface and the next() method, which used to be declared void, now returns boolean indicating whether there is another row. This is quite likely to break existing code, but the fix is easy; simply replace: RowSequence rseq = table.getRowSequence(); while ( rseq.hasNext() ) { rseq.next(); ... } with RowSequence rseq = table.getRowSequence(); while ( rseq.next() ) { ... }

TableBuilder streaming

A new method has been added to the TableBuilder interface to provide improved support for table streaming.

GUI table choosers changed

There have been several changes in the uk.ac.starlink.gui package. StarTableChooser and StarTableSaver have been replaced by TableLoadChooser and TableSaveChooser, and these both now use a graphical widget which can view files in remote filestores (such as MySpace and SRB) if the relevant classes are present.

Minor changes:

• Added Tables.sortTable method.
• Added ExplodedStarTable.
• Added ConcatStarTable.
• VOTables now write arraysize="1" explicitly for scalar character fields.
• VOTable BINARY input handler refuses to attempt reading assumed-size character fields.
• Several bugfixes in JDBC output handler for writing new SQL tables; now writes String (VARCHAR) fields, better NULL value handling, avoids some SQL reserved words for column names.
• Better NULL value handling for some text-like output formats.

Version 2.3 (29 April 2005)

• New streaming convenience method introduced on StarTableFactory.
• New Axis-based StarTable<->VOTable serializer/deserializer classes in uk.ac.starlink.votable.soap package.
• New TableContentHandler class provides table-aware SAX processing of VOTable document streams.
• Improved documentation of storage policies in SUN/252.
• Missing arraysize attribute for character fields is now interpreted by default according to the VOTable standard rather than by default being worked around - i.e. an unspecified votable.strict system property now counts as true rather than false. This is the reverse of the behaviours in versions 2.1 and 2.2. See .
• Now overwrites existing tables when attempting to write tables to SQL database if a table of the same name already exists.
• Fixed PostgreSQL bug - can now write String columns correctly.
• tablecopy command deprecated and tpipe withdrawn - these are now available within the new package STILTS.

Version 2.3-1 (30 June 2005)

• Added convenience methods writeInlineTableElement, writeHrefTableElement to VOSerializer.
• Fixed a bug in VOStarTable parameter setting.
• Fixed a bug in ConcatStarTable which was leading to ClassCastExceptions when used sequentially.

Version 2.3-2 (30 September 2005)

• Some changes to RowMatcher class.
• Fixed some bugs in the VOTable DOM implementation connected with null values.
• MatchEngine now returns metadata on match scores.
• The string "null" (unquoted) in ASCII input handler is interpreted as a blank entry.
• Fixed bug in ASCII input handler which misidentified blank lines, or DOS-format line ends, as end of file.

Version 2.4 (10 May 2006)

The following API change has taken place:

• New method StarTableWriter.getMimeType has been introduced.

Additionally, there are the following minor improvements and bugfixes:

• Now copes with 'K'-format FITS binary table columns (64-bit integers).
• Added IPAC Table Format input handler.
• Added noheader option to CSV output format.
• Added mark.workaround system property.
• Blank values in boolean columns are now handled as null rather than false (changes to FITS handlers, VOTable handlers and cell renderer).
• Fixed bug which was writing some integer null values as empty TD elements (illegal) - now uses magic bad value where available.
• CSV & ASCII input handlers now (try to) detect sexagesimal and ISO-8601 format data columns and mark the unit string appropriately.
• Fixed bug writing unclosed LINK elements in output VOTables.

Version 2.5 (7 July 2006)

• Support for new column-oriented FITS file format (, outColfits).
• New StoragePolicy SIDEWAYS storage ().
• FITS-PLUS files now only recognised if VOTMETA header card has the value "T", not just if it is present.
• Increased the maximum field width written by text and (especially) ascii output handlers.
• TUCDnn header cards now used in FITS files to transmit UCDs (non-standard mechanism).

Version 2.6 (3 August 2006)

• Replaced RowStepper class by RowSequence in votable package. As well as being a bit tidier, this improves efficiency considerably for column-oriented access in some cases (esp. fits-plus/colfits-plus).
• Dramatically improved efficiency of fits-plus & (especially) colfits-plus format access in some situations (related to above point).
• colfits-basic format is now auto-detected.
• Added TST (tab-separated table) input and output handlers.
• Efficiency improvements for column-oriented access.

Version 2.6-1 (Starlink Hokulei release)

• Modified and extended API for more flexible use with creating tables in RDBMS.
• Modified presentation of HTML version of SUN/252 using CSS.
• Fixed bug in handling of single quotes in FITS file metadata.

Version 2.6-2 (23 July 2007)

• Add new exception .
• Add new classes and which are StarTable implementations built directly on JDBC objects.
• interface and implementation changed. Now better at deduplicating the names of joined tables.
• Fix error in output of FITS table TNULLn header cards - write them as numeric not string values.
• Improve error message for broken CSV files.

Version 2.6-3 (4 Sep 2007)

• Added and utility methods.
• Added interface and implementations and new createOutputSink methods in .
• FITS files now read/write Utypes using TUTYPnn header cards.

Version 2.6-4 (30 Oct 2007)

• Minor changes to interface and implementation of and .

Version 2.6-5 (6 Dec 2007)

• Improvements and modifications to crossmatching functionality in uk.ac.starlink.table.join, including multi-pair join.
• FITS reader now imports table HDU header cards as table parameters.
• Embedded spaces in output ASCII format table column names are now substituted with underscores.
• Added quoting of SQL identifiers for JDBC statement execution.

Version 2.6-6 (28 Jan 2008)

• Downgraded from WARNING to INFO log messages about the (extremely common) VOTable syntax error of omitting a FIELD/PARAM element's datatype attribute.
• Avoid some truncations of double (and float?) fields in text-mode output (may result in longer fields too).

Version 2.6-7 (4 Apr 2008)

• Some missing classes reinstated in the stil.jar file.
• Minor changes to matching classes.

Version 2.7 (19 Aug 2008)

• Variable-length arrays are now mostly supported for FITS binary tables:
  • Columns with TFORM cards containing the 'P' or 'Q' data type descriptors will be read correctly for FITS BINTABLE extensions read from random access sources (which basically means from disk). Tables read from a sequential-only stream will, as before, fail to read variable length array-valued columns.
  • The new TableWriter implementation can write tables in which variable-length columns are represented in the FITS BINTABLE extension by columns with the 'P' or 'Q' data type descriptors.
• New method introduced in class StoragePolicy.
• Various ValueInfo keys for FITS-specific column auxiliary metadata items are now available as static members of .
• Fixes to JDBCFormatter - safer checks on column and table name syntax.
• Sexagesimal field identification for ASCII input files less stringent (now permits minutes or seconds equal to 60).
• HEALPix bug fix (PixTools bug fix update).
• Take more steps to use StoragePolicy when loading JDBC tables, avoiding some JDBC-driver-based out of memory issues.

Version 2.7-1 (27 Mar 2009)

• More careful header consistency checks in fits-plus files - corrupted/modified fits-plus less likely to generate errors.
• Fits BINTABLE TZERO/TSCAL value reading improvements:
  • Columns with integer TZERO values now read as integers rather than floating point values where possible. This includes unsigned longs ('K'), which were previously represented as doubles with lost precision. Unsigned longs which are too large however (>2⁶³) are read as nulls.
  • It's now configurable whether byte columns are written as signed bytes (TFORM=B,TZERO=-128) or as signed shorts (TFORM=I).
  • More comprehensive testing.
  • Fixed bug in calculating value scaled double ('D') values.
  • Fixed bug in typing value for scaled float ('E') arrays.
• The fixed length Substring Array Convention for string arrays (TFORMnn=rAw) is now understood for FITS binary tables.

Version 2.7-2 (17 July 2009)

• VOTable xtype and ref column attributes can be read and written by use of suitable ColumnInfo aux data keys, defined as static members of VOStarTable.

Version 2.8 (2 Oct 2009)

• VOTable namespace handling has been improved. Previously VOTable documents with xmlns namespacing declarations were mostly rejected by STIL. Now the behaviour is configurable.
  • A new class is introduced which takes care of pluggable namespace handling.
  • The default is now lax handling; anything that looks like it is probably a VOTable element is treated as such. This means that documents both with and without xmlns declarations should work. The behaviour of previous versions was approximately that corresponding to none.
  • A new system property votable.namespacing has been introduced to control behaviour from outside the JVM.
  • New VOElement methods getElementsByVOTagName and getVOTagName have been introduced for convenience of use of elements in the VOTable namespace.
  • The semantics of the VOElement methods getChildByName and getChildrenByName are slightly changed (now return only elements in VOTable namespace.
• VOTable 1.2 is now supported. The supported version is the PR 2009-09-29, which is not expected to differ significantly from the REC when it is approved. Support for versions 1.0 and 1.1 is unaffected. API changes are:
  • FieldRefElement and ParamRefElement have new getUcd and getUtype methods.
  • FieldElement has new getXtype method.
• Be more careful in XML, including VOTable, output; fix VOTable output encoding to be UTF-8, and ensure no illegal XML characters are written.
• HTML table output is now HTML 4.01 by default (includes THEAD and TBODY tags).
• Work around illegally truncated type declarations in IPAC tables.
• Bug fixed in crossmatching output: entries which should have been null were sometimes written as non-null (typically large negative numbers) in FITS and in non-TABLEDATA VOTable output. This affected cells in otherwise non-nullable columns where the entire row was absent. The previous behaviour is not likely to have been mistaken for genuine results.

Versions 2.9*x

STIL versions 2.9x, 2.9-1x, 2.9-2x, 2.9-3x did not get a public release, since the backwardly-incompatible changes they contained were not stable, but were present in some versions of TOPCAT and STILTS. Details of what changed in which of these versions are only available by examining relevant versions of the XML sources for SUN/252 (sun252.xml) in the version control system. All the changes are amalgamated into version 3.0.

Version 3.0 (23 December 2010)

This major release contains some new capabilities and some backward incompatibilities with respect to the previous public release, version 2.8. The major changes are in the following areas:

• Multiple table read (new capability)
• Multiple table write (new capability)
• GUI load/save dialogues (major overhaul)
• New Adaptive storage policy as default

Anyone implementing a table read handler, write handler, load dialogue or save dialogue will need to make some adjustments since the relevant interfaces have changed. Anyone using the GUI load dialogue classes in package (as far as I know, nobody apart from me is) will require significant rewriting. Other users of the library will probably find no or only minor issues if compiling against the new version. In most cases significant changes will be marked by compilation errors rather than silent changes in behaviour. The exception is use of the new Adaptive storage policy which is now the default; this change is expected to be beneficial rather than problematic in most cases.

If you experience any difficulties in upgrading from a previous version to this one, please contact the author, who will be happy to advise or assist.

The changes in more detail are as follows:

Multiple table read:

• New interface which s may implement if they know how to load multiple tables from the same source. The FITS and VOTable handlers now implement this.
• Three new makeStarTables methods added to StarTableFactory. These return a which contains multiple tables. Multiple tables are only a possibility if the relevant handler implements the new interface (of the supplied handlers, FITS and VOTable).
• StarTableFactory methods makeStarTable(URL) and makeStarTable(URL,String) are deprecated. They are very thin utility wrappers around existing methods which may be replaced easily in calling code using the class. These methods may be removed in a future release.

Multiple table write:

• New interface , which extends , for output handlers which can write multiple tables to the same container. Corresponding methods added to StarTableOutput.
• MultiStarTableWriter is implemented for most variants of the FITS and VOTable output handlers, to generate multi-extension FITS and multi-TABLE VOTable outputs respectively. Implementations for some other output handlers generating output that may not be machine-readable as a multi-table file are provided as well.

GUI load/save dialogues:

There have been substantial changes to the GUI load framework, mainly to support multiple table load and non-modal load dialogues. The main interface is still called , but its definition has changed considerably. See the javadocs for details.

The save dialogue framework has also undergone some incompatible changes to support writing of multiple files, though these are less dramatic. There are backwardly incompatible effects on the APIs of SaveWorker, TableSaveChooser and TableSaveDialog and its implementations.

Storage policies:

• New StoragePolicy , which effectively uses memory for relatively small tables and scratch disk files for relatively large ones. The intention is that for most purposes this can be used without the user or the programmer having to guess whether small or large tables are likely to be in use. The implementation makes use in some circumstances of direct byte buffer allocation (=malloc()), which means that the size of the controlling java process can grow beyond the size of the maximum specified java heap. The Adaptive storage policy is the new default.
• Added method to ByteStore class.
• Implementation changes in Disk storage policy to reduce memory footprint.

Other:

• Library now distributed as zip of jars (stil_jars.zip) as an alternative to monolithic jar file (stil.jar). This may be more appropriate for those using the library in a framework that contains other third party class libraries.
• VOTableBuilder.makeStarTable now works in a streaming fashion. This should be much faster in the case of a VOTable document containing many TABLE elements. There is a possibility that behaviour will change slightly (some post-positioned INFO/PARAM elements may not get picked up, tables may be successfully loaded from invalid XML documents) - I don't believe these are likely to cause trouble, but please alert me if you disagree.
• Updated VOTable 1.2 schema to final version (elementFormDefault="qualified").
• New attribute Utype for s. ValueInfo has new method getUtype, DefaultValueInfo has new method setUtype, and Tables.get/setUtype is deprecated.
• FITS files now store table names in EXTNAME (and possibly EXTVAR) header cards.
• Added configurable JDBC type conversion framework for reading results from SQL queries. By default JDBC results with Date-type contents will now be turned into String values, but this can be configured by supplying a custom . Previously they were left as Date-type objects, which meant that without special attention they could not be written by general-purpose table output handlers.
• Better behaviour (warn + failover) when attempting to read large files on 32-bit OS or JVM.
• VOTable PARAM output now works out nullability and unspecified array and element size values from the data rather than leaving them as unspecified.
• The wantRandom parameter of TableBuilder.makeStarTable has been deprecated in the documentation.
• Fixed an obscure bug which could under rare circumstances cause truncation of strings with leading/trailing whitespace read from text-format files.
• Fixed bug in TST table output.
• Fixed bug in CSV file parsing that could ignore header row in absence of non-numeric columns.
• Fixed minor bug in CSV file parsing which ignored first row in no-header CSV file when calculating column Element Size values.

Version 3.0-1 (9 May 2011)

• Random Groups HDUs are now tolerated, though not interpreted, within FITS files.
• JDBC table input handler now effectively downcasts BigInteger/BigDecimal types to Long/Double. The PostgreSQL JDBC driver seems to use the Big* types routinely for numeric values (which I don't think it used to do).
• Add getLength method to ByteStore interface.
• Add workaround for J2SE bug #4795134, which could cause errors when reading compressed FITS files.
• Fix FITS character handling bug which could cause corrupted FITS files on output in presence of non-ASCII characters.

Version 3.0-2 (30 June 2011)

• Fixed a significant crossmatching bug in SkyMatchEngine. If all points in a table were on one side of the RA=0 line, but the error radius extended across that line, matches on the other side could be missed. Matches could also be missed if different tables used different conventional ranges for RA (e.g. -180..180 in one case and 0..360 in another). This fix may in some, but not most, cases result in slower matching than previously.
• Added new public class which provides a SAX ContentHandler implementation with similar functionality to VOElementFactory.

Version 3.0-3 (27 October 2011)

• PARAMref with no referent no longer causes uncaught NullPointerException.

Version 3.0-4 (17 December 2012)

• Support for VOTable version 1.3 is now implemented. When reading version-1.3 VOTables, empty TD elements, indicating null values, are permitted for all data types, and the new BINARY2 serialization format is supported. When writing version-1.3 VOTables in the TABLEDATA serialization format, empty TD elements are used rather than marking magic values with the VALUES/null attribute.
• Selection of VOTable version for output is now done on a more configurable basis; see the new subsection .
• VOTable output no longer writes schemaLocation attribute by default.
• Infinite values are now encoded correctly in VOTable output ("+Inf"/"-Inf", not "Infinity"/"-Infinity").
• The VOTable MIME type is now parameterised with the (standard, VOTable 1.3) parameter "serialization" rather than the (non-standard) parameter "encoding" to indicate serialization type (TABLEDATA, BINARY etc)
• You can now reference tables in multi-extension FITS files by name (EXTNAME or EXTNAME-EXTVER) as an alternative to by HDU index.
• IPAC output format is now supported.
• IPAC format "long"/"l" column type, which has apparently now become official, is now accepted in input without warning.
• IPAC headers may now use minus signs instead of whitespace.
• Now copes with 1-column CSV files.
• Fix bug which failed when attempting to read FITS files with complex array columns (TFORMn=rC/rM).
• Fix integer arithmetic bug in FileByteStore which failed when attempting to cache very large (multi-Gb) buffers.
• Fix serious and long-standing bug (bad TZERO header, causes subsequent reads to fail) for FITS output of boolean array columns.

Version 3.0-5 (1 July 2013)

• Add read-only support for CDF (NASA Common Data Format) files.
• Fix CSV regression bug introduced at v3.0-4 - CSV files now work again with MSDOS-style line breaks.
• Fixed FITS output bug which could result in badly-formed string-valued header cards (no closing quote).

Version 3.0-6 (4 July 2014)

• Move getCellRenderer method out of ValueInfo interface, and getCellEditor out of DefaultValueInfo. Those methods never belonged there.
• Add getDomainMappers method to ValueInfo.
• Fix TST input handler so TST files with fewer than 3 columns can be read.
• Removed the (GPL) LICENCE.txt file from the distribution. The software is to be considered as LGPL.

Version 3.0-7 (3 October 2014)

• Add support in package uk.ac.starlink.table.gui for displaying table models up to 2^31 rows (larger than 2^27) in a JTable.
• Attempting to write FITS tables with >999 columns now fails with a more helpful error message.
• Improved Unicode handling in VOTables. Fixed a serious bug in VOTable BINARY or BINARY2 output which generated unreadable output if any non-empty column had datatype="unicodeChar". Round-tripping VOTables will now also preserve datatype unicodeChar, rather than squashing the type to char (the serializer tests for a column aux metadata item of VOStarTable.DATATYPE_INFO with value "unicodeChar"). Some lurking Unicode-related issues remain.

Version 3.0-8 (13 November 2014)

• Add (experimental) read-only support for Gaia/DPAC GBIN format.
• Add utility code to table.jdbc.TypeMappers for convenience of JDBC export when you need a 'T' separator in Timestamp ISO-8601 serializations.

Version 3.0-9 (6 Feb 2015)

• Reworked part of the FITS input implementation, in particular adjusting the way memory mapping is done to reduce resource requirements on some platforms. If you notice any difference, it should be reduced virtual and perhaps resident memory usage, and some (~10%?) performance improvements, when reading large FITS/colfits files. If you were having problems with large memory allocations leading to disk thrashing and system lockup when scanning files larger than RAM (this didn't happen on all OSes), these will hopefully have gone away. However, please report anything that appears to be working worse than before, or continued memory usage issues.
• Colfits files can now be accessed from streams, not just uncompressed disk files (though that's not necessarily a good idea).
• Fixed error in fits-var output (PCOUNT header card did not include block alignment gap).
• Add a hack that allows LDAC FITS tables to be treated sensibly in auto-format-detection mode.
• StarTable columns from VOTable FIELDs with unknown (illegal) datatypes are now interpreted as String rather than String[] types. This is much more sensible and avoids some serious problems when serializing to FITS.
• Add experimental OnceRowPipe2 utility class.

Version 3.0-10 (26 Feb 2015)

• Work round nom.tam.fits read bug that could cause EOFExceptions when reading VOTables with the inline FITS serialization, and possibly elsewhere. The symptoms appear to be new since v3.0-9, but could have caused problems elsewhere before that.
• An auxiliary metadata item is now available to mark columns representing unsigned byte values. This is set on input and respected on output by the VOTable and FITS I/O handlers (and on input only for CDF).

Version 3.0-11 (14 April 2015)

• Remove (broken and useless) signatures from jar files in stil_jars.zip distribution file.
• Be less strict about recognising colfits files (tolerate implicit TDIMn headers).
• New system command option input table syntax; you can now use "<syscmd" or "syscmd|" to supply input byte streams from Un*x pipelines.

Version 3.0-12 (10 Jul 2015)

• Fix serious bug in time conversion for CDF TIME_TT2000 data types.
• Update JCDF library to v1.1 (minor changes to do with leap seconds).
• Modify the heuristics that determine whether the first row of a CSV file is a header.

Version 3.0-13 (17 Aug 2015)

• Fix a serious bug in processing of FITS bit vector (TFORMn='rX') columns. Values read from these columns are presented as a boolean[] array. In all previous versions of STIL the bits have appeared in that array in the wrong sequence (LSB..MSB per byte rather than the other way round). Apologies to anyone who may have got incorrect science results from this error in the past, and thanks to Paul Price for helping to diagnose it.
• Fix a less serious bug with TFORMn='rX' processing; attempting to read a single-element bit vector column (TFORMn=1X or X) previously resulted in an error making the file unreadable. Values read from such columns are now presented as Boolean scalars.
• Fix a VOTable reading bug relating to similar data (datatype="bit") appearing in BINARY/BINARY2 serializations. This one was more obvious, it would usually generate an error when attempting to read the file. Since this bug has been present for ever, I assume that bit vectors in binary-encoded VOTables are not widely used.

Version 3.0-14 (22 Oct 2015)

• Fix broken assertion; when reading multi-buffer (large) FITS files with assertions enabled, there was a spurious AssertionError.
• Upgrade to JCDF v1.2 - fixes a read failure when reading large (multi-Gb) CDF files.
• Adjust GBIN input handler: avoid descending into Class-typed members of gbin list objects, add logging for object->column translations, and improve configurability.
• VOTable DOM getParentNode implementation adjusted; I think it's more correct now, and this may make the VOTable DOM behave better with XPath.

Version 3.1 (27 Nov 2015)

• Fix a long-standing crossmatch range-restriction bug in uk.ac.starlink.table.join classes. This could have caused missed associations (but not false positives) near the edge of coverage regions when using per-row errors, if the scale of the errors differed (especially differed significantly) between the matched tables. It affected MatchEngines ErrorCartesianMatchEngine, ErrorSkyMatchEngine, EllipseCartesianMatchEngine and EllipseSkyMatchEngine. Thanks to Grant Kennedy (IoA) for reporting this bug.
• There is a change in the signature of the MatchEngine.getMatchBounds method. This change is backwardly incompatible (hence the minor version number update), but I don't think anybody else is using this API directly, so hopefully the impact on users will be low.
• Try harder to identify epoch columns (suitable for time plot), in particular look for VOTable xtype of JD or MJD, and units of year.

Version 3.1-1 (10 Jun 2016)

• This and subsequent releases target Java SE 6, so can no longer be used to build Java 5 compatible applications or run under the (now very ancient) Java 5 runtime.
• Fix bugs that led to timezone-dependent results when reading ISO-8601 or decimal year time columns.
• Fix numeric field truncation bug in LaTeX table output.
• Fix read failure for FITS files with non-blank TDIM for zero-length columns.

Version 3.1-2 (13 Sep 2016)

• The GBIN input handler can now pick up more metadata from the classpath. For suitable tables, metadata included in datamodel classes if present can be interrogated to provide table and column descriptions and UCDs. There are still some deficiencies of this functionality (no column order, utypes and units missing, large file "temp.xml" written to current directory) dependent on issues in the upstream Gaia libraries and ICD.
• Fix bug that caused read failures for large (>0.5Gb) FITS files outside of current directory on 32-bit JVMs. This was a regression bug since v3.0-9.
• Fix long-standing bug in FitsTableBuilder that failed to close streams (hence release file descriptors) when opening FITS tables. Also implement java.io.Closeable in BintableStarTable and ColFitsStarTable to allow explicit release of file descriptors.
• Update JCDF library to v1.2-2 (2017-01-01 leap second).

Version 3.2 (8 Mar 2017)

• Add a new method getScoreScale() to the uk.ac.starlink.table.join.MatchEngine interface. This is required to support reasonable behaviour for Best matching of CombinationMatchEngines. It is a backwardly-incompatible change, but I'm not aware of MatchEngine implementations outside of the starjava codebase, so I doubt if it will cause trouble.

Version 3.2-1 (29 Sep 2017)

• A non-standard convention has been introduced which allows all the FITS-based I/O handlers to use FITS BINTABLE extensions for tables with more than 999 columns. See . In earlier versions, attempting to write such wide tables to a FITS file failed with an error.
• FITS headers using the ESO HIERARCH convention are now recognised on input. Previously they were ignored.
• The VOTable input handler now looks for COOSYS elements referenced from FIELD elements and makes the relevant information available as column auxiliary metadata. The relevant aux metadata keys are defined in the class as COOSYS_*_INFO.
• The VOTable and FITS-plus output handlers now write and reference COOSYS elements in output VOTables corresponding to the requirements of any .COOSYS_*_INFO aux metadata items attached to table columns (FIELD elements). This means that COOSYS information associated with table columns can be round-tripped during read-write cycles that use VOTable-based serialization formats for both input and output. Note though that it doesn't currently work for references from PARAMs.
• The default version for output VOTables is now VOTable 1.3. New output formats votable-binary2-inline and votable-binary2-href are now offered alongside the five previously available VOTable variants.
• The FITS-plus output handler now writes VOTables using the default output VOTable version, as described in . Previously VOTable 1.1 was used.
• Modified the thread-safe implementation of methods performing random data access (getRow, getCell) to mapped byte buffers on ColFitsStarTable and BintableStarTable. These now use ThreadLocal accessors for reads from the underlying ByteBuffers rather than synchronized blocks. This performs much much better in the case of heavy contention (random access to table data from multiple concurrent threads) than before. There may be a small degradation in single-threaded performance - tests indicate around 1-2%.
• Add new method for creating a GBIN-like table from a collection of existing GaiaRoot-like objects, without having to read them directly from a GBIN file. See GbinStarTable.createCollectionTable.
• Update PixTools to 2017-09-06 version (https://github.com/kuropat/eag-HEALPix, 447a7be073876dba32). This fixes a bug in vect2pix_ring that gave the wrong value for HEALPix tile index in one half of each tile straddling the longitude=0 line in the equatorial region. It seems possible that this might have led to very infrequent missed associations when crossmatching in these regions, but tests appear to indicate that no such errors would actually have resulted.
• Long fields (>10240 characters) in output CSV files are no longer truncated.

Version 3.2-2 (7 Nov 2017)

• Add tweak to VOTable reader that discards any INFO elements with names starting "uk.ac.starlink.topcat.plot2.TopcatLayer". This is a nasty workaround for a nasty bug in TOPCAT v4.5, which added such metadata items in large numbers to tables that were plotted and then saved.

Version 3.2-3 (13 Nov 2017)

• STIL classes should now build and run with recent versions (1.15.2, and probably later) of the nom.tam.fits library. It is still bundled with a custom nom.tam.fits (based on v0.96), but replacing that with a recent version ought to work. Behaviour in that case is expected, but not guaranteed, to be similar to using the bundled version.

Version 3.3 (24 Apr 2018)

• Update JCDF to v1.2-3; fixes some CDF reading bugs.
• The IPAC table reader now matches data type specifications case-insensitively.
• Modify VOTable (Multi)TableBuilder implementation; it now picks up more of the DOM (everything up to the start of the next TABLE element, rather than just everything up to the end of the current TABLE) when a table is read. This is important for picking up DataLink-style Service Descriptors.
• The VOTable I/O handlers can now de/serialise DataLink-style Service Descriptors. These show up as -typed table parameters.
• If you use a ColumnInfo as one of the parameters of a StarTable and serialise it to VOTable, its AuxData will now be honoured, for instance to write VOTable-specific PARAM attributes like ref and ID.
• Multiple-table VOTable documents are now written with each table in its own RESOURCE element, rather than all TABLEs as siblings within the same RESOURCE element. This is to enable grouping of Service Descriptor RESOURCE elements with their associated TABLE. The output is unchanged for single-table VOTable output documents. Hopefully this slight change of output format will not cause compatibility problems with other software.

Version 3.3-1 (18 May 2018)

• VOTable serializer will now write attribute values in single quotes if they contain a lot of double quote characters.

Version 3.3-2 (2 Nov 2018)

• Slight improvements to the JDBC Configuration section of this manual.
• GBIN reading fix to work around changed behaviour in recent GaiaTools versions (>=21.1.0 and >=20.3.0) that caused GBIN table reading to fail.

Version 3.3-3 (9 May 2019)

• Experimental support for VOTable 1.4 and its new TIMESYS element. This support corresponds to WD-VOTable-1.4-20190403. The functionality may be modified in future releases depending on how the VOTable 1.4 specification evolves.
  • For VOTable columns that reference TIMESYS elements, the relevant information is now accessible as column auxiliary metadata keyed by the .TIMESYS_*_INFO static members.
  • Any columns referencing TIMESYS elements read from VOTable-based formats (VOTable or FITS-plus) can now be written out to VOTable-based formats with equivalent TIMESYS references included, so TIMESYS round-tripping for columns works; however this will only be done if the VOTable output version is set to 1.4 (=V14). By default (at least as long as 1.4 is not finalised) the output version is 1.3; it can be set either explicitly on output methods or globally as documented in that class. Currently this TIMESYS output works only for table columns (FIELDs) not parameters (PARAMs).
  • New VOElement subclass and some associated methods.
  • Columns referencing TIMESYS elements now provide a suitable from their method.
• FIELD/@ref attributes are no longer imported as column aux metadata items, since they often interfere with TIMESYS references. Doing this was probably always a bad idea since the referencing is not kept track of within the application, so withdrawing this functionality makes sense, but beware that it might change or break some existing behaviour.
• Various changes to support the semi-standard HEALPix-FITS serialization convention. A new class is added to define metadata relating to a table containing HEALPix pixel data. The FITS input and output handlers attempt to honour this information on a best-efforts basis, and a new fits-healpix output handler is provided that tries harder to write HEALPix data tables conforming to the convention.
• The signatures of some metadata access methods in the core classes have been redefined to use generics. The altered methods, which previously used raw types, are getParameters(), getColumnAuxDataInfos() in , and getAuxData(), setAuxData(List<DescribedValue>) in . These changes enforce behaviour that was previously required by contracts stated in the javadocs. Because of the backward compatibility features of generics this should not cause new errors at compilation or run time as long as the methods were being used in accordance with the existing documentation, but compilation warnings may change.
• Artifacts comprising a Maven package for STIL-IO (the classes excluding the matching capabilities) are now assembled as part of the build process.
• Fix bug/misfeature in CDF table parameter construction: CDF global attributes were ignored (with a "WARNING: Omitting complicated global attribute" message) if they contained any null entries. Now such entries are just ignored and the table parameter is constructed from the global attribute using the non-null entries.
• Be a bit more careful when writing FITS headers TCOMMn, TUCDn and TUTYPn if their value may overflow the 80-character mark, including refusing to write them even if nom-tam-fits long header support is switched on.
• Adjust implementation to work round occasional Debian-Astro build failures.

Version 3.4 (18 November 2019)

• Provide String-based support for offset (e.g. unsigned) 64-bit integers in FITS files. 64-bit integer columns (TFORMn='K') with non-zero integer offsets (TSCALn=1, TZEROn<>0) are now read from FITS as Strings (see ), and such Strings can be written to FITS as long integers (see ). In previous versions an attempt was made to represent in-range values as Java longs, using a null value for out-of-range values.
• Avoid sometimes losing precision when reading ASCII/CSV values in the range +/-(1e-38..1e-45).
• Permit FITS and VOTable files with zero-length string columns. Previously all-null or zero-length string columns were sometimes forced to single-character values.
• Update mapped file unmapping implementation to work for java9+.

Version 3.4-1 (5 June 2020)

• The ECSV (Enhanced Character Separated Values) storage format is now supported for input and output.
• The Feather storage format is now supported for input and output.
• Replace PixtoolsHealpixSkyPixellator with CdsHealpixSkyPixellator in the uk.ac.starlink.table.join package, based on the cds-healpix-java library from F-X Pineau (CDS). The new implementation is generally faster.
• FITS ASCII table extensions with TFORM values of In are now treated as 64-bit integers for n>=10 rather than n>10.
• Improve performance when reading long String values from FITS files.

Version 4.0 (11 January 2021)

This major release provides support for parallel processing of table data, enhances table I/O specification options, addresses some long-standing issues that require backwardly incompatible API changes, and breaks out crossmatching classes into a separate package. The documentation has been extended accordingly. Library users using existing STIL functionality for table I/O should not need to make too many code changes when upgrading from STIL v3, but those providing StarTable or I/O handler implementations may need to modify their implementations in accordance with the API changes; see below for details.

Notable new functionality

• Support for parallel table data processing added, see .
• Improved patterns for threadsafe random access using StarTable.getRowAccess method, see .
• I/O handler configuration options may be configured by user-supplied string, see . Config options are provided for several handler implementations; more may be forthcoming in future releases.
• Table Schemes introduced, which can be used to load tables not serialized as byte streams, see .
• Auto file format detection now examines filenames to help guess input file format.
• StarTable.close() can now be called to release global resources such as file descriptors and (where unmapping is supported) mapped files.

API changes

• New data access methods added to StarTable interface to support multithreaded processing: getRowAccess, getRowSplittable.
• New default methods added to StarTable interface for convenience in parameter handling: getParameterByName and setParameter.
• StarTable has a new close method that should relinquish any non-heap-based resources (file descriptors, mapped files). In most cases this can be implemented as a no-op.
• RowSequence and StarTable now implement java.io.Closeable so they can be used in try-with-resources statements.
• New method looksLikeFile added to TableBuilder interface, to enable filename-based (file extension-based) input format guessing.
• Aux data access methods getAuxData, getAuxDatumByName, setAuxDatum are now on the ValueInfo interface, rather than on (ValueInfo's subtype) the ColumnInfo class. This means that table parameters, as well as table columns, can now sport auxiliary metadata. That should really always have been the case.
• Add ValueInfo.getXtype method.
• Clarified requirements of class RandomStarTable; implemented data access methods must be thread-safe.
• RowData introduced as super-interface of RowSequence.
• Class RandomWrapperStarTable has been withdrawn; the implementation was complicated and hard to upgrade, and it was probably(?) never used.
• Some other minor API changes.

Package contents

• The classes in the namespace uk.ac.starlink.table.join (table joins and crossmatches) are no longer part of the STIL library. Those classes are still available elsewhere, but the functionality is distinct from the I/O that STIL mostly provides, there was never corresponding tutorial text in the user document, they required external dependencies that sometimes complicated STIL deployment, and they are not needed by most STIL users. The dependencies cdshealpix.jar and htmIndex.jar are no longer required.

Documentation

• Descriptions of input handlers () and output handlers () updated.
• Add new documentation section describing the the FITS-plus convention.
• Add Documented interface to improve auto-documentation of I/O handlers. Handler documentation now resides in the code itself, and is extracted programmatically to generate entries in user documentation.

Miscellaneous enhancements and bugfixes

• COOSYS and TIMESYS attributes are now preserved during VOTable I/O for table PARAMs (as well as for FIELDs, which was already the case).
• Add miscellaneous utility methods in Tables class etc.
• Fix ECSV output bug: encoding was incorrect for metadata scalars with certain non-alphanumeric first characters, leading to invalid YAML.
• Remove some unhelpful per-column metadata items from ECSV output.
• Prevented HealpixSkyPixellator.calculateDefaultK from returning -1 for large angles.
• Improve exception handling in StreamTableSink/OnceRowPipe implementations, make interruption work better.
• Fix bug that could give unhelpful table load error message for very short non-FITS files in auto-detection mode.

Version 4.0-1 (16 April 2021)

• Variable-length array-valued columns in FITS tables (P/Q descriptors) can now be read even in compressed or streamed input.
• Add configuration option header for CSV input handler, to indicate whether header line is present.
• Add configuration option maxSample for CSV and ASCII input handlers to reduce 2-pass read time.
• The fits-var output handler now avoids use of the THEAP keyword (no pre-heap gap is written). Heap padding is legal FITS, but bugs in other FITS software mean that some third party components (including fverify) have problems with such files.
• Fix bugs that meant writing long (>2Gb) fits-var files could output illegal/corrupted FITS.
• Fixed long-standing bug in FileByteStore.copy(); this potentially serious issue could have caused broken file caching of any/all streams, but mostly seemed to affect large (>2Gb) streams in practice.
• JDBC output no longer attempts to create VARCHAR(0) columns.

Version 4.0-2 (10 June 2021)

New functionality:

• Apache Parquet format is now supported for input and output (note not all distributions include Parquet-MR support libraries).
• AAS Machine-Readable Table (MRT) format is now supported for input.
• ECSV format input and output handlers are upgraded to version 1.0 of the ECSV format, meaning they can now read and write array-valued columns.
• and subclasses now have configuration methods setAllowSignedByte, setAllowZeroLengthString, setWide and setPadCharacter. These should be used in preference to the various custom constructors, which have now been deprecated.
• Add new table scheme test (TestTableScheme).

Minor enhancements and behaviour changes:

• Space-delimited ECSV files now write empty fields quoted.
• Unknown or unsupported column datatype values in ECSV files are now treated like string rather than causing table read failure.
• Empty strings in FITS 1-character columns are now returned as blank values rather than ASCII NUL ('\0').
• Undersized, including zero-length, strings written to FITS columns are now by default terminated with an ASCII NUL rather than in some cases padded with spaces.

Bugfixes:

• Fix regression bug since v4.0 that meant table drag'n'drop wasn't working.
• Fix regression bug since v4.0 that meant jdbc: URLs didn't work.
• Fix long-standing logic error in ASCII/CSV input handler that could misidentify column types and cause read failures.
• FITS TZERO headers are now written correctly with numeric values rather than string values.

Version 4.0-3 (2 July 2021)

• STIL is now available at the Maven Central Repository under the groupId uk.ac.starlink. Packaging details may change in future releases.
• Bzip2 decoding is now done by internal classes (uk.ac.starlink.util.bzip2) rather than external apache library; this reduces the dependency requirement in the POM.
• Guess meaning for some non-standard COORDSYS values in HEALPix-FITS files, e.g. allow "GALACTIC" instead of "G".

Version 4.0-4 (15 Oct 2021)

• Columns that are all blank in ASCII-like tables (CSV, ASCII, TST) are now interpreted as type String not boolean.
• Fix serious threading bug that could return nonsense values from fixed-length string fields during parallel processing of large cached/randomised tables.
• Files compressed using multi-stream bzip2 compression (e.g. pbzip2 output) are now supported alongside single-stream bzip2.

Version 4.0-5 (31 January 2022)

• The PDS4 (NASA's Planetary Data System v4) file format is now supported for input tables.
• Improve identification of TIME_TT2000 columns as time values in certain CDF files.
• Bugfix update of JCDF to v1.2-4.
• Fix failure when attempting to read unsigned 32-bit integer values from parquet files.

Version 4.1 (6 April 2022)

• Dependency on the nom.tam.fits library is removed All FITS I/O is now implemented internally.
• New classes are introduced in the package, to replace nom.tam.fits usage: see , , , , , , .
• Other API changes: java.io.DataOutput parameter is replaced by java.io.OutputStream in some FITS and VOTable I/O methods (e.g. ).
• FITS header I/O now supports reading FITS 4.0 long-string headers (CONTINUE records).
• FITS header values of the form "(a,b,c,...)" used as BINTABLE table parameters are now interpreted where possible as numeric arrays; this works for long-string values (CONTINUE records) as well.
• New classes introduced in package UTIL for fast buffered I/O: , .
• Substantial I/O performance improvements, mainly for FITS and VOTable, e.g.: writing to FITS 2x, reading FITS from a stream 2x, reading VOTable with inline BINARY/BINARY2 4x, writing VOTable with inline BINARY/BINARY2 2x, writing VOTable with TABLEDATA 1.5x.
• Add configuration options compact and encoding to VOTable output handler. By default thin (<=4 column) TABLEDATA VOTables are now written in "compact" mode, using reduced whitespace.
• ECSV format now preserves table name.
• FITS BINTABLE reader now copes with (illegal?) embedded spaces in TDIMn headers.
• Adjust MRT null handling; "-" in a single-character field is no longer interpreted as null.

Version 4.1-1 (10 June 2022)

• Fix FITS parsing issue that could result in StackOverflowError for long array-valued headers.
• Fix bug in multi-threaded read of string columns from colfits files.
• Reduce number of file mapping calls by FITS readers.

Version 4.1-2 (8 July 2022)

• The .Config class now has a pluggable option to control handling of FITS header string/comment values that contain illegal FITS header characters. Previously such characters would case FITS output to fail; now by default such characters are replaced with a '?' character.

Version 4.1-3 (5 October 2022)

• Fix PDS4 reader to accept columns of type ASCII_Numeric_Base16 without the read operation failing.
• Fix VOTable reader so BINARY/2 VOTables with no columns don't read forever.

Version 4.1-4 (20 April 2023)

• Modify behaviour when writing URL-valued aux metadata items in VOTable FIELDs: null name for ValueInfo now results in title-less LINK instead of no LINK.
• The JDBC input handler should now cope with columns that are array-valued in the database, viewing their contents as String[] or primitive array values where possible. Previously they showed up as java.sql.Array objects, which are generally not easy to deal with in STIL.
• Add some configuration options to the GBIN input handler.
• Fail with an error rather than silently reading a broken table when encountering GaiaTools/zStd-jni bug during GBIN input.
• Empty/invalid fields encountered by the PDS4 reader no longer cause the table read to fail.
• Make FITS and VOTable output handlers robust against input tables that declare incorrect row counts.
• Modify column width determination in text-like output formats (text, ascii, ipac) to avoid occasional unwanted truncation of formatted values. Tables are now read in two passes, the first to establish column widths and the second to write the data. By default all rows are sampled, but the sampledRows option can be configured so that only some rows are sampled, which is more like the old behaviour.
• PDS4 reader now reads Ascii_Numeric_Base16/8/2 fields as numeric not string (updated pds4-jparser library code).
• Slight change to RESOURCE structure of Primary HDU metadata in multi-table FITS-plus output. This fixes a problem in which saved Service Descriptors could end up associated with the wrong tables.
• Write empty string not semi-standard "nan" token for NaN in ECSV writer.
• Improve ECSV reader performance, especially for Gaia DR3 bulk download files (which use semi-standard "null" token).
• The class now has access to contentType and exampleURL parameters introduced in DataLink 1.1. These are preserved by VOTable I/O.

Version 4.2 (1 November 2023)

From this version the access to HTTP(S) URLs is by default done using the AUTH package, which provides transparent authenticated access to resources protected by authentication declared according to the (currently draft) VO authentication standards. This ought not to affect STIL behaviour unless steps are taken to initialise authentication, but unforseen changes are possible. See .

Version 4.2-1 (29 February 2024)

• Implement HAPI input handler and scheme support.
• VOTable COOSYS/@refposition attribute is now available from column aux metadata as . STIL now fully(?) supports WD-VOTable-1.5-20231120.
• Improve error reporting for corrupted/truncated FITS files.
• Use authentication and HTTP-level compression for external (href-referenced) VOTable STREAM data.
• Upgrade snakeyaml library (used for ECSV headers) from 1.25 to 2.2. No change in behaviour (or security) expected, but prevents vulnerability warnings in some circumstances.
• Fix bug that could generate spurious EOFExceptions when reading multi-table basic FITS files.

Version 4.3 (7 August 2024)

• At this version the source code and build system have been substantially tidied up so that the packages can be built straightforwardly using modern versions of Java (8, 11, 17, 21) with a minimum of warnings. These changes should be mostly invisible to users, but some behaviour changes are possible.
• Some changes have been made to URL handling. Syntactically invalid URIs are now mostly rejected. This should not be noticeable in most cases, but questionable usages like embedded spaces in URLs may need to be replaced by their %-encoded equivalents.
• Documentation for input and output handler config options now mostly reports their default values.
• FITS output handling improved and reorganised to provide more flexible configuration options. Most FITS output handlers are now deprecated in favour of . This is all now documented under the FITS heading, there is no longer a separate colfits section in the documentation.
• New test scheme options "g" and "w".
• New config option date for VOTable output handler to control whether a datestamp comment is written.
• Fits-plus output handlers now honour date config option for VOTable metadata as well as the DATE-HDU header card.
• Upgrade parquet support libraries to parquet-mr 1.13.1. This means that snappy compression is now supported for MacOS ARM, as well as other, architectures.
• Add support for LZ4_RAW compression to Parquet I/O handlers.
• Parquet reader now copes with some other variants of array-valued columns.
• New config option tryUrl in parquet input handler, set false by default to avoid cryptic error messsages when trying to open remote parquet files.
• New config option compression in Parquet output handler.
• New config option usedict in Parquet output handler.
• Decrease size of parquet output files in most cases when writing NaNs and empty strings/arrays.
• Fix parquet input handler bug related to compression.
• Fix VOTable output bug that wrote infinite floating point array elements to TABLEDATA as "+/-Infinity" rather than "+/-Inf".
• Xtypes are now written into FITS headers using the non-standard header card TXTYPnnn.
• Non-standard FITS headers TUCDnnn and TUTYPnnn are now written with comment parts, space permitting.
• Avoid writing arraysize="1" in FIELD/PARAM elements of VOTables with version >=1.3, in accordance with VOTable 1.3 Erratum #3.
• New config option useFloat in MRT input handler; this is set false by default to avoid a rare bug that read large values as infinite.

Version 4.3-1 (6 November 2024)

• Update JSON-java library to 20240303. Behaviour is not expected to change in interesting ways, but some vulnerabilities are addressed.
• More units (seconds with SI prefixes) are now properly handled when assigning Time for VOTable TIMESYS-referencing fields.
• Fix regression bug (since v4.2) in HTTP 3xx redirect handling that failed to cope with relative Location field values.

Version 4.3-2 (7 March 2025)

• Parquet input and output handlers now support the VOParquet convention for embedded metadata.
• Fix some significant parquet I/O bugs related to array-valued columns.
• The parquet reader can now read data with the un-annotated BINARY (BYTE_ARRAY) type; it is interpreted as a byte array.
• The CDF input handler can now extract multiple tables from a single CDF file.
• Upgrade JCDF to version 1.2.5. Significant improvements to CDF read performance.
• Embedded underscores may be used in numeric literals when specifying table schemes loop and test.
• Fix NullPointerException when writing TABLEDATA VOTable output for null entries in String[]-valued columns.
• Add kvmap config option to parquet output handler.
• Fix ivoa_cookie authentication handling to work in presence of multiple cookies.

Version 4.3-3 (30 July 2025)

• VOTable and associated output handlers can now write String[]-valued columns without known element lengths - see configured via new class . Related: earlier VOSerializer.makeSerializer methods are now deprecated in favour of makeSerializer(VOSerializerConfig,StarTable).
• Fix bug when writing long HIERARCH FITS headers that could cause output failure.
• Add new delimiter config option to CSV input and CSV output handlers.
• CSV and ASCII input handlers now have a method to control what column types are inferred.

Version 4.3-4 (12 September 2025)

• Fix regression issue (v4.3-3) that caused output trouble for tables read from ASCII/CSV files including completely blank columns.