RowListStarTable.JoinStarTable can now
deduplicate column names.ConcatStarTable
permits adding the rows of one table after the rows of another.RowSequence interface
has been modified; a new
close
method has been introduced, and the old
advance() and getRowIndex() methods have been
withdrawn (these latter were not very useful and in some cases
problematic to implement).
setName() and setURL()
have been added to the StarTable
interface.
StoragePolicy class was
introduced, which allows you to influence whether cached table data are
stored in memory or on disk.
This has led to backwardly-incompatible changes to public interfaces and
classes:
makeStarTable
now takes a new StoragePolicy argument, and
VOElementFactory's methods
are now instance methods rather than static ones.
StarTableFactory class's
makeStarTable methods now come in two flavours - with and
without a format name. This corresponds to two table reading modes:
named format mode and automatic format detection mode.
In named format mode you specify the format of the table you are
trying to read and in automatic format detection mode you rely
on the factory to work it out
using magic numbers. Although automatic detection works well
for VOTable and FITS, it's poor for text-based formats like
ASCII and CSV.
This has resulted in addition of some new two-argument
makeStarTable methods and the withdrawal of the
getBuilders method in favour of two new methods
getDefaultBuilders and
getKnownBuilders
(similarly for setter methods) which deal with the handlers
used in automatic detection mode and the ones available for
named format mode respectively.
Note that the ASCII table format is not automatically detected,
so to use ASCII tables you now have to specify the format explicitly.
VOElement class
has been rewritten and now implements the DOM
Element interface.
This means that the hierarchical structure which you can navigate
to obtain information about the VOTable document and extract
table data actually is a DOM rather than just being sat
on top of one. You can therefore now use it just as a normal
DOM tree (making use of the methods defined in the
org.w3c.dom interface,
interoperating with third-party components which require a DOM).
This has had a number of additional benefits and consequences:
VOElementFactory
class now has instance methods rather than static methods.
StoragePolicy.DISCARD
into a VOElementFactory
it is now possible to obtain a data-less
(structure only, hence minimal resource) VOTable DOM.
TableSink methods
now throw exceptions.
StarTableFactory's list for automatic format detection,
but CSV-format tables can be loaded using named format mode.
The format is intended to match the (widely-used) variety used by
Microsoft Excel amongst others (with optional column names).
basic-fits").
StoragePolicy).
Short/Float
types in preference to Integer/Double
if the input data make this appropriate.
d or D as an exponent letter
as well as e or E.
uk.ac.starlink.table.join..
These work and have full javadocs, but
they are still experimental, subject to substantial change in future
releases, and not documented properly in this document.
NULL_VALUE_INFO)
Nulls in FITS and VOTable/FITS tables are now preserved correctly
on output.
TCOMMx
headers.
EmptyStarTable
added.
StarTable objectwantRandom flag has been changed in name and
semantics to requireRandom in
StarTableFactory.
When set, any table returned from the factory is now guaranteed
to have random access.
StarTableOutput now has a new method
writeStarTable
which writes a table to an OutputStream as well
as the one which writes to a location string (usually filename).
This is supported by changes to the writeStarTable
methods which StarTableWriter
implementations must provide.
uk.ac.starlink.table.gui.StarTableChooser
table loader dialogue has been improved in several ways.
Loading is now done asynchronously, so that the GUI does not
lock up when a long load is taking place (a load cancel button can
be pressed).
Additionally, custom load dialogues have been made pluggable,
so that you can add new load sub-dialogues by implementing
TableLoadDialog
(most likely subclassing BasicTableLoadDialog)
and naming the class in the
startable.load.dialogs
property.
A dialogue for browsing AstroGrid's MySpace remote filestore is
available, but for reasons of size STIL is not by default
packaged with all the
classes required to make it work (AXIS and the CDK are missing).
setParameter
has been added to the StarTable interface.
BeanStarTable which can store
Java Beans has been introduced. This is handy for storing arrays
of objects of the same kind without having to write a custom table
implementation.
arraysize workaround
arraysize attribute; by default
it is now assumed to have the value "*" rather
than "1" as the standard dictates.
See Section 7.3.4.
xsi:schemaLocation="http://www.ivoa.net/xml/VOTable/v1.1
http://www.ivoa.net/xml/VOTable/v1.1"
instead of
xsi:noNamespaceSchemaLocation="http://www.ivoa.net/xml/VOTable/v1.1"
(thanks to Paul Harrison for suggesting this correction).tpipe command introduced
tpipe command has been tentatively introduced
at this release.
This useful command-line tool is experimental and may undergo
major changes or be moved to a separate package altogether
in future releases.
There have been changes to some of the main interfaces:
hasNext() method has been withdrawn
from the RowSequence
interface and the next() method, which used to be
declared void, now returns
boolean indicating whether there is another row.
This is quite likely to break existing code, but the fix is easy;
simply replace:
RowSequence rseq = table.getRowSequence();
while ( rseq.hasNext() ) {
rseq.next();
...
}
with
RowSequence rseq = table.getRowSequence();
while ( rseq.next() ) {
...
}
streamStarTable
has been added to the TableBuilder interface
to provide improved support for table streaming.
uk.ac.starlink.gui package.
StarTableChooser and StarTableSaver
have been replaced by TableLoadChooser and
TableSaveChooser, and these both now use a
graphical widget which can view files in remote filestores
(such as MySpace and SRB) if the relevant classes are present.
Minor changes:
Tables.sortTable method.ExplodedStarTable.ConcatStarTable.arraysize="1" explicitly for
scalar character fields.StarTable<->VOTable
serializer/deserializer classes in
uk.ac.starlink.votable.soap package.TableContentHandler class provides table-aware
SAX processing of VOTable document streams.arraysize attribute for character fields
is now interpreted by default according to the VOTable standard
rather than by default being worked around - i.e. an unspecified
votable.strict system property now counts as
true rather than false. This is the reverse of the behaviours
in versions 2.1 and 2.2. See Section 7.3.4.tablecopy command deprecated and tpipe
withdrawn - these are now available within the new package
STILTS.writeInlineTableElement, writeHrefTableElement
to VOSerializer.VOStarTable parameter setting.ConcatStarTable which was leading to
ClassCastExceptions when used sequentially.RowMatcher class.null values.null" (unquoted) in ASCII input handler
is interpreted as a blank entry.StarTableWriter.getMimeType
has been introduced.mark.workaround
system property.outColfits).text and
(especially) ascii output handlers.RowStepper class by RowSequence in
votable package. As well as being a bit tidier, this
improves efficiency considerably for column-oriented access in
some cases (esp. fits-plus/colfits-plus).fits-plus &
(especially) colfits-plus format access in some situations
(related to above point).colfits-basic format is now auto-detected.JDBCFormatter
API for more flexible use with creating tables in RDBMS.UnrepeatableSequenceException.SequentialResultSetStarTable
and
RandomResultSetStarTable
which are StarTable implementations built directly on
JDBC ResultSet
objects.JoinFixAction interface and
implementation changed. Now better at deduplicating the names of
joined tables.TNULLn header
cards - write them as numeric not string values.getUtype and
setUtype
utility methods.RowPipe interface and
implementations and
new createOutputSink methods in
StarTableOutput.TUTYPnn header
cards.RowPipe and
OnceRowPipe.uk.ac.starlink.table.join,
including multi-pair join.datatype attribute.text-mode output (may result in longer fields too).VariableFitsTableWriter
TableWriter implementation can write tables in which variable-length
columns are represented in the FITS BINTABLE extension by columns
with the 'P' or 'Q' data type descriptors.
makeByteStore
introduced in class StoragePolicy.ValueInfo keys for FITS-specific
column auxiliary metadata items are now available as static members of
BintableStarTable.JDBCFormatter - safer checks on column and table
name syntax.StoragePolicy when loading JDBC
tables, avoiding some JDBC-driver-based out of memory issues.TFORMnn=rAw)
is now understood for FITS binary tables.xtype and ref column attributes
can be read and written by use of suitable ColumnInfo aux data keys,
defined as static members of VOStarTable.xmlns namespacing
declarations were mostly rejected by STIL.
Now the behaviour is configurable.
Namespacing
is introduced which takes care of pluggable namespace handling.
getElementsByVOTagName and
getVOTagName have been introduced
for convenience of use of elements in the VOTable namespace.
getChildByName and getChildrenByName
are slightly changed (now return only elements in VOTable
namespace.
FieldRefElement and ParamRefElement
have new getUcd and getUtype methods.
FieldElement has new getXtype method.
uk.ac.starlink.table.gui
(as far as I know, nobody apart from me is)
will require significant rewriting.
Other users of the library will probably find no or only minor issues
if compiling against the new version.
In most cases significant changes will be marked by compilation errors
rather than silent changes in behaviour.
The exception is use of the new Adaptive storage policy which is now
the default; this change is expected to be beneficial rather than
problematic in most cases.
If you experience any difficulties in upgrading from a previous version to this one, please contact the author, who will be happy to advise or assist.
The changes in more detail are as follows:
MultiTableBuilder
interface which
TableBuilders
may implement if they know how to load multiple tables from the
same source. The FITS and VOTable handlers now implement this.
makeStarTables methods added to
StarTableFactory.
These return a TableSequence which contains multiple tables.
Multiple tables are only a possibility if the relevant handler
implements the new
MultiTableBuilder
interface (of the supplied handlers, FITS and VOTable).
makeStarTable(URL) and
makeStarTable(URL,String)
are deprecated.
They are very thin utility wrappers around existing methods
which may be replaced easily in calling code using the
URLDataSource class.
These methods may be removed in a future release.
MultiStarTableWriter,
which extends
StarTableWriter,
for output handlers which can write multiple tables to the same
container.
Corresponding methods added to StarTableOutput.
TableLoadDialog,
but its definition has changed considerably.
See the javadocs
for details.
The save dialogue framework has also undergone some
incompatible changes to support writing of multiple files,
though these are less dramatic. There are backwardly incompatible
effects on the APIs of
SaveWorker, TableSaveChooser
and TableSaveDialog and its implementations.
ADAPTIVE,
which effectively uses memory for relatively small tables
and scratch disk files for relatively large ones.
The intention is that for most purposes this can be used
without the user or the programmer having to guess whether
small or large tables are likely to be in use.
The implementation makes use in some circumstances of
direct byte buffer allocation (=malloc()),
which means that the size of the controlling java process can
grow beyond the size of the maximum specified java heap.
The Adaptive storage policy is the new default.
toByteBuffers
to ByteStore class.
stil_jars.zip)
as an alternative to monolithic jar file (stil.jar).
This may be more appropriate for those using the library in a
framework that contains other third party class libraries.
VOTableBuilder.makeStarTable now works in a streaming
fashion.
This should be much faster in the case of a VOTable document
containing many TABLE elements. There is a possibility that
behaviour will change slightly (some post-positioned INFO/PARAM
elements may not get picked up, tables may be successfully
loaded from invalid XML documents) - I don't believe these
are likely to cause trouble, but please alert me if you disagree.
ValueInfos.
ValueInfo has new method getUtype,
DefaultValueInfo has new method setUtype,
and Tables.get/setUtype is deprecated.
TypeMapper.
Previously they were left as Date-type objects, which meant that
without special attention they could not be written by
general-purpose table output handlers.
wantRandom parameter of
TableBuilder.makeStarTable has been deprecated in
the documentation.
getLength method to ByteStore
interface.VOTableDOMBuilder
which provides a SAX ContentHandler implementation
with similar functionality to VOElementFactory.
TD
elements, indicating null values, are permitted for all data types,
and the new BINARY2 serialization format is supported.
When writing version-1.3 VOTables in the TABLEDATA serialization
format, empty TD elements are used rather than marking magic
values with the VALUES/null attribute.+Inf"/"-Inf",
not "Infinity"/"-Infinity").long"/"l" column type,
which has apparently now become official,
is now accepted in input without warning.TFORMn=rC/rM).FileByteStore which failed
when attempting to cache very large (multi-Gb) buffers.getCellRenderer method out of ValueInfo
interface, and getCellEditor out of
DefaultValueInfo. Those methods never belonged there.getDomainMappers method to ValueInfo.uk.ac.starlink.table.gui
for displaying table models up to 2^31 rows (larger than 2^27)
in a JTable.table.jdbc.TypeMappers for
convenience of JDBC export when you need a 'T' separator in
Timestamp ISO-8601 serializations.OnceRowPipe2 utility class.Tables.UBYTE_FLAG_INFO
is now available to mark columns representing unsigned byte values.
This is set on input and respected on output by the VOTable and FITS
I/O handlers (and on input only for CDF).
stil_jars.zip distribution file.<syscmd" or "syscmd|"
to supply input byte streams from Un*x pipelines.TFORMn='rX') columns.
Values read from these columns are presented as a
boolean[] array. In all previous versions of STIL
the bits have appeared in that array in the wrong sequence
(LSB..MSB per byte rather than the other way round).
Apologies to anyone who may have got incorrect science results
from this error in the past, and thanks to Paul Price for helping
to diagnose it.TFORMn='rX' processing;
attempting to read a single-element bit vector column
(TFORMn=1X or X) previously
resulted in an error making the file unreadable.
Values read from such columns are now presented as
Boolean scalars.datatype="bit") appearing in BINARY/BINARY2
serializations. This one was more obvious, it would usually
generate an error when attempting to read the file.
Since this bug has been present for ever, I assume that
bit vectors in binary-encoded VOTables are not widely used.uk.ac.starlink.table.join classes.
This could have caused missed associations (but not false positives)
near the edge of coverage regions when using per-row errors,
if the scale of the errors differed
(especially differed significantly) between the matched tables.
It affected MatchEngines ErrorCartesianMatchEngine,
ErrorSkyMatchEngine, EllipseCartesianMatchEngine and
EllipseSkyMatchEngine.
Thanks to Grant Kennedy (IoA) for reporting this bug.MatchEngine.getMatchBounds method.
This change is backwardly incompatible (hence the minor version
number update), but I don't think anybody else is using this API
directly, so hopefully the impact on users will be low.xtype of JD or MJD,
and units of year.FitsTableBuilder
that failed to close streams (hence release file descriptors)
when opening FITS tables.
Also implement java.io.Closeable
in BintableStarTable and ColFitsStarTable
to allow explicit release of file descriptors.getScoreScale()
to the uk.ac.starlink.table.join.MatchEngine interface.
This is required to support reasonable behaviour for Best
matching of CombinationMatchEngines.
It is a backwardly-incompatible change, but I'm not aware of
MatchEngine implementations outside of the starjava
codebase, so I doubt if it will cause trouble.
COOSYS elements
referenced from FIELD elements and makes the relevant
information available as column auxiliary metadata.
The relevant aux metadata keys are defined in the
VOStarTable class as
COOSYS_*_INFO.COOSYS elements in output VOTables corresponding to
the requirements of any
VOStarTable.COOSYS_*_INFO
aux metadata items attached to table columns
(FIELD elements).
This means that COOSYS information associated with table columns
can be round-tripped during read-write cycles that use
VOTable-based serialization formats for both input and output.
Note though that it doesn't currently work for
references from PARAMs.votable-binary2-inline and
votable-binary2-href are now offered alongside
the five previously available VOTable variants.getRow, getCell)
to mapped byte buffers
on ColFitsStarTable and BintableStarTable.
These now use ThreadLocal accessors for reads from the
underlying ByteBuffers rather than synchronized blocks.
This performs much much better in the case of heavy contention
(random access to table data from multiple concurrent threads)
than before. There may be a small degradation in single-threaded
performance - tests indicate around 1-2%.447a7be073876dba32).
This fixes a bug in vect2pix_ring that gave
the wrong value for HEALPix tile index in one half of each tile
straddling the longitude=0 line in the equatorial region.
It seems possible that this might have led to very infrequent
missed associations when crossmatching in these regions,
but tests appear to indicate that no such errors would
actually have resulted.uk.ac.starlink.topcat.plot2.TopcatLayer".
This is a nasty workaround for a nasty bug in TOPCAT v4.5,
which added such metadata items in large numbers to tables that
were plotted and then saved.Multi)TableBuilder
implementation; it now picks up more of the DOM
(everything up to the start of the next TABLE element,
rather than just everything up to the end of the
current TABLE) when a table is read.
This is important for picking up DataLink-style
Service Descriptors.ServiceDescriptor-typed table parameters.ref and ID.TIMESYS element.
This support corresponds to WD-VOTable-1.4-20190403.
The functionality may be modified in future releases depending
on how the VOTable 1.4 specification evolves.
VOStarTable.TIMESYS_*_INFO static members.VOTableVersion=V14).
By default (at least as long as 1.4 is not finalised)
the output version is 1.3; it can be set either explicitly
on output methods or globally as documented in that class.
Currently this TIMESYS output works only for table columns
(FIELDs) not parameters (PARAMs).TimesysElement
VOElement subclass and some associated
methods.TimeMapper
from their
getDomainMappers method.FIELD/@ref attributes are no longer imported as
REF_INFO
column aux metadata items,
since they often interfere with TIMESYS references.
Doing this was probably always a bad idea since the referencing
is not kept track of within the application, so withdrawing
this functionality makes sense, but beware that it might
change or break some existing behaviour.HealpixTableInfo
is added to define metadata relating to a table containing
HEALPix pixel data.
The FITS input
and output handlers attempt to honour
this information on a best-efforts basis, and a new
fits-healpix output handler is provided that
tries harder to write HEALPix data tables
conforming to the convention.getParameters(),
getColumnAuxDataInfos()
in StarTable,
and
getAuxData(),
setAuxData(List<DescribedValue>)
in ColumnInfo.
These changes enforce behaviour that was previously required by
contracts stated in the javadocs.
Because of the backward compatibility features of generics
this should not cause new errors at compilation or run time
as long as the methods were being used in accordance with the
existing documentation, but compilation warnings may change.
TCOMMn, TUCDn and TUTYPn
if their value may overflow the 80-character mark,
including refusing to write them even if nom-tam-fits
long header support is switched on.PipeReaderThread
implementation to work round occasional Debian-Astro
build failures.TFORMn='K')
with non-zero integer offsets
(TSCALn=1, TZEROn<>0)
are now read from FITS as Strings (see Section 3.6.1),
and such Strings can be written to FITS as long integers
(see Section 3.7.1).
In previous versions an attempt was made to represent
in-range values as Java longs, using a null value
for out-of-range values.PixtoolsHealpixSkyPixellator with
CdsHealpixSkyPixellator
in the uk.ac.starlink.table.join package,
based on the
cds-healpix-java library
from F-X Pineau (CDS).
The new implementation is generally faster.In
are now treated as 64-bit integers for n>=10
rather than n>10.StarTable or I/O handler implementations
may need to modify their implementations in accordance with the API
changes; see below for details.
StarTable.getRowAccess method,
see Section 2.3.2.StarTable.close() can now be called to
release global resources such as file descriptors and
(where unmapping is supported) mapped files.StarTable interface
to support multithreaded processing:
getRowAccess, getRowSplittable.default methods added to StarTable interface
for convenience in parameter handling:
getParameterByName and setParameter.StarTable has a new close method
that should relinquish any non-heap-based resources
(file descriptors, mapped files).
In most cases this can be implemented as a no-op.RowSequence and StarTable
now implement java.io.Closeable
so they can be used in try-with-resources statements.looksLikeFile added to
TableBuilder interface,
to enable filename-based (file extension-based)
input format guessing.getAuxData, getAuxDatumByName,
setAuxDatum are now on the ValueInfo
interface, rather than on (ValueInfo's subtype)
the ColumnInfo class. This means that
table parameters, as well as table columns, can now sport
auxiliary metadata. That should really always have been
the case.ValueInfo.getXtype method.RandomStarTable;
implemented data access methods must be thread-safe.RowData introduced as super-interface of
RowSequence.RandomWrapperStarTable has been withdrawn;
the implementation was complicated and
hard to upgrade, and it was probably(?) never used.uk.ac.starlink.table.join
(table joins and crossmatches)
are no longer part of the STIL library.
Those classes are still available elsewhere,
but the functionality is distinct from the I/O that
STIL mostly provides, there was never corresponding
tutorial text in the user document,
they required external dependencies that sometimes
complicated STIL deployment, and they are not needed
by most STIL users.
The dependencies cdshealpix.jar and
htmIndex.jar are no longer required.Documented interface to improve
auto-documentation of I/O handlers.
Handler documentation now resides in the code itself,
and is extracted programmatically to generate entries
in user documentation.Tables
class etc.HealpixSkyPixellator.calculateDefaultK
from returning -1 for large angles.StreamTableSink/OnceRowPipe
implementations, make interruption work better.P/Q descriptors)
can now be read even in compressed or streamed input.header for
CSV input handler,
to indicate whether header line is present.maxSample for
CSV and ASCII
input handlers to reduce 2-pass read time.fverify) have problems
with such files.FileByteStore.copy();
this potentially serious issue could have caused broken
file caching of any/all streams, but mostly seemed to
affect large (>2Gb) streams in practice.AbstractFitsTableWriter
and subclasses now have configuration methods
setAllowSignedByte,
setAllowZeroLengthString,
setWide and setPadCharacter.
These should be used in preference to the various custom
constructors, which have now been deprecated.TestTableScheme).string rather than
causing table read failure.\0').jdbc:
URLs didn't work.uk.ac.starlink.
Packaging details may change in future releases.uk.ac.starlink.util.bzip2) rather than
external apache library; this reduces the dependency
requirement in the POM.pbzip2 output) are now supported
alongside single-stream bzip2.uk.ac.starlink.fits package,
to replace nom.tam.fits usage: see
CardImage,
CardType,
CardFactory,
ParsedCard,
FitsHeader,
FitsUtil,
AsciiTableStarTable.java.io.DataOutput
parameter is replaced by java.io.OutputStream
in some FITS and VOTable I/O methods (e.g.
writeHrefDataElement).DataBufferedInputStream,
DataBufferedOutputStream.compact and encoding to
VOTable output handler.
By default thin (<=4 column) TABLEDATA VOTables
are now written in "compact" mode, using reduced whitespace.CardFactory.Config
class now has a pluggable option to control handling of
FITS header string/comment values that contain illegal
FITS header characters. Previously such characters would
case FITS output to fail; now by default such characters
are replaced with a '?' character.ASCII_Numeric_Base16
without the read operation failing.FIELDs:
null name for ValueInfo
now results in title-less LINK
instead of no LINK.String[] or primitive array values where possible.
Previously they showed up as java.sql.Array objects,
which are generally not easy to deal with in STIL.text,
ascii,
ipac)
to avoid occasional unwanted truncation of formatted values.
Tables are now read in two passes, the first to establish
column widths and the second to write the data.
By default all rows are sampled, but the
sampledRows option can be configured so that
only some rows are sampled, which is more like
the old behaviour.Ascii_Numeric_Base16/8/2
fields as numeric not string
(updated pds4-jparser library code).nan"
token for NaN in ECSV writer.null" token).ServiceDescriptor
class now has access to contentType and
exampleURL parameters introduced in DataLink 1.1.
These are preserved by VOTable I/O.COOSYS/@refposition attribute
is now available from column aux metadata as
VOStarTable.COOSYS_REFPOSITION_INFO.
STIL now fully(?) supports WD-VOTable-1.5-20231120.%-encoded equivalents.UnifiedFitsTableWriter.
This is all now documented under the FITS
heading, there is no longer a separate colfits
section in the documentation.test scheme options
"g" and "w".date for
VOTable output handler
to control whether a datestamp comment is written.date
config option for VOTable metadata
as well as the DATE-HDU header card.tryUrl
in parquet input handler,
set false by default to avoid cryptic error messsages
when trying to open remote parquet files.compression in
Parquet output handler.usedict in
Parquet output handler.+/-Infinity"
rather than "+/-Inf".TXTYPnnn.TUCDnnn and TUTYPnnn
are now written with comment parts, space permitting.arraysize="1"
in FIELD/PARAM elements
of VOTables with version >=1.3,
in accordance with VOTable 1.3 Erratum #3.useFloat
in MRT input handler;
this is set false by default to avoid a rare bug
that read large values as infinite.