The FITS BINTABLE standard (FITS Standard v3.0, section 7.3) permits a maximum of 999 columns in a binary table extension. Up to version 3.2 of STIL, attempting to write a table with more than 999 columns using one of the supported FITS-based writers failed with an error. In later versions, a non-standard convention is used which can store wider tables in a FITS BINTABLE extension. The various STIL FITS-based readers can (in their default configurations) read these tables transparently, allowing round-tripping of arbitrarily wide tables to FITS files. Note however that other FITS-compliant software is not in general aware of this convention, and will see a 999-column table. The first 998 columns will appear as intended, but subsequent ones will effectively be hidden.
The rest of this section describes the convention that is used to store tables with more than 999 columns in FITS BINTABLE extensions.
The BINTABLE extension type requires table column metadata
to be described using 8-character keywords of the form
TXXXX represents one of an open set of mandatory, reserved
or user-defined root keywords up to five characters in length,
nnn part is an integer between 1 and 999 indicating the
index of the column to which the keyword in question refers.
Since the header syntax confines this indexed part of the keyword
to three digits, there is an upper limit of 999 columns in
Note that the FITS/BINTABLE format does not entail any restriction on the storage of column data beyond the 999 column limit in the data part of the HDU, the problem is just that client software cannot be informed about the layout of this data using the header cards in the usual way.
The following convention is used by STIL FITS-based I/O handlers to accommodate wide tables in FITS files:
N_TOTis the total number of data columns to be stored
N_TOTinclusive are known as extended columns. Their data is stored within the container column.
N_TOT) is laid out in the data part of the HDU in exactly the same way as if there were no 999-column limit.
TFIELDSheader is declared with the value 999.
TFORM999value corresponding to the total field length required by all the extended columns ('
B' is the obvious data type, but any legal
TFORMvalue that gives the right width MAY be used). The byte count implied by
TFORM999MUST be equal to the total byte count implied by all extended columns.
TXXXX999headers MAY optionally be declared to describe the container column in accordance with the usual rules, e.g.
TTYPE999to give it a name.
NAXIS1header is declared in the usual way to give the width of a table row in bytes. This is equal to the sum of all the BINTABLE column widths as usual. It is also equal to the sum of all the data column widths, which has the same value.
XT_ICOLindicates the index of the container column. It MUST be present with the integer value 999 to indicate that this convention is in use.
XT_NCOLindicates the total number of data columns encoded. It MUST be present with an integer value equal to
HIERARCH XT TXXXXnnnnn', where
TXXXXare the same keyword roots as used for normal BINTABLE extensions, and
nnnnnis a decimal number written as usual (no leading zeros, as many digits as are required). Thus the formats for data columns 999, 1000, 1001 etc are declared with the keywords
HIERARCH XT TFORM999,
HIERARCH XT TFORM1000,
HIERARCH XT TFORM1001, etc. Note this uses the ESO HIERARCH convention described at https://fits.gsfc.nasa.gov/registry/hierarch_keyword.html. The name space token has been chosen as '
XT' (extended table).
The resulting HDU is a completely legal FITS BINTABLE extension. Readers aware of this convention may use it to extract column data and metadata beyond the 999-column limit. Readers unaware of this convention will see 998 columns in their intended form, and an additional (possibly large) column 999 which contains byte data but which cannot be easily interpreted.
An example header might look like this:
XTENSION= 'BINTABLE' / binary table extension BITPIX = 8 / 8-bit bytes NAXIS = 2 / 2-dimensional table NAXIS1 = 9229 / width of table in bytes NAXIS2 = 26 / number of rows in table PCOUNT = 0 / size of special data area GCOUNT = 1 / one data group TFIELDS = 999 / number of columns XT_ICOL = 999 / index of container column XT_NCOL = 1204 / total columns including extended TTYPE1 = 'posid_1 ' / label for column 1 TFORM1 = 'J ' / format for column 1 TTYPE2 = 'instrument_1' / label for column 2 TFORM2 = '4A ' / format for column 2 TTYPE3 = 'edge_code_1' / label for column 3 TFORM3 = 'I ' / format for column 3 TUCD3 = 'meta.code.qual' ... TTYPE998= 'var_min_s_2' / label for column 998 TFORM998= 'D ' / format for column 998 TUNIT998= 'counts/s' / units for column 998 TTYPE999= 'XT_MORECOLS' / label for column 999 TFORM999= '813I ' / format for column 999 HIERARCH XT TTYPE999 = 'var_min_u_2' / label for column 999 HIERARCH XT TFORM999 = 'D' / format for column 999 HIERARCH XT TUNIT999 = 'counts/s' / units for column 999 HIERARCH XT TTYPE1000 = 'var_prob_h_2' / label for column 1000 HIERARCH XT TFORM1000 = 'D' / format for column 1000 ... HIERARCH XT TTYPE1203 = 'var_prob_w_2' / label for column 1203 HIERARCH XT TFORM1203 = 'D' / format for column 1203 HIERARCH XT TTYPE1204 = 'var_sigma_w_2' / label for column 1204 HIERARCH XT TFORM1204 = 'D' / format for column 1204 HIERARCH XT TUNIT1204 = 'counts/s' / units for column 1204 END
This general approach was suggested by William Pence on the FITSBITS list in June 2012, and by François-Xavier Pineau (CDS) in private conversation in 2016. The details have been filled in by Mark Taylor (Bristol), and discussed in some detail on the FITSBITS list in July 2017.