4.1.1.11 Parquet

Parquet is a columnar format developed within the Apache project. Data is compressed on disk and read into memory before use. The file format is described at https://github.com/apache/parquet-format. This software is written with reference to version 2.10.0 of the format.

This input handler will read columns representing scalars, strings and one-dimensional arrays of the same. It is not capable of reading multi-dimensional arrays, more complex nested data structures, or some more exotic data types like 96-bit integers. If such columns are encountered in an input file, a warning will be emitted through the logging system and the column will not appear in the read table. Support may be introduced for some additional types if there is demand.

Parquet files typically do not contain rich metadata such as column units, descriptions, UCDs etc. To remedy that, this reader supports the VOParquet convention (version 1.0), in which metadata is recorded in a DATA-less VOTable stored in the parquet file header. If such metadata is present it will by default be used, though this can be controlled using the votmeta configuration option below.

Depending on the way that the table is accessed, the reader tries to take advantage of the column and row block structure of parquet files to read the data in parallel where possible.

Note:

The parquet I/O handlers require large external libraries. In previous releases of this software they were not always bundled with the library/application software because of their size, but in the current release they are included in the topcat-full.jar, stilts.jar and topcat-all.dmg files, so it is likely they will be present. So you usually will not encounter this problem, but if these libraries are absent then attempts to read or write parquet files will result in a message like:

   Parquet-mr libraries not available

If you can supply the relevant libaries on the classpath at runtime, the parquet support will work; they can be identified as dependencies of org.apache.parquet/parquet-cli/1.13.1.

The bundled libraries support most, but not all, of the compression formats defined for parquet, currently uncompressed, gzip, snappy, zstd, lz4 and lz4_raw. Supplying more of the parquet-mr/parquet-java dependencies at runtime would extend this list. Unlike the rest of TOPCAT/STILTS/STIL which is written in pure java, some of these libraries (currently the snappy and zstd compression codecs) contain native code, which means they may not work on all architectures. At time of writing all common architectures are covered, but there is the possibility of failure with a java.lang.UnsatisfiedLinkError on other platforms if attempting to read/write files that use those compression algorithms.

The handler behaviour may be modified by specifying one or more comma-separated name=value configuration options in parentheses after the handler name, e.g. "parquet(cachecols=true,nThread=4)". The following options are available:

cachecols = true|false|null

Forces whether to read all the column data at table load time. If true, then when the table is loaded, all data is read by column into local scratch disk files, which is generally the fastest way to ingest all the data. If false, the table rows are read as required, and possibly cached using the normal STIL mechanisms. If null (the default), the decision is taken automatically based on available information. (Default: null)

nThread = <int>

Sets the number of read threads used for concurrently reading table columns if the columns are cached at load time - see the cachecols option. If the value is <=0 (the default), a value is chosen based on the number of apparently available processors. (Default: 0)

tryUrl = true|false

Whether to attempt to open non-file URLs as parquet files. This usually seems to fail with a cryptic error message, so it is not attempted by default, but it's possible that with suitable library support on the classpath it might work, so this option exists to make the attempt. (Default: false)

votmeta = true|false|null

If true, the content of the parquet extra metadata key-value list item with key IVOA.VOTable-Parquet.content will be read to supply the metadata for the input table, following the VOParquet convention. If false, any such VOTable metadata is ignored. If set null, the default, then such VOTable metadata will be used only if it is present and apparently consistent with the parquet data and metadata. (Default: null)

maml = true|false|null

If true, the content of the parquet extra metadata key-value list item with key maml will be read to supply additional metadata for the input table. If set false, such an item will be ignored. If set null, the default, then such MAML metadata will be used if it is present and no better source of additional metadata is apparent. MAML is a YAML-based metadata format defined at https://github.com/asgr/MAML-Format. Partial MAML support is implemented; some of the more useful per-column and per-table items are applied, but not everything is. (Default: null)

votable = <filename-or-url>

Location of a UTF-8-encoded data-less VOTable that will supply additional metadata for a parquet table being read, according to the VOParquet convention. This is normally not required, but if present it overrides any such metadata VOTable embedded within the parquet file. This value will only be used if the votmeta configuration is not false. (Default: null)

This format can be automatically identified by its content so you do not need to specify the format explicitly when reading parquet tables, regardless of the filename.