uk.ac.starlink.parquet

## Class ParquetTableBuilder

- java.lang.Object
- - uk.ac.starlink.table.formats.DocumentedTableBuilder
  - - uk.ac.starlink.parquet.ParquetTableBuilder

- All Implemented Interfaces:
    Documented, DocumentedIOHandler, TableBuilder

      public class ParquetTableBuilder
      extends DocumentedTableBuilder

  TableBuilder for parquet files. As well as reading basic Parquet data and metadata, embedded metadata following the VOParquet convention is by default read and applied.
  Since:
    25 Feb 2021

- - ### Constructor Summary

    Constructors
    Constructor and Description
    `ParquetTableBuilder()` Constructor.

  - ### Method Summary

    All Methods Static Methods Instance Methods Concrete Methods
    Modifier and Type | Method and Description
    --- | ---
    `boolean` | `canImport(java.awt.datatransfer.DataFlavor flavor)` Indicates whether this builder is able to turn a resource of media type indicated by `flavor` into a table.
    `boolean` | `canStream()` Returns false; parquet metadata is in the footer.
    `boolean` | `docIncludesExample()` Indicates whether the serialization of some (short) example table should be added to the user documentation for this handler.
    `java.lang.Boolean` | `getCacheCols()` Returns policy for table construction.
    `java.lang.String` | `getFormatName()` Returns the name of the format which can be read by this handler.
    `java.lang.Boolean` | `getMamlMetadata()` Indicates whether a MAML block in the input parquet file will be used to supply metadata.
    `int` | `getReadThreadCount()` Returns the number of read threads to use when caching column data.
    `boolean` | `getTryUrl()` Indicates whether an attempt is made to open parquet files from non-file URLs.
    `java.lang.String` | `getVOTableLocation()` Returns the location of a data-less VOTable document that will be used to supply additional metadata for a parquet table being read.
    `java.lang.Boolean` | `getVOTableMetadata()` Indicates whether a DATA-less VOTable stored in input parquet file will be used to supply metadata.
    `java.lang.String` | `getXmlDescription()` Returns user-directed documentation in XML format.
    `StarTable` | `makeStarTable(DataSource datsrc, boolean wantRandom, StoragePolicy storage)` Constructs a `StarTable` based on a given `DataSource`.
    `static java.lang.String` | `readUtf8FromLocation(java.lang.String loc)` Reads all the characters from a given location into a string.
    `void` | `setCacheCols(java.lang.Boolean cacheCols)` Determines policy for table construction.
    `void` | `setMamlMetadata(java.lang.Boolean mamlMeta)` Determines whether a MAML block in the input parquet file will be used to supply metadata.
    `void` | `setReadThreadCount(int nThread)` Sets the number of read threads to use when caching column data.
    `void` | `setTryUrl(boolean tryUrl)` Configures whether an attempt is made to open parquet files from non-file URLs.
    `void` | `setVOTableLocation(java.lang.String votableLoc)` Sets the location of a data-less VOTable document that will be used to supply additional metadata for a parquet table being read.
    `void` | `setVOTableMetadata(java.lang.Boolean votMeta)` Determines whether a DATA-less VOTable stored in input parquet file will be used to supply metadata.
    `void` | `streamStarTable(java.io.InputStream istrm, TableSink sink, java.lang.String pos)` Reads a table from an input stream and writes it a row at a time to a sink.

    - ### Methods inherited from class uk.ac.starlink.table.formats.DocumentedTableBuilder

      `getExtensions, looksLikeFile`

    - ### Methods inherited from class java.lang.Object

      `clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait`

    - ### Methods inherited from interface uk.ac.starlink.table.formats.DocumentedIOHandler

      `matchesExtension, readText, toLink`

- - ### Constructor Detail

    - #### ParquetTableBuilder

          public ParquetTableBuilder()

      Constructor.

  - ### Method Detail

    - #### getFormatName

          public java.lang.String getFormatName()

      Description copied from interface: `TableBuilder`
      Returns the name of the format which can be read by this handler. Matching against this string may be used by callers to identify or select this handler from a list.
      Returns:
        one-word description of this handler's format

    - #### canStream

          public boolean canStream()

      Returns false; parquet metadata is in the footer.
      Specified by:
        `canStream` in class `DocumentedTableBuilder`
      Returns:
        true iff this handler can read from streams

    - #### docIncludesExample

          public boolean docIncludesExample()

      Description copied from interface: `DocumentedIOHandler`
      Indicates whether the serialization of some (short) example table should be added to the user documentation for this handler. Binary formats, or instances for which the `Documented.getXmlDescription()` method already includes some example output, should return false.
      Returns:
        true if the user documentation would benefit from the addition of an example serialization

    - #### getXmlDescription

          public java.lang.String getXmlDescription()

      Description copied from interface: `Documented`
      Returns user-directed documentation in XML format.

      The output should be a sequence of one or more <P> elements, using XHTML-like XML. Since rendering may be done in a number of contexts however, use of the full range of XHTML elements is discouraged. Where possible, the content should stick to simple markup such as the elements P, A, UL, OL, LI, DL, DT, DD EM, STRONG, I, B, CODE, TT, PRE.

      Returns:
        XML description of this object

    - #### makeStarTable

          public StarTable makeStarTable(DataSource datsrc,
                                         boolean wantRandom,
                                         StoragePolicy storage)
                                  throws java.io.IOException

      Description copied from interface: `TableBuilder`
      Constructs a `StarTable` based on a given `DataSource`. If the source is not recognised or this builder does not know how to construct a table from it, then a `TableFormatException` should be thrown. If this builder thinks it should be able to handle the source but an error occurs during processing, an `IOException` can be thrown.

      The `wantRandom` parameter is used to indicate whether, ideally, a random-access table should be returned. There is no requirement for the builder to honour this request, but if it knows how to make both random and non-random tables, it can use this flag to decide which to return.

      Note: the presence of the `wantRandom` parameter is somewhat misleading. TableBuilder implementations usually should, and do, ignore it (it would be removed from the interface if it were not for backward compatibility issues). Regardless of the value of this parameter, implementations should return a random-access table only if it is easy for them to do so; in particular they should not use the supplied `storagePolicy`, or any other resource-expensive measure, to randomise a sequential table just because the `wantRandom` parameter is true.

      Parameters:
        `datsrc` - the DataSource containing the table resource
        `wantRandom` - whether, preferentially, a random access table should be returned
        `storage` - a StoragePolicy object which may be used to supply scratch storage if the builder needs it
      Returns:
        a StarTable made out of `datsrc`
      Throws:
        `TableFormatException` - if the table is not of a kind that can be handled by this handler
        `java.io.IOException` - if an unexpected I/O error occurs during processing

    - #### streamStarTable

          public void streamStarTable(java.io.InputStream istrm,
                                      TableSink sink,
                                      java.lang.String pos)
                               throws TableFormatException

      Description copied from interface: `TableBuilder`
      Reads a table from an input stream and writes it a row at a time to a sink. Not all implementations will be able to do this; for instance, extracting the table from the data may be a two-pass process. Implementations which are unable to perform this function should throw a `TableFormatException`.

      The input stream should be prepared for use prior to calling this method, so implementations should not in general attempt to decompress or buffer `istrm`.

      Parameters:
        `istrm` - input stream containing table data
        `sink` - destination of the table
        `pos` - position identifier describing the location of the table within the stream; see `DataSource.getPosition()` (may be null)
      Throws:
        `TableFormatException` - if the table can't be streamed or the data is malformed

    - #### canImport

          public boolean canImport(java.awt.datatransfer.DataFlavor flavor)

      Description copied from interface: `TableBuilder`
      Indicates whether this builder is able to turn a resource of media type indicated by `flavor` into a table. It should return `true` if it thinks that its `TableBuilder.streamStarTable(java.io.InputStream, uk.ac.starlink.table.TableSink, java.lang.String)` method stands a reasonable chance of successfully constructing a `StarTable` from a `DataSource` whose input stream is described by the `DataFlavor` `flavor`. It will typically make this determination based on the flavor's MIME type.

      This method should only return `true` if the flavor looks like it is targeted at this builder; for instance a builder which uses a text-based format should return false for a flavor which indicates a MIME type of `text/plain`.

      This method is used in supporting drag and drop functionality (see `StarTableFactory.canImport(java.awt.datatransfer.DataFlavor[])`).

      Parameters:
        `flavor` - the DataFlavor whose suitability as stream input is to be assessed
      Returns:
        `true` iff this builder reckons it stands a good chance of turning a stream of type `flavor` into a `StarTable`

    - #### setCacheCols

          @ConfigMethod(property="cachecols",
                        usage="true|false|null",
                        example="true",
                        doc="<p>Forces whether to read all the column data at table load\ntime.  If <code>true</code>, then when the table is loaded,\nall data is read by column into local scratch disk files,\nwhich is generally the fastest way to ingest all the data.\nIf <code>false</code>, the table rows are read as required,\nand possibly cached using the normal STIL mechanisms.\nIf <code>null</code> (the default), the decision is taken\nautomatically based on available information.\n</p>")
          public void setCacheCols(java.lang.Boolean cacheCols)

      Determines policy for table construction. If true, the `makeStarTable` method returns a `CachedParquetStarTable` and if false a `SequentialParquetStarTable`. If null, the decision is made automatically on the basis of whether it looks like random access is required and file size etc.
      Parameters:
        `cacheCols` - column data read policy

    - #### getCacheCols

          public java.lang.Boolean getCacheCols()

      Returns policy for table construction.
      Returns:
        true for caching, false for read as required, null for adaptive

    - #### setReadThreadCount

          @ConfigMethod(property="nThread",
                        usage="<int>",
                        example="4",
                        doc="<p>Sets the number of read threads used for concurrently\nreading table columns if the columns are cached at load time\n- see the <code>cachecols</code> option.\nIf the value is &lt;=0 (the default), a value is chosen\nbased on the number of apparently available processors.\n</p>")
          public void setReadThreadCount(int nThread)

      Sets the number of read threads to use when caching column data. This is the value passed to the `CachedParquetStarTable` constructor, and ignored when constructing a `SequentialParquetStarTable`.
      Parameters:
        `nThread` - read thread count, or <=0 for auto

    - #### getReadThreadCount

          public int getReadThreadCount()

      Returns the number of read threads to use when caching column data.
      Returns:
        read thread count, or <=0 for auto

    - #### setTryUrl

          @ConfigMethod(property="tryUrl",
                        doc="<p>Whether to attempt to open non-file URLs as parquet files.\nThis usually seems to fail with a cryptic error message,\nso it is not attempted by default, but it\'s possible that with\nsuitable library support on the classpath it might work,\nso this option exists to make the attempt.\n</p>",
                        example="true")
          public void setTryUrl(boolean tryUrl)

      Configures whether an attempt is made to open parquet files from non-file URLs.
      Parameters:
        `tryUrl` - true to attempt opening non-file URLs

    - #### getTryUrl

          public boolean getTryUrl()

      Indicates whether an attempt is made to open parquet files from non-file URLs.
      Returns:
        true to attempt non-file URLs, false to just give up

    - #### setVOTableMetadata

          @ConfigMethod(property="votmeta",
                        example="false",
                        doc="<p>If true, the content of the parquet extra metadata\nkey-value list item with key\n<code>IVOA.VOTable-Parquet.content</code>\nwill be read to supply the metadata for the input table,\nfollowing the\n<webref url=\'https://www.ivoa.net/documents/Notes/VOParquet/\'>VOParquet convention</webref>.\nIf false, any such VOTable metadata is ignored.\nIf set null, the default, then such VOTable metadata\nwill be used only if it is present and apparently consistent\nwith the parquet data and metadata.\n</p>")
          public void setVOTableMetadata(java.lang.Boolean votMeta)

      Determines whether a DATA-less VOTable stored in input parquet file will be used to supply metadata. A null value (the default) uses such metadata if it is present.
      Parameters:
        `votMeta` - whether to read metadata from VOTable text

    - #### getVOTableMetadata

          public java.lang.Boolean getVOTableMetadata()

      Indicates whether a DATA-less VOTable stored in input parquet file will be used to supply metadata. A null value (the default) uses such metadata if it is present.
      Returns:
        whether to read metadata from VOTable text

    - #### setMamlMetadata

          @ConfigMethod(property="maml",
                        example="true",
                        doc="<p>If true, the content of the parquet extra metadata\nkey-value list item with key\n<code>maml</code> will be read\nto supply additional metadata for the input table.\nIf set false, such an item will be ignored.\nIf set null, the default, then such MAML metadata\nwill be used if it is present and no better source of\nadditional metadata is apparent.\nMAML is a YAML-based metadata format defined at\n<webref url=\'https://github.com/asgr/MAML-Format\'/>.\nPartial MAML support is implemented;\nsome of the more useful per-column and per-table items are\napplied, but not everything is.\n</p>")
          public void setMamlMetadata(java.lang.Boolean mamlMeta)

      Determines whether a MAML block in the input parquet file will be used to supply metadata. A null value (the default) uses such metadata if it is present.
      Parameters:
        `mamlMeta` - whether to read metadata from a MAML block

    - #### getMamlMetadata

          public java.lang.Boolean getMamlMetadata()

      Indicates whether a MAML block in the input parquet file will be used to supply metadata. A null value (the default) uses such metadata if it is present.
      Returns:
        whether to read metadata from a MAML block

    - #### setVOTableLocation

          @ConfigMethod(property="votable",
                        doc="<p>Location of a UTF-8-encoded data-less VOTable\nthat will supply additional metadata for a parquet table\nbeing read, according to the\n<webref url=\'https://www.ivoa.net/documents/Notes/VOParquet/\'>VOParquet convention</webref>.\nThis is normally not required, but if present it overrides\nany such metadata VOTable embedded within the parquet file.\nThis value will only be used if the <code>votmeta</code>\nconfiguration is not false.\n</p>",
                        usage="<filename-or-url>",
                        example="./metadata.vot")
          public void setVOTableLocation(java.lang.String votableLoc)

      Sets the location of a data-less VOTable document that will be used to supply additional metadata for a parquet table being read. Normally not required, but if present it overrides any such metadata VOTable embedded within the parquet file. This value will only be used if `getVOTableMetadata()` returns a non-FALSE value.
      Parameters:
        `votableLoc` - filename or URL of UTF-8-encoded data-less VOTable

    - #### getVOTableLocation

          public java.lang.String getVOTableLocation()

      Returns the location of a data-less VOTable document that will be used to supply additional metadata for a parquet table being read.
      Returns:
        override location for a UTF-8-encoded data-less VOTable

    - #### readUtf8FromLocation

          public static java.lang.String readUtf8FromLocation(java.lang.String loc)
                                                       throws java.io.IOException

      Reads all the characters from a given location into a string. UTF-8 encoding is assumed.
      Parameters:
        `loc` - filname or URL
      Returns:
        content of stream
      Throws:
        `java.io.IOException`