Next Previous Up Contents
Next: ecsv
Up: Input Formats
Previous: cdf

#### 5.1.1.5 `csv`

Comma-separated value ("CSV") format is a common semi-standard text-based format in which fields are delimited by commas. Spreadsheets and databases are often able to export data in some variant of it. The intention is to read tables in the version of the format spoken by MS Excel amongst other applications, though the documentation on which it was based was not obtained directly from Microsoft.

The rules for data which it understands are as follows:

- Each row must have the same number of comma-separated fields.
- Whitespace (space or tab) adjacent to a comma is ignored.
- Adjacent commas, or a comma at the start or end of a line (whitespace apart) indicates a null field.
- Lines are terminated by any sequence of carriage-return or newline characters ('\r' or '\n') (a corollary of this is that blank lines are ignored).
- Cells may be enclosed in double quotes; quoted values may contain linebreaks (or any other character); a double quote character within a quoted value is represented by two adjacent double quotes.
- The first line may be a header line containing column names rather than a row of data. Exactly the same syntactic rules are followed for such a row as for data rows.

Note that you can not use a "`#`" character (or anything else) to introduce "comment" lines.

Because the CSV format contains no metadata beyond column names, the handler is forced to guess the datatype of the values in each column. It does this by reading the whole file through once and guessing on the basis of what it has seen (though see the `maxSample` configuration option). This has the disadvantages:

- Sometimes it guesses a different type than what you want (e.g. 32-bit integer rather than 64-bit integer)
- It's slow to read.

This means that CSV is not generally recommended if you can use another format instead. If you're stuck with a large CSV file that's misbehaving or slow to use, one possibility is to turn it into an ECSV file file by adding some header lines by hand.

The `delimiter` option makes it possible to use non-comma characters to separate fields. Depending on the character used this may behave in surprising ways; in particular for space-separated fields the `ascii` format may be a better choice.

The handler behaviour may be modified by specifying one or more comma-separated name=value configuration options in parentheses after the handler name, e.g. "`csv(header=true,delimiter=|)`". The following options are available:

`header = true|false|null`
  Indicates whether the input CSV file contains the optional one-line header giving column names. Options are:
  - `true`: the first line is a header line containing column names
  - `false`: all lines are data lines, and column names will be assigned automatically
  - `null`: a guess will be made about whether the first line is a header or not depending on what it looks like
  The default value is `null` (auto-determination). This usually works OK, but can get into trouble if all the columns look like string values. (Default: `null`)
`delimiter = <char>|0xNN`
  Field delimiter character, by default a comma. Permitted values are a single character like "`|`", a hexadecimal character code like "`0x7C`", or one of the names "`comma`", "`space`" or "`tab`". Some choices of delimiter, for instance whitespace characters, might not work well or might behave in surprising ways. (Default: `,`)
`maxSample = <int>`
  Controls how many rows of the input file are sampled to determine column datatypes. This file format provides no header information about column type, so the handler has to look at the column data to see what type of value appears to be present in each column, before even starting to read the data in. By default it goes through the whole table when doing this, which can be time-consuming for large tables. If this value is set, it limits the number of rows that are sampled in this data characterisation pass, which can reduce read time substantially. However, if values near the end of the table differ in apparent type from those near the start, it can also result in getting the datatypes wrong. (Default: `0`)
`notypes = <type>[;<type>...]`
  Specifies a semicolon-separated list of names for datatypes that will not appear in the columns of the table as read. Type names that can be excluded are `blank`, `boolean`, `short`, `int`, `long`, `float`, `double`, `date`, `hms` and `dms`. So if you want to make sure that all integer and floating-point columns are 64-bit (i.e. `long` and `double` respectively) you can set this value to "`short;int;float`".
`encoding = ASCII|UTF-8|UTF-16|...`
  Specifies the character encoding used to interpret the input file. (Default: `UTF-8`)

This format cannot be automatically identified by its content, so in general it is necessary to specify that a table is in CSV format when reading it. However, if the input file has the extension "`.csv`" (case insensitive) an attempt will be made to read it using this format.

An example looks like this:

    RECNO,SPECIES,NAME,LEGS,HEIGHT,MAMMAL
    1,pig,Pigling Bland,4,0.8,true
    2,cow,Daisy,4,2.0,true
    3,goldfish,Dobbin,,0.05,false
    4,ant,,6,0.001,false
    5,ant,,6,0.001,false
    6,queen ant,Ma'am,6,0.002,false
    7,human,Mark,2,1.8,true

See also ECSV as a format which is similar and capable of storing more metadata.

Next Previous Up Contents
Next: ecsv
Up: Input Formats
Previous: cdf

STILTS - Starlink Tables Infrastructure Library Tool Set
Starlink User Note256
STILTS web page: http://www.starlink.ac.uk/stilts/
Author email: m.b.taylor@bristol.ac.uk
Mailing list: topcat-user@jiscmail.ac.uk