TOPCAT Frequently Asked Questions


Software version: 4.9-1
FAQ version: $Id: faq.xml,v 1.130 2024/03/01 14:54:33 mbt Exp $
TOPCAT web page: http://www.starlink.ac.uk/topcat/
TOPCAT manual: SUN/253
TOPCAT mailing lists: topcat-user@bristol.ac.uk, topcat-announce@bristol.ac.uk
Author email: m.b.taylor@bristol.ac.uk

Contents



1 General

Q1.1 What is TOPCAT?

TOPCAT is an interactive graphical tool for analysis and manipulation of tabular data. Its manifesto is:

Does what you want with tables


Q1.2 What is a table?

A table, for TOPCAT's purposes, is a data structure with columns, so that all the values in a given column are of the same kind. The types of value in each column are usually, but not necessarily, numbers, strings, or arrays of these. The most important example in astronomy of a table is an object or source catalogue. Two- or multi-dimensional arrays, such as images or data cubes are not tables - TOPCAT will not be able to help you with these.


Q1.3 What are TOPCAT's strengths?

Scalability
Considerable effort has been put into allowing TOPCAT to work with fairly large datasets, even on modestly-specified machines. It can usually cope well with files tens or hundreds of megabytes in size (a few million rows and/or a few hundred columns).
Ease of Use
Although full documentation is provided, it is hoped that users can work out how to use a lot of the available functionality by playing around with the toolbar buttons and generally clicking things and seeing what happens.
Crossmatching
The crossmatching facilities are very flexible and pretty fast. They work well up to around between a million and ten million rows, depending on the details.
Interactive Graphics
You can do a number of different 1-, 2- and 3-dimensional plots of one or many data sets, and it's easy to see where single points or data selections in one plot appear in other plots.
Expression Language
A powerful and flexible expression language allows calculations involving table values. A rich set of general and astronomical functions is available. This can be extended by the user (with a little bit of java programming) if required.


Q1.4 What are its weaknesses?

Some of these capabilities (limited scriptability, large tables) are however provided by TOPCAT's sister package, STILTS.


Q1.5 Is TOPCAT just for astronomy?

The large majority of TOPCAT's user base is currently in astronomy. However many of its facilities are not astronomy-specific; it is quite suitable for use in other contexts, and has been so used. I'm very open to approaches from other scientific domains who are interested in using it, and possibly customising it accordingly, but the amount of effort to publicise the software effectively in another user community would be considerable, and I'm not currently doing that.


Q1.6 Is TOPCAT just for the Virtual Observatory?

No. In many ways, TOPCAT is a traditional data analysis tool, in that it will perform most of its functions quite happily with no network connection. It does however provide a number of VO-friendly facilities.


Q1.7 Why is TOPCAT written in Java?

Several reasons, some of the important ones being:


Q1.8 But isn't Java slow?

No.


Q1.9 Gosh what a terrific bit of software. How can I express my appreciation?

Why thank you. The best thing you can do is to help publicise its use by telling your colleagues how you've found it useful. Acknowledging its use in published papers and talks is also very helpful. Telling me what you've used it for can (a) give me a warm fuzzy feeling inside and (b) come in useful when it comes to justifying development effort, e.g. to funding agencies.


Q1.10 How should I acknowledge TOPCAT?

It is much appreciated if you can acknowledge TOPCAT in your publications and presentations. The preferred method is to cite one or both of these papers:

though note that because the software continues to evolve, the papers do not document up to date versions.

Alternatively, you can cite one or both of the following permanent URLs:

I keep an eye on the ADS links above for citations of these papers. If you publish anything that cites or otherwise uses this software outside of the normal astronomy literature, I'd be most interested to hear about it.


Q1.11 What other papers are there relating to TOPCAT?

The following papers relate to TOPCAT and STILTS development and functionality:

2005ASPC..347...29T TOPCAT & STIL: Starlink Table/VOTable Processing Software
Initial publication on TOPCAT; preferred citation for TOPCAT, but content is quite outdated. (Conference paper, 4pp)
2006ASPC..351..666T STILTS - A Package for Command-Line Processing of Tabular Data
Initial publication on STILTS; preferred citation for STILTS, but content is quite outdated. (Conference paper, 4pp)
2008ASPC..394..422T Column-Oriented Table Access Using STIL: Fast Analysis of Very Large Tables
Discusses row vs. column-oriented table arrangements and memory mapping. (Conference paper, 4pp)
2009ASPC..411..510T STILTS Plotting Tools: Scriptable graphics generation from small and large tables
Discusses old-style STILTS plotting commands; mostly outdated. (Conference paper, 4pp)
2014ASPC..485..257T Visualising Large Datasets in TOPCAT v4
New-style plots, including hybrid scatter plot/density map. (Conference paper, 4pp)
2015ASPC..495..177T External Use of TOPCAT's Plotting Library
New-style STILTS plotting commands and use of STILTS plots from external applications. (Conference paper, 4pp)
2017ASPC..512..589T TOPCAT's TAP Client
TAP client, including service discovery and GUI. (Conference paper, 4pp)
10.3390/informatics4030018 TOPCAT: Desktop Exploration of Tabular Data for Astronomy and Beyond
Detailed discussion of TOPCAT as a tool for exploratory visualisation; written for non-astronomy readers. (Refereed paper for Informatics journal, 18pp)
2020ASPC..522...67T TOPCAT: Working with Data and Working with Users
Overview of approach to the technical and human issues that have been tackled during TOPCAT development history. (Invited conference paper, 10pp)
2019ASPC..523...43T TOPCAT and Gaia
Discusses some of the features of TOPCAT introduced, or relevant, for data from Gaia, especially DR2. (Conference paper, 4pp)
2022ASPC..532....3T TOPCAT Visualisation Over the Web
Explains how to provide server-side TOPCAT-like interactive graphics over HTML (Conference paper, 4pp)
2021arXiv211208003T Taplint, the TAP Service Validator
Presents taplint (Conference paper, 4pp)
2022arXiv221116913T Improving Performance in Java: TOPCAT and STILTS
Presents performance improvement work between TOPCAT versions 4.6-3 and 4.8-7 (Coference paper, 4pp)


Q1.12 Where can I find old/new versions?

The most recent public release is available from the web page. Previous versions can be found in the release archive. New releases are notified on both the mailing lists.

There will sometimes be a bleeding-edge version in the pre-release directory - this may contain fixes and new features not yet available in the public release which you can experiment with at your own risk.


Q1.13 How do I get the source code to look at?

The source code for TOPCAT and friends is available in the Starlink/starjava repository on Github.

If you just want all the source code just to look at, an alternative may be to grab the topcat_src.zip file from the web page, which contains all the java source code for TOPCAT, STILTS and the uk.ac.starlink.* classes it uses. Third party library code is not included.

See also Q1.14.


Q1.14 How can I compile TOPCAT from source code?

The proper way to build the application from scratch is to check out the starjava build system from the starjava git repository. No password is required for read access. Do something like:

   % git clone git://github.com/Starlink/starjava.git starjava/source
You can then read the file starjava/source/README for instructions on what to do next. Your mileage may vary somewhat, but the sequence will be along these lines:
   % cd starjava/source
   % setenv STAR_JAVA /usr/java/jdk1.8.0/bin/java .. or wherever
   % cd ant
   % bin/ant build install
   % cd ..
   % ../bin/ant build
   % ../bin/ant install
To build the standalone TOPCAT jar files topcat-full.jar and topcat-lite.jar you can do:
   % cd topcat
   % ../../bin/ant build-standalone install
Your JDK may be any version >=1.6, and must have JAI (Java Advanced Imaging) installed. Note that this process builds the whole starjava suite of software, which includes more than just TOPCAT.

Since the standalone jar file topcat-full.jar contains all the library classes, a quick and dirty alternative is to compile from the sources in the topcat_src.zip file from the web page by putting topcat-full.jar on your classpath at compile time.


Q1.15 I think I've found a bug.

If there's something wrong which is not noted in Section 4, please mail me and let me know. I may be able to give you a workaround or updated copy, and more importantly it gives me a chance to fix it in future versions.


Q1.16 Why doesn't TOPCAT display interactive sky images/show movies/speak my weight?

I have a long list of features that I'd like to add eventually, some of which will probably get done and some of which will probably not. However, I'm always interested in new suggestions. The existing feature set is very largely due to suggestions and requests from existing users.

Having said that, there are some things which I am unlikely to add, in particular things which are not basically to do with manipulation of tables, and things which would require non-Java code to be introduced.


Q1.17 How can I find out when new versions are released?

Subscribe to the topcat-announce mailing list - see list information here.


Q1.18 What license is TOPCAT released under?

The application is released under the GNU General Public Licence (GPL). However, many of the underlying libraries are available under more permissive licenses such as the LGPL. There is some more information in the STILTS LICENSE.txt file.


Q1.19 What related software is there?

The following items are related to TOPCAT:

STIL
The Starlink Tables Infrastructure Library is a public API which provides the table I/O, manipulation and crossmatching functions used by TOPCAT. It is available for use by other software projects which need to do table (including VOTable) manipulation.
STILTS
The STIL Tool Set is a package of command-line utilities which provides many of the same facilities as TOPCAT, but without a graphical user interface. It's suitable for server-side use, can be scripted, and under certain circumstances can cope with larger tables than TOPCAT (since some operations can work in streaming mode).


Q1.20 Who has developed TOPCAT?

Most of the code has been written by Mark Taylor. Much help has been supplied by other members of various user and developer communities, and some external libraries are used. More details are available in the Acknowledgements section of the manual.


Q1.21 What project(s) support(s) TOPCAT?

TOPCAT has been supported by a number of grants and projects over its lifetime, nearly all ultimately funded by UK or European astronomy research organisations. In chronological order:

(some of these grants cover projects additional to TOPCAT/STILTS/STIL, but included support for that work).

The work has mostly been done in the Astrophysics Group of the School of Physics at the University of Bristol in the UK (though some was done in various senses at the Institute of Astronomy in Cambridge and ARI in Heidelberg).


Q1.22 What are its future prospects?

TOPCAT has no assured long term funding, but has managed for the last decade or two to hop from one funding source to another without falling between the cracks. At time of writing things are looking positive for the near future: funding is secured until mid 2025 by grants from STFC, the UK's main funding body for astronomical research.

If because of terminated funding or for some other reason I cease work on it, another project or individual could take on some level of maintenance, support and/or development in my absence. I like to think that the code is sufficiently well organised and documented that this is a realistic possibility. Feel free to take a look to see whether you agree.



2 Downloading and Starting TOPCAT

Q2.1 What do I need to run TOPCAT?

You need a Java SE (Java 2 Standard Edition) runtime environment version 8 or above (see Q2.2) and, usually, one of the TOPCAT jar files (topcat-lite.jar or topcat-full.jar, though it's available in other forms too). Since Java is available for almost all modern machines, TOPCAT will run on almost any computer with a screen, for instance Linux, Solaris, Mac OSX and MS Windows.

If you are using the MacOS DMG file you don't need anything else, since that bundles its own Java runtime.

TOPCAT will not run on a phone or iPad, or within a web browser.


Q2.2 What version of Java do I need?

Current versions of TOPCAT/STILTS require Java SE 8 or later. This Java version has been available since 2014, so it is likely to be installed already, and if not should be available, for any but the most ancient platforms.

Development and testing is done using Oracle Java, but OpenJDK or whatever else is easy to install on your system is probably OK, as long as it's Java 8 or later.

If you are unable or unwilling to upgrade your Java installation, you can use an older TOPCAT version: v4.6-3 (2019) or earlier will run on Java 6, v4.3-2 (2015) or earlier will run on Java 5, and v3.5-2 (2010) or earlier will run on Java 4.


Q2.3 I get an error like "java.lang.UnsupportedClassVersionError"

This means that the Java version you have installed is too old to run TOPCAT. Since TOPCAT v4.7 (STILTS v3.2) you will need at least Java 8 ( Q2.1). Upgrade your java to a more recent version Q2.2 and try again.


Q2.4 What's the best way to download and run TOPCAT?

The recommended method is to grab http://www.starlink.ac.uk/topcat/topcat-full.jar from the TOPCAT web page. On a Un*x system you can also get (and chmod +x) the http://www.starlink.ac.uk/topcat/topcat shell script and put it in the same directory. Then just invoke "topcat" (if you have the shell script)" or "java -jar topcat-full.jar" and you're off.

If you are tight for space, you can use topcat-lite.jar instead - it is missing only a few less important facilities.

If you have a Mac instead, you may prefer to get the topcat-all.dmg file instead (which bundles Java so does not need an existing Java installation), but note items Q4.4 and Q2.6 in this case.

If you are running an Ubuntu-derived Linux distribution, see the question about Debian Astro.


Q2.5 What about Debian Astro?

Since early 2017, the TOPCAT and STILTS packages have been included in the Debian Astro suite (thanks to the hard work of Ole Streicher). That means that in due course those packages will be made available in Debian-derived Linux distributions such as Ubuntu. In such distributions, you should be able to just do

  # apt install topcat stilts
to get both packages. These should appear from Ubuntu 17.10 and Debian 10. This is not however guaranteed to give you the most recent versions.

Note however that these versions are not identical to the "official" releases available from http://www.starlink.ac.uk/topcat/. They should work OK, but the version may lag behind the official release, some of the external library versions are a bit different, a few of the more obscure features may be missing, etc. Version skew apart, differences in behaviour between the two are possible, but not expected to be significant.


Q2.6 How do I set command-line flags from Mac OS X?

If you have downloaded the topcat.dmg file from the web page and installed it on your Mac, you will be able to run it in a Mac-like way by pointing and clicking. However, this does not permit you to specify important command-line flags such as -Xmx8192M, not to mention specifying tables on the command line to load at startup etc (see the manual for more information on command-line arguments).

To set command-line flags for use with the DMG file, since TOPCAT v4.9, you can edit the file

    /Volumes/topcat/TOPCAT.app/Contents/vmoptions.txt
Note that in earlier versions the advice was different (also more complicated and less reliable).

Alternatively, run the program from the command line. Depending where you have installed the .dmg file, the topcat script should be present at a location like:

   /Volumes/topcat/TOPCAT.app/Contents/Resources/app/topcat
so invoking the script at that location or putting that directory on your path and invoking topcat with the relevant flags should do the trick. You can run STILTS in a similar way.

Failing that, you should be able to find the topcat-extra.jar file somewhere. In that case, something like this ought to work:

   java -Xmx8192M -jar /Volumes/topcat/TOPCAT.app/Contents/Resources/app/topcat-extra.jar

You don't have to use the DMG file of course; you can just download the topcat-full.jar (or topcat-extra.jar) jar file and optionally the topcat script directly and use them in the same way that other Un*x users do. This may be a good idea if you have trouble getting the DMG file working.

Thanks to Frank Primini, Juan de Dios Santander Vela, Klaus Rübke, Nicholas Seymour and Ming Yang for help with this question.


Q2.7 I have a Retina hi-res display on my Mac and TOPCAT looks blurry

Thanks to Christophe Deil for supplying this answer.

TOPCAT does work on Macbook Pro with Retina display, although some plots are drawn as bitmaps by the application and then scaled by 2x by the java implementation before being rendered to the Retina screen, resulting in unsharp image and text. Everything else (text, buttons, other plots) will look Retina-sharp if you start TOPCAT in high-resolution mode.

From the command line TOPCAT will start in high-resolution mode automatically. To make TOPCAT start in high-resolution mode when double-clicked on the icon in Applications or the Dock, you have to manually declare TOPCAT to be high-resolution capable. Add

   <key>NSHighResolutionCapable</key>
   <true/>
toward the end of TOPCAT's Info.plist file with the text editor of your choice, right before the closing </dict></plist> lines. See the previous question more information about finding and editing the Info.plist.

Finally you can check by right-clicking on TOPCAT and selecting "Get Info" in Applications, you should see the "Open in Low Resolution" checkbox disabled. A simpler check is to just start TOPCAT, if low-resolution mode is enabled you will see it right away because then everything in TOPCAT (text, buttons, plots) will look unsharp.

I may decide to change the default in the future, but for now there seems a question about whether setting hi-res might cause problems. Any feedback about positive or negative experiences with this is very welcome.


Q2.8 How do I set up TOPCAT to run on Microsoft Windows?

The following instructions have kindly been supplied by Clive Page for use with Windows XP:

  1. Download Java (the JRE) from Sun's website as explained elsewhere in the TOPCAT instructions. The remaining instructions assume that the java.exe file is installed as C:\WINDOWS\System32\java.exe and they should be modified in the obvious way if this is not the case.
  2. Download either topcat-lite.jar or topcat-full.jar to a suitable location; the following instructions assume topcat-full.jar and that you put it in your own "My Documents" folder, but any folder will do.
  3. With the cursor in any blank area of your desktop, right-click the mouse and select New|Shortcut. A window appears asking for the location of this item: what is needed is of the following form, assuming that you put the topcat-full.jar in your own "My Documents" folder:
        C:\WINDOWS\System32\java.exe -jar "C:\Documents and Settings\your name\My Documents\topcat-full.jar"
        
    If you don't get this right first time it can be altered later, using the Properties window as described below. Note that you need quotation marks around the string after the -jar switch if it contains spaces, as it does in the example above. If you want to provide a switch for example to specify more memory, such as -Xmx100M for 100 Megabytes, then this switch should be inserted before the -jar switch. Click NEXT, then you get prompted for a name: enter "Topcat" or whatever seems appropriate. Click FINISH and a new icon should appear on your desktop (with a Java logo).
  4. Right click on this icon and select Properties. You will see a box labelled Start in which will be filled by default with the location of your java.exe file. You will probably find it convenient to alter this to some other folder, such as the one where your FITS files are generally located, or where your browser downloads them from the web. After editing it, finish by clicking Apply and OK.
When you click on the Topcat icon, it puts up first a text window, and then after a short interval the usual main Topcat window appears on top. You can minimise this text window, but do not delete it as this also deletes the Topcat window.

Christopher Bishop adds some notes relating to Vista and Firefox downloads:

If you are downloading Topcat using Firefox on Vista it is important that you make sure that you set the Firefox download to the folder you want to save it in. If you don't then the topcat-full.jar will end up by default in a temporary file. If you then follow steps 3 and 4 above a Topcat icon will be created, but you may find Vista will have deleted your download in the temp file. At least it did in mine! The solution I found was to go to Firefox Tools - Options - Main and then look at the downloads window. In the downloads window there is a box that enables you to specify where you want the download to go. So I put C:\user\yourname\documents (I would check in your documents folder that topcat-full.jar is really there and that Vista hasn't deleted it!)


Q2.9 Can I run it using Java WebStart?

WebStart is a Java technology which enables one-click download, installation, updating and invocation of Java applications over the web. It can be quite convenient, but it sometimes doesn't work properly and in more recent versions of Java it is deprecated (Java 9+) or no longer provided (Java 11+).

Since early versions of TOPCAT, it has been possible to run it using WebStart, if you have WebStart installed, from a URL like http://www.starlink.ac.uk/topcat/topcat-full.jnlp. But newer Java installations don't support WebStart and enabling some functionality requires signing the topcat jar files, which is complicated and expensive.

Therefore from TOPCAT 4.8-8 (April 2023) WebStart invocation is provisionally deprecated; the jar file is no longer signed, so although you may still be able to run it from WebStart some (maybe many) things won't work. You are therefore strongly advised to avoid WebStart and run TOPCAT in one of the other ways; that's probably the Standalone Jar file topcat-full.jar if you don't have a Mac, or the DMG file topcat-all.dmg if you do.

If people complain I may reinstate this for a while, but WebStart is on the way out, so it's likely to disappear eventually.


Q2.10 Can TOPCAT be run as an applet?

No. It is possible that with some changes to the code it could be done, but I don't have any plans to do that. Applets were once popular but have long been out of fashion, and TOPCAT anyway seems a bit heavyweight to run inside a web browser (for instance it tends to need a lot of windows at once).


Q2.11 It doesn't seem to work!

Installing a java application such as TOPCAT is usually pretty straightforward, and there's not too much that can go wrong - at least compared to applications written in compiled languages such as C. If it's not starting, or failing to complete even very simple actions, the most likely explanation is that the version of Java you are using is broken or incomplete. You should try to install or reinstall Java for your system; see Q2.1 and Q2.2.

If you have a Mac, see also Q4.4.

If neither of those seems to be the problem, please mail me and ask for help.



3 Using TOPCAT

Q3.1 What table format should I use?

The most efficient table format to use is FITS (either basic or "FITS-plus" - see Q3.2) specifically FITS BINTABLE extensions in uncompressed files on disk. If your tables are large, using FITS rather than some other format will both save memory (enabling you to work with larger datasets) and speed up table loading. For very large, and especially very wide tables (bigger than available RAM), the colfits format may be a better choice. If your data is initially in some other format (e.g. an ASCII-based one) it is often a good idea to convert once to a FITS-based format and subsequently use the FITS file within TOPCAT.

If your tables are small then the efficiency doesn't matter so much. VOTable and FITS-plus, and to a lesser extent basic FITS, can store rich metadata (units, UCDs, column descriptions etc) which may be something you want.

If you want to do processing of a table in some other language such as Python, IDL or Perl, a number of other formats are available. ECSV is good for interoperating with Python, Feather is efficient for interoperating with R, and CSV (Comma-Separated Values), while inefficient and having various problems, is generally a workable lowest-common-denominator solution.


Q3.2 What's FITS-plus?

TOPCAT can read and write two variants of FITS binary tables, "FITS-basic" and "FITS-plus" (these are just terms used by STIL/TOPCAT to distinguish the two variants). Both of these are perfectly legal FITS tables, but TOPCAT squirrels away some extra metadata in the primary HDU when using FITS-plus. In most cases, TOPCAT writes FITS-plus by default, and reads either. Nearly all of the time, you shouldn't need to worry about the difference.

There is more detailed information in the manual; see also SUN/252.


Q3.3 How do I read Multi-Extension FITS files?

If you load a FITS file, any table extensions it contains (BINTABLE or TABLE extensions) will be loaded as separate tables by default.


Q3.4 How do I read multi-table VOTable documents?

If you load a VOTable document, any tables it contains (<TABLE> elements) will be loaded as separate tables by default.


Q3.5 What size tables can TOPCAT cope with?

This depends a bit on one thing and another, but as a rule of thumb a few million rows and a few tens of columns should be quite usable while maintaining quite good performance. If you're having trouble with much smaller tables than this, there is almost certainly something you can do about it. For more details see Q3.6.

Larger tables are not out of the question; there are reports of interactive use with 300 million rows, and 20 thousand columns (not at the same time).


Q3.6 Help! I've run out of memory.

A lot of effort has gone into making TOPCAT scalable. Some data sets may simply be too large to process, but up to a few million rows it should be possible to use the program with good performance and without running out of resources. You may have to take some steps to ensure that this is so however. Some good tips are:

See the manual for more discussion.

Some operations which would run out of memory in TOPCAT can be done using STILTS, since it can stream data in some circumstances.


Q3.7 Can I save a Row Subset?

The Session save option in the Save Window can be used to save your entire session (for some or all of the loaded tables). Unlike the Current Table save option, this saves things like lists of subsets, hidden columns, etc, in addition to the usual table data and metadata. It can be used to save a single table or several in the same file. To reload the session data, simply reload the saved file into TOPCAT like a normal table.

An alternative way, for instance if you want to communicate the subset contents to a non-TOPCAT application, is to create a new boolean column representing the subset(s) to save, and save the table with that column. You could create the new column from the Subset Window using the Subsets|To Column menu item or equivalent toolbar button; this column then contained true/false according to whether each row is/is not in the subset in question. When the table was reloaded into TOPCAT, new subsets will be created for any such boolean columns.


Q3.8 TOPCAT won't read my VOTable

It is possible that there is a bug in TOPCAT, but it's much more likely that there is an error in the VOTable. The best thing to do in this case is to diagnose the problem by using the STILTS votlint tool. Do not be afraid, this is very easy to do, something like:

   % stilts votlint foo.xml
or
   % topcat -stilts votlint foo.xml
This will print WARNING or ERROR messages to the terminal describing questionable or illegal aspects of the file. You may be able to see what's wrong, and possibly fix it with careful use of a text editor from this information. Either way, if there are errors it would be a good idea to contact the originator of the VOTable with the votlint output and tell them that they are generating broken VOTables.


Q3.9 Where has my VOTable metadata gone?

TOPCAT's data access is format-neutral - it has its own model of what constitutes table data and metadata rather than being tied to a particular data format. This is good in that it means it can work with tables in various different formats. However, it also means that data/metadata structure which is very specific to a particular data format may not be accommodated. Things like VOTable <GROUP> and <RESOURCE> elements fall into this category, except in some specific cases (e.g. where they related to TIMESYS/COOSYS elements).

To put it another way, both load and save operations in TOPCAT result in a sort of format translation (to and from TOPCAT's internal data model). Some things can get lost or changed in both of these stages, so unfortunately loading a table into TOPCAT and saving it without making any explicit changes can result in loss of data.

To put it a third way, TOPCAT is (amongst other things) a table editor, but if you think of it as a VOTable editor you may be disappointed.


Q3.10 Can I indicate missing upper/lower limits when plotting error bars?

There is no specific facility for displaying missing limits, but you can probably generate a plot that looks like you want it to. You can represent missing limits using the error bar style which resembles little arrow heads. Set up two subsets in the plot, one for points with error limits and one for points without. You can set up two data sets in the plot with the same point data but different error data, one containing the subset of points with limits and one containing the points without. For the points without limits, provide fake error values and use a suitable error plotting style, such as the error bars with the little arrow heads.


Q3.11 How can I join two tables with the same row numbers side by side?

TOPCAT's crossmatching facilities are fairly smart and flexible. So much so that they can do stupid, easy things as well as clever, hard ones. If you have tables that want joining together without reordering the rows, choose the Exact Value algorithm and use the magic value "$0" for the Matched Value Column for each input table. $0 evaluates to the row index in each case, so that all the row 1's will be joined together, as will all the row 2's etc.

In STILTS it's easier, there is a command that does just this: tjoin.


Q3.12 The crossmatching is giving the wrong results!

The crossmatching code in TOPCAT/STILTS is quite well tested, and reports of actual bugs are rare. However, I take the correctness seriously, and if you think you've found an anomaly, please report it.

The following bugs affecting correctness of crossmatching have been reported in recent years:

  1. A bug was reported on 23 May 2011 by Yannick Roehlly concerning sky matches near the line RA=0. If all the points in one table are on one side of this line, but some are within an error radius of it, the matcher may fail to identify the corresponding matches on the other side of the line. A related bug means that matches may be missed if tables in a match do not use a single normalised range for RA (e.g. 0<=RA<=360). This bug is fixed at TOPCAT v3.9 and STILTS v2.3-1.
  2. A minor bug was reported by Martin Hennemann in Feb 2012 that could result in some unwanted single-table output rows in multi-table group-mode matches; this was fairly harmless since it was obvious from the output table what was wrong. This bug was fixed at TOPCAT v4.0b and STILTS v2.5-1.
  3. A more serious bug was reported by Grant Kennedy in Oct 2015. This related to the way in which the matcher attempts to narrow the overlap region between tables prior to doing the actual crossmatching (so it can avoid doing unnecessary work that can't lead to actual associations). For match types that involved per-row (rather than fixed) match errors, if the scale of errors differed, especially differed significantly, between the tables being matched, this could result in missed associations near the edge of the coverage region of some tables (those with smaller maximum errors). It affected only these STILTS/TOPCAT matchers: This bug is fixed in TOPCAT v4.3-2 and STILTS v3.0-6.
  4. In September 2017 Alexey Mints reported a bug in the healpixRingIndex function, which resulted from a bug in the vect2pix_ring function of the underlying PixTools HEALPix library. This function is used during sky crossmatching, so seems possible that it could have resulted in some missed associations. If so, the problems would have been for associations with a point on each side of the longitude=0 line in the equatorial half of the celestial sphere. Tests have failed to show up any actual match failures, so it looks like this did not affect match results, but it's hard to say for sure that it could not have happened. This bug is fixed in any case at TOPCAT v4.5 and STILTS v3.1-1.
  5. In March 2023 Georgios Vasilopoulos identified instabilities when using the (symmetric) Best pair matching mode; it was not actually symmetric in the input tables, so that STILTS v3.4-6/TOPCAT v4.8-6 and STILTS v3.4-7/TOPCAT v4.8-7 might have given different answers in crowded fields. Using Best in crowded fields is a bad idea anyway, and it's not obvious that either of the results it could give was better or worse than the other, but this behaviour has been fixed so that versions starting from STILTS v3.4-8/TOPCAT v4.8-8 will act in a more predictable way.
  6. In August 2023, Claire Greenwell reported that the match subset created in some matching modes by TOPCAT's Pair Match window (the same issue affected the match<N> subsets created by the Multiple Match window) contained completely the wrong rows. This affected TOPCAT v4.8-4 to v4.8-8, but is fixed in later versions. The rest of the matching was unaffected (the output tables contained the right row and column content) and it did not affect STILTS.

No other crossmatching correctness bug has been reported in recent years.

One thing which can be confusing is matching objects in crowded regions, since in crowded regions (where the typical distance between positions is similar to or less than the match radius) the results depend strongly on the details of the matching criteria.

Note in particular that TOPCAT's "Best Match Symmetric" mode is symmetric between tables. If you select "Best Match Symmetric" for the Match Selection toggle (or equivalently "find=best" in STILTS tmatch* tasks), then each row from either table will only appear a maximum of once in the output table. That means if object A in table 1 is a match for object B in table 2, it cannot be a match for any other objects in table 2. This may lead to fewer matches than you were expecting.

If what you want to do is to find, for each object in table 1, the closest match in table 2, you should use one of the asymmetric options: "Best match for each Table 1 row" (or "Table 2 row") in TOPCAT's Match Selection chooser, or "find=best1 (or "best2") in STILTS. Note that these options were only introduced in TOPCAT v3.9/STILTS v2.4, prior to that only symmetric best and all options were available. So, if you have an earlier version and want to do this, you should upgrade.

If you've got doubts about the correctness of a crossmatch, it's a very good idea to plot the results to see whether it's doing what you think or hope it is. Since v4.1 this is usually fairly easy; see the manual.


Q3.13 Can I crossmatch more than 2 tables at once?

Yes. The multi-table crossmatch options are available from the Joins menu in the control menu. They don't appear on the toolbar just because they would take up too much space. Currently a 4-way match is the most you can do at once. But that's only because nobody has asked me for more. If you want the option to match more than that, let me know and I will add higher-numbered options.

Somewhat more flexible multi-table crossmatches are available using the STILTS tmatchn command. Equivalent functionality will be added to TOPCAT at some point.


Q3.14 Can I concatenate more than 2 tables at once?

No. You can do this using the STILTS tcat or tcatn commands however.


Q3.15 Can TOPCAT read SExtractor output?

It cannot read the ASCII files that SExtractor outputs by default. If you want to use TOPCAT with the output from SExtractor, set it to write its output as FITS tables. You can do this by setting SExtractor's CATALOG_TYPE parameter to "FITS_1.0".


Q3.16 Can I script TOPCAT?

No. The only ways you can control TOPCAT itself are

  1. by giving arguments (including tables to load at startup) on the command line
  2. by pointing and clicking
  3. using SAMP (or PLASTIC) messages from other applications

However TOPCAT's sister application STILTS is a set of command-line tools which provide much of the same functionality as TOPCAT. You can write scripts using these applications if you want to automate the kind of table processing that TOPCAT can do. For generating STILTS plots corresponding to TOPCAT plots, the Stilts Control in TOPCAT's plot windows may be useful.


Q3.17 Can I save/restore the state of a TOPCAT session?

Since version 3.6 (August 2010), yes you can. Use the Session option in the Save Window to save, and load the table that outputs in the normal way to restore.

Note however that this saves (most of) per-table state, and not everything, so for instance currently displayed plots will be lost. This may be improved in a future release.


Q3.18 Can I see a log of the actions TOPCAT has performed during a session?

No, I'm afraid not.


Q3.19 Can TOPCAT talk to other tools/processes?

Yes, by using SAMP or PLASTIC.


Q3.20 What's SAMP and/or PLASTIC all about then?

SAMP is the Simple Application Messaging Protocol.

It allows different tools, usually running on the same desktop, to talk to each other, for instance by sending tables from one to another. This topic is discussed in more detail in the manual.

Briefly, if a SAMP "hub" is running then some of TOPCAT's controls allow you to send instructions or data to other tools. Usually these options are controlled from an Interop menu on the relevant window.

A SAMP hub may already be running as part of another application (such as Aladin). If it is not, TOPCAT usually starts one internally when it starts up.

PLASTIC is the PLatform for AStronomical Tool InterConnection; it was an earlier protocol that did effectively the same thing as SAMP, but is now more or less obsolete.


Q3.21 Can I calculate the difference between column values in adjacent rows?

The expression language used by TOPCAT works entirely within a single row, so doing things like adding a new column B whose value in row n is the difference between the values of another column A in row n and its value in row n+1 is not directly possible.

There is a hack which you can use to achieve this however. If you join (match) the table to itself so that the same column appears twice in the resulting table, but offset by one row, you can then do calculations involving both values. The join can be done in the Pair Match Window, by using the Exact Value match algorithm, using the same table for both inputs, with the Matched Value Column set to "$index" and "$index+1" respectively. The "$index" (or equivalently "$0") token is a magic column which evaluates to the row number in the table. You can then add a column B with an expression like "A_2-A_1" to get the difference. If required, you could then match the part of the table containing the new column back to the original table.

Yes this is a bit of an effort. Here is a STILTS command which effectively does what is described above:

   stilts tmatch2 in1=table.fits in2=table.fits \
                  matcher=exact values1='$index+1' values2='$index' join=1or2 \
                  ocmd='addcol b a_2-a_1'



4 Known Bugs and Issues (Current version)

Q4.1 I can't see my Documents/Downloads/etc folders on MacOS Catalina+

It seems that in Catalina (and later versions?) of MacOS, certain directories are effectively invisible from the various file choosers, e.g. the Filestore Browser used in the Load and Save windows.

This seems to be related to new security restrictions introduced in this version of MacOS; I think it might be to do with the way that TOPCAT is launched from the .dmg file (further reading for enthusiasts here and here). I currently can't work out how to solve it, but some workarounds are possible: from the Load and Save windows, there is the option to use the System Browser as an alternative to the Filestore Browser. If you do that, you should get the standard MacOS Chooser, which ought to be able to see all the directories. Alternatively, if you start topcat using

   java -jar topcat-full.jar
rather than by clicking on the icon, it seems to work OK.

Something else that might work is accessing System Preferences/Security & Privacy/Privacy and giving topcat Full Disk Access.

Apparently, if you use the System Browser to look at these folders once, then subsequent access with the Filestore Browser works as well, maybe even in different TOPCAT invocations; see this report.

Thanks to Davide Lena (SRON) Marina Kounkel (U.Michigan), and Jarle Brinchmann (Leiden) for help investigating this.


Q4.2 Overwriting mapped FITS file causes Java crash

If a FITS file has been mapped for reading (this is the normal way that uncompressed FITS files on disk are read) and it is overwritten, either by TOPCAT or by some other program, it can result in a crash of the Java Virtual Machine (ugly error message to the console and instant death of TOPCAT). TOPCAT will not normally perform such an overwrite, but it can't stop you doing it from outside the program.

The best thing is: don't overwrite FITS files currently in use. However, you can work around this by deleting the file before you overwrite it, or by loading it into TOPCAT using a file:-type URL, e.g "file://localhost/foo/bar/cat.fits" rather than "/foo/bar/cat.fits". This will load without mapping (and as a result will be much slower for large files).


Q4.3 Expressions containing self-references cause trouble

If an algebraic expression for a synthetic column or row subset contains a reference to itself via a $id identifier, the program may hang. It might recover - if you can edit the expression so it doesn't contain such a reference any more, do so. In version 4.6-3 and later, TOPCAT will usually prevent you from doing this.


Q4.4 On MacOS the .dmg file is reported "damaged" or "can't be opened"

On the Mountain Lion release (and later?) of Mac OS X, attempting to install the topcat-all.dmg file apparently can result in the message:

"TOPCAT is damaged and can't be opened, you should eject the disk image"
or maybe
"TOPCAT.app can't be opened because Apple can't check it for malicious software
or
"TOPCAT can't be opened because it is from an unidentified developer"

It seems that in Mountain Lion and later, security settings (a.k.a. GateKeeper) default to not letting you run anything from anywhere other than the Mac App Store or from signed applications. Unfortunately, this results in it reporting java-based non-signed applications as "damaged".

There are several workarounds for this (though that may be MacOS-version dependent). Try one of the following:

I think the underlying problem is that I'm not a trusted Apple developer, because I'm not keen to pay the non-negligable amount of money that costs. If anybody has other/better ways to fix it, advice is welcome.

Thanks to various people who have helped with answering this question, including Juan de Dios Santander Vela, Jarle Brinchmann, Marina Kounkel, Alexander Mayer and Andy Lawrence.


Q4.5 Tables are scrambled when scrolling

In some cases apparently the table viewer window and other JTables displayed can get their pixels scrambled when they are scrolled. This must be some kind of bug in Java or maybe some display drivers. It seems to have more than one possible cause.

If the scrambling happens when you scroll to the right, the issue may be the one described here; to fix it try running with one or other of these system properties set:


Q4.6 The text on my high-resolution screen is too small

Some users report that running TOPCAT on high-resolution (4k, HiDPI, ??) screens results in a GUI that comes out too small, e.g. including fonts that are hard to read. The details of this problem seem to depend on exactly what hardware and software you have. Here are some measures that might fix it:

(references: https://wiki.archlinux.org/index.php/HiDPI#Java_applications, https://developer.gnome.org/gtk3/stable/gtk-x11.html; should the GDK_DPI_SCALE be set in some circumstances as well?? )

If you have any experience on what does and doesn't work in different environments, please get in touch so I can improve the advice here.


Q4.7 The TOPCAT window is unexpectedly dark/black on my M1 Mac

Under some circumstances, the TOPCAT windows can apparently go black or enter some kind of dark mode and make it hard to see what's going on, when running on (new M1 architecture?) Macs. I don't know whether this is a single issue or several different ones, but here is some relevant information and things which have been found to fix it:

Thanks to Ben Maughan (Bristol) and Ariane Lançon (Strasbourg) for investigating this. If you have more/different/better information about what works, please get in touch.



5 Known Bugs and Issues (Older versions)

Q5.1 I can't find a resource in the NVO/VAO registry from the Cone/SIA/SSA/TAP window

Between mid-2010 and mid-2014 searching the NVO (a.k.a. VAO) registry at STScI did not work. This turned out to be the result of bugs in both TOPCAT and the STScI registry, as finally uncovered by the estimable Menelaus Perdikeas. In version 4.1 and later, these searches should work. In version 4.2 and later, the RegTAP protocol is used by default, and the problem ought not to arise.


Q5.2 It crashes immediately on startup, or when I open the Load window

There is a bug which may be seen by people who are set up to use Starlink applications as well as TOPCAT. It may provoke messages like:

   no such constant AST__KYCIR
or
   #
   # An unexpected error has been detected by HotSpot Virtual Machine:
   #
   #  SIGSEGV (0xb) at pc=0xf79af28b, pid=5153, tid=3888896928

You should be able to fix it by unsetting the LD_LIBRARY_PATH environment variable, for instance if you use a C-like shell:

   % unsetenv LD_LIBRARY_PATH
or for bash/sh:
   $ unset LD_LIBRARY_PATH
   $ export LD_LIBRARY_PATH
If you continue to have problems, please contact me and I'll try to provide more help.


Q5.3 3D error bars extending outside the box are invisible

In the Cartesian/Spherical 3D plots, if an error bar would extend outside of the box/sphere then it is simply not plotted, so that even the part which is within the box/sphere is not visible.


Q5.4 Scisoft 7.0 invocation script broken

TOPCAT is included in the ESO Scisoft software bundle. The "topcat" invocation script included with that bundle does not have the same functionality as the one documented in the manual, so for instance command-line arguments cannot be supplied to it. If it's not doing what you want, you can run using java -jar topcat-full.jar.

This problem should be fixed in Scisoft releases after 7.0.


Q5.5 I get an error about a SizingScrollPane on my Java 1.7 Mac

On some Macs running java 1.7 (Mountain Lion??), doing things like trying to open the table or statistics windows can give you a stack trace like:

   Exception in thread "AWT-EventQueue-0" java.lang.NullPointerException
           at uk.ac.starlink.topcat.SizingScrollPane.getConfig(SizingScrollPane.java:81)
           at uk.ac.starlink.topcat.SizingScrollPane.getPreferredSize(SizingScrollPane.java:40)
           ...
I can't tell if this is my fault or OSX's, but it is fixed in version 4.0 and later.


Q5.6 My plot axis labels come out backwards on OS X!

It seems that in recent versions (java 1.7?) of MacOS X, the text written sideways up the Y axis of plots sometimes comes out written backwards, e.g. it appears to read "EDUTINGAM_B" rather than "B_MAGNITUDE". In fact any text written non-horizontally is placed wrong (e.g. sky axis labels). I am pretty sure this is a Mac Java bug, which I hope will fix itself at some point (does anybody know how to report this kind of thing?).

The new-style plotting windows in v4.1 can work around this issue (I believe) which seems to go away if you use antialiased text. As well as getting the text the right way round it does look a bit nicer like this, but it is also perceptibly slower. TOPCAT attempts to determine if its running in MacOS, and if so it sets antialiasing on by default. However at version 4.1, it doesn't always guess right. You can turn antialiasing on/off by using the Antialias checkbox in the Grid tab of the Axes control in the various plot windows, so try that if your labels are backwards.

In version 4.2 and later, text antialiasing will be set using the Antialias text type in the Font selector instead. By then, it ought to be better at guessing OS, so you hopefully will not need to worry about it.


Q5.7 When I load an LDAC FITS file I can't see my data

Files in the LDAC FITS format are organised in an unusual way with the first (LDAC_IMHEAD) table extension containing a single-cell table where the cell's content is an array of FITS headers. This confuses TOPCAT into thinking that the file is in colfits format, which in turn means it ignores the second (LDAC_OBJECTS) table extension, and that's the one you probably want to look at.

A hack is present in TOPCAT v4.2-2 and later to avoid this, but there are various ways to work around it anyway. One is to select "FITS" (rather than "(auto)") as the input file format. Another is to use the Hierarchy Browser to examine the file and select the table you want. You can also select the correct table by name or HDU index as, e.g., "D1_r_sex.cat#LDAC_OBJECTS" or "D1_r_sex.cat#2", either on the command line or using the Position in file field at the bottom right of the Filestore Browser window.



6 Last Resorts

Q6.1 I still don't know...

Try consulting the manual. Or at least the relevant part - click the Help button in the window you're in for some context-specific documentation. The Find field in the Help Window is also good for locating the right part of the documentation.


Q6.2 It's not in/I'm too lazy to read the manual.

The best thing is to subscribe to, then send a message to, the topcat-user mailing list (if it's a question you think others might be interested in too) or to m.b.taylor@bristol.ac.uk (if it's quite specific). I am usually monitoring the mailing list, and you will probably get a reply quite soon.