TOPCAT Frequently Asked Questions


Software version: 4.1
FAQ version: $Id: faq.xml,v 1.61 2014/03/28 10:18:40 mbt Exp $
TOPCAT web page: http://www.starlink.ac.uk/topcat/
TOPCAT manual: SUN/253
TOPCAT mailing lists: topcat-user@bristol.ac.uk, topcat-announce@bristol.ac.uk
Author email: m.b.taylor@bristol.ac.uk

Contents



1 General

Q1.1 What is TOPCAT?

TOPCAT is an interactive graphical tool for analysis and manipulation of tabular data. Its manifesto is:

Does what you want with tables


Q1.2 What is a table?

A table, for TOPCAT's purposes, is a data structure with columns, so that all the values in a given column are of the same kind. The types of value in each column are usually, but not necessarily, numbers, strings, or arrays of these. The most important example in astronomy of a table is an object or source catalogue. Two- or multi-dimensional arrays, such as images or data cubes are not tables - TOPCAT will not be able to help you with these.


Q1.3 What are TOPCAT's strengths?

Scalability
Considerable effort has been put into allowing TOPCAT to work with fairly large datasets, even on modestly-specified machines. It can usually cope well with files tens or hundreds of megabytes in size (a few million rows and/or a few hundred columns).
Ease of Use
Although full documentation is provided, it is hoped that users can work out how to use a lot of the available functionality by playing around with the toolbar buttons and generally clicking things and seeing what happens.
Crossmatching
The crossmatching facilities are very flexible and pretty fast. They work well up to around between a million and ten million rows, depending on the details.
Interactive Graphics
You can do a number of different 1-, 2- and 3-dimensional plots of one or many data sets, and it's easy to see where single points or data selections in one plot appear in other plots.
Expression Language
A powerful and flexible expression language allows calculations involving table values. A rich set of general and astronomical functions is available. This can be extended by the user (with a little bit of java programming) if required.


Q1.4 What are its weaknesses?

Some of these capabilities (limited scriptability, large tables) are however provided by TOPCAT's sister package, STILTS.


Q1.5 Is TOPCAT just for astronomy?

The large majority of TOPCAT's user base is currently in astronomy. However many of its facilities are not astronomy-specific; it is quite suitable for use in other contexts, and has been so used. I'm very open to approaches from other scientific domains who are interested in using it, and possibly customising it accordingly, but the amount of effort to publicise the software effectively in another user community would be considerable, and I'm not currently doing that.


Q1.6 Is TOPCAT just for the Virtual Observatory?

No. In many ways, TOPCAT is a traditional data analysis tool, in that it will perform most of its functions quite happily with no network connection. It does however provide a number of VO-friendly facilities.


Q1.7 Why is TOPCAT written in Java?

Several reasons, some of the important ones being:


Q1.8 But isn't Java slow?

No.


Q1.9 Gosh what a terrific bit of software. How can I express my appreciation?

Why thank you. The best thing you can do is to help publicise its use by telling your colleagues how you've found it useful. Acknowledging its use in published papers and talks is also very helpful. Telling me what you've used it for can (a) give me a warm fuzzy feeling inside and (b) come in useful when it comes to justifying development effort, e.g. to funding agencies.


Q1.10 How should I acknowledge TOPCAT?

It is much appreciated if you can acknowledge TOPCAT in your publications and presentations. The preferred method is to cite one or more of these papers:

though note that because the software continues to evolve, the papers do not document up to date versions.

Alternatively, you can cite one or both of the following permanent URLs:

If you publish papers or preprints acknowledging this software, it would be both interesting and helpful for me if you could let me know where to find them.


Q1.11 Where can I find old/new versions?

The most recent public release is available from the web page. Previous versions can be found in the release archive.

There will sometimes be a bleeding-edge version in the pre-release directory - this may contain fixes and new features not yet available in the public release which you can experiment with at your own risk.


Q1.12 How do I get the source code to look at?

If you want all the source code just to look at, the most convenient thing may be to grab the topcat_src.zip file from the web page. This contains all the java source code for TOPCAT, STILTS and the uk.ac.starlink.* classes it uses. Third party library code is not included.

Another possibility is to browse or download the source code repository, currently Starlink/starjava on github.

See also Q1.13.


Q1.13 How can I compile TOPCAT from source code?

Since the standalone jar file topcat-full.jar contains all the library classes, a quick and dirty way is to compile from the sources in the topcat_src.zip file from the web page by putting topcat-full.jar on your classpath at compile time.

The proper way to build the application from scratch is to check out the starjava build system from the starjava git repository. No password is required for read access. Do something like:

   % git clone git://github.com/Starlink/starjava.git starjava/source
You can then read the file starjava/source/README for instructions on what to do next. Your mileage may vary somewhat, but the sequence will be along these lines:
   % cd starjava/source
   % setenv STAR_JAVA /usr/java/jdk1.5.0/bin/java .. or wherever
   % cd ant
   % bin/ant build install
   % cd ..
   % ../bin/ant build
   % ../bin/ant install
To build the standalone TOPCAT jar files topcat-full.jar and topcat-lite.jar you can do:
   % cd topcat
   % ../../bin/ant build-standalone install
Your JDK may be any version >=1.5, and must have JAI (Java Advanced Imaging) installed. Note that this process builds the whole starjava suite of software, which includes more than just TOPCAT.


Q1.14 I think I've found a bug.

If there's something wrong which is not noted in Section 4, please mail me and let me know. I may be able to give you a workaround or updated copy, and more importantly it gives me a chance to fix it in future versions.


Q1.15 Why doesn't TOPCAT display sky images/show movies/speak my weight?

I have a long list of features that I'd like to add eventually, some of which will probably get done and some of which will probably not. However, I'm always interested in new suggestions. The existing feature set is very largely due to suggestions and requests from existing users.

Having said that, there are some things which I am unlikely to add, in particular things which are not basically to do with manipulation of tables, and things which would require non-Java code to be introduced.


Q1.16 How can I find out when new versions are released?

Subscribe to the topcat-announce mailing list - see list information here.


Q1.17 What licence is TOPCAT released under?

The GNU General Public Licence (GPL). If this presents a problem for you, it may be possible to arrange use under some other licence.


Q1.18 What related software is there?

The following items are related to TOPCAT:

STIL
The Starlink Tables Infrastructure Library is a public API which provides the table I/O, manipulation and crossmatching functions used by TOPCAT. It is available for use by other software projects which need to do table (including VOTable) manipulation.
STILTS
The STIL Tool Set is a package of command-line utilities which provides many of the same facilities as TOPCAT, but without a graphical user interface. It's suitable for server-side use, can be scripted, and under certain circumstances can cope with larger tables than TOPCAT (since some operations can work in streaming mode).


Q1.19 Who has developed TOPCAT?

Most of the code has been written by Mark Taylor. Much help has been supplied by other members of various user and developer communities, and some external libraries are used. More details are available in the Acknowledgements section of the manual.


Q1.20 What project(s) support(s) TOPCAT?

TOPCAT has been supported by a number of grants and projects over its lifetime, nearly all ultimately funded by UK or European astronomy research organisations. In chronological order:

(some of these grants cover projects additional to TOPCAT/STILTS/STIL, but included support for that work).

The work has mostly been done in the Astrophysics Group of the Physics Department at Bristol University in the UK (though some was done in various senses at the Institute of Astronomy in Cambridge and ARI in Heidelberg).


Q1.21 What are its future prospects?

TOPCAT has no assured long term funding, but has managed for the last decade or so to hop from one funding source to another without falling between the cracks. At time of writing things are looking positive for the near future: funding is expected until around the end of 2015 by grants from STFC, the UK's main funding body for astronomical research, and the EU FP7 budget, for development and support for astronomical use both in general, and specifically with the archive generated by ESA's Gaia astrometry satellite.

If because of terminated funding or for some other reason I cease work on it, another project or individual could take on some level of maintenance, support and/or development in my absence. I like to think that the code is sufficiently well organised and documented that this is a realistic possibility. Feel free to take a look to see whether you agree.



2 Downloading and Starting TOPCAT

Q2.1 What do I need to run TOPCAT?

You need a J2SE (Java 2 Standard Edition) runtime environment version 1.5 or above (see Q2.2) and one of the TOPCAT jar files (topcat-lite.jar or topcat-full.jar). Since Java is available for almost all modern machines, TOPCAT will run on almost any computer with a screen, for instance Linux, Solaris, Mac OSX and MS Windows.


Q2.2 What version of Java do I need?

The short answer is that any non-ancient version of the Java 2 Standard Edition should be OK - specifically J2SE1.5 or later. On Mac OS X the supplied version of Java should work (though there are one or two known issues). On other platforms, Sun's JRE is recommended. If you do not already have this, you can obtain it from Oracle's Java SE page "Java Runtime Environment" is what you want. If you need to run using Java 1.4, you can use a TOPCAT version older than version 3.6.

Recent versions of OpenJDK have been reported to work well, though some people have also experienced problems with them. For the present I will continue to develop using Oracle's Java, so it remains the recommended platform on non-Mac platforms.

At the time of writing GNU's gcj/gij does not appear to work well. Unfortunately gij is distributed as /usr/bin/java in some Linux distributions.


Q2.3 I'm running Ubuntu and my TOPCAT windows look ugly

The implementation of Java that runs by default on current versions of Ubuntu at time of writing is based on OpenJDK. This works to some extent, but various things, including the size and shape of some graphical elements, is substandard (see the previous question - but this may only apply to older Ubuntu versions?). You are advised to use Sun's java instead. This can be done on Ubuntu as follows:

   % sudo update-java-alternatives -s java-6-sun
If that doesn't work, try "update-java-alternatives -l" to see what options are available, and try to find one with "sun" in the name.

If no such java version is offered, you might need to install it, using the Adept package installer. From the Universe repository. Or something. (Note: I'm not really up to speed on Ubuntu workings - I'm happy to accept more authoritative text from somebody who is better informed).


Q2.4 What's the best way to download and run TOPCAT?

The recommended method is to grab http://www.starlink.ac.uk/topcat/topcat-full.jar from the TOPCAT web page. On a Un*x system you can also get (and chmod +x) the http://www.starlink.ac.uk/topcat/topcat shell script and put it in the same directory. Then just invoke "topcat" (if you have the shell script)" or "java -jar topcat-full.jar" and you're off.

If you are tight for space, you can use topcat-lite.jar instead - it is missing only a few less important facilities.


Q2.5 How do I set command-line flags from Mac OS X?

If you have downloaded the topcat.dmg file from the web page and installed it on your Mac, you will be able to run it in a Mac-like way by pointing and clicking. However, this does not permit you to specify important command-line flags such as -Xmx256M, not to mention specifying tables on the command line to load at startup etc (see the manual for more information on command-line arguments).

To do that, you will have to run it from the command line. Depending where you have installed the .dmg file, the topcat script should be present at a location like:

   /Applications/TOPCAT.app/bin/topcat
so invoking /Applications/TOPCAT.app/bin/topcat <flags> or putting /Applications/TOPCAT.app/bin on your path and invoking topcat <flags> should do the trick.

Another possibility if there are command-line flags that you want to use on a regular basis is to edit the Info.plist file associated with the application bundle. You should find this file at TOPCAT.app/Contents/Info.plist; Apple provides documentation on the Info.plist file format. In particular if you want to increase the heap memory you can fill in the <string> value of the VMOptions key in the Java section with a value like "-Xmx512M".

Thanks to Frank Primini, Juan de Dios Santander Vela and Klaus Rübke for help with this question.


Q2.6 I have a Retina hi-res display on my Mac and TOPCAT looks blurry

Thanks to Christophe Deil for supplying this answer.

TOPCAT does work on Macbook Pro with Retina display, although some plots are drawn as bitmaps by the application and then scaled by 2x by the java implementation before being rendered to the Retina screen, resulting in unsharp image and text. Everything else (text, buttons, other plots) will look Retina-sharp if you start TOPCAT in high-resolution mode.

From the command line TOPCAT will start in high-resolution mode automatically. To make TOPCAT start in high-resolution mode when double-clicked on the icon in Applications or the Dock, you have to manually declare TOPCAT to be high-resolution capable. Add

   <key>NSHighResolutionCapable</key>
   <true/>
toward the end of the /Applications/TOPCAT.app/Contents/Info.plist file with the text editor of your choice, right before the closing </dict></plist> lines (see also the previous question). Because Mac OS X caches the Info.plist content you have do a little trick to make it reload the Info.plist content change. Go to Applications in Finder and select TOPCAT. Type CMD+C, then CMD+V and you'll get a copy called "TOPCAT copy". Delete the original TOPCAT, then rename "TOPCAT copy" to TOPCAT. This seems fishy, but it is the standard suggestion for plist changes.

Finally you can check by right-clicking on TOPCAT and selecting "Get Info" in Applications, you should see the "Open in Low Resolution" checkbox disabled. A simpler check is to just start TOPCAT, if low-resolution mode is enabled you will see it right away because then everything in TOPCAT (text, buttons, plots) will look unsharp.

I may decide to change the default in the future, but for now there seems a question about whether setting hi-res might cause problems. Any feedback about positive or negative experiences with this is very welcome.


Q2.7 How do I set up TOPCAT to run on Microsoft Windows?

The following instructions have kindly been supplied by Clive Page for use with Windows XP:

  1. Download Java (the JRE) from Sun's website as explained elsewhere in the TOPCAT instructions. The remaining instructions assume that the java.exe file is installed as C:\WINDOWS\System32\java.exe and they should be modified in the obvious way if this is not the case.
  2. Download either topcat-lite.jar or topcat-full.jar to a suitable location; the following instructions assume topcat-full.jar and that you put it in your own "My Documents" folder, but any folder will do.
  3. With the cursor in any blank area of your desktop, right-click the mouse and select New|Shortcut. A window appears asking for the location of this item: what is needed is of the following form, assuming that you put the topcat-full.jar in your own "My Documents" folder:
        C:\WINDOWS\System32\java.exe -jar "C:\Documents and Settings\your name\My Documents\topcat-full.jar"
        
    If you don't get this right first time it can be altered later, using the Properties window as described below. Note that you need quotation marks around the string after the -jar switch if it contains spaces, as it does in the example above. If you want to provide a switch for example to specify more memory, such as -Xmx100M for 100 Megabytes, then this switch should be inserted before the -jar switch. Click NEXT, then you get prompted for a name: enter "Topcat" or whatever seems appropriate. Click FINISH and a new icon should appear on your desktop (with a Java logo).
  4. Right click on this icon and select Properties. You will see a box labelled Start in which will be filled by default with the location of your java.exe file. You will probably find it convenient to alter this to some other folder, such as the one where your FITS files are generally located, or where your browser downloads them from the web. After editing it, finish by clicking Apply and OK.
When you click on the Topcat icon, it puts up first a text window, and then after a short interval the usual main Topcat window appears on top. You can minimise this text window, but do not delete it as this also deletes the Topcat window.

Christopher Bishop adds some notes relating to Vista and Firefox downloads:

If you are downloading Topcat using Firefox on Vista it is important that you make sure that you set the Firefox download to the folder you want to save it in. If you don't then the topcat-full.jar will end up by default in a temporary file. If you then follow steps 3 and 4 above a Topcat icon will be created, but you may find Vista will have deleted your download in the temp file. At least it did in mine! The solution I found was to go to Firefox Tools - Options - Main and then look at the downloads window. In the downloads window there is a box that enables you to specify where you want the download to go. So I put C:\user\yourname\documents (I would check in your documents folder that topcat-full.jar is really there and that Vista hasn't deleted it!)


Q2.8 Can TOPCAT be run as an applet?

No. It is possible that with some changes to the code it could be done, but I don't have any plans to do that. Applets are a bit out of fashion now, and in general TOPCAT seems a bit heavyweight to run inside a web browser (for instance it tends to need a lot of windows at once).

It can be run using Java WebStart, which is a technology that allows you to make one click on a web page and start the application (downloading it automatically first if necessary). You can see an example at topcat-lite.jnlp.


Q2.9 It doesn't seem to work!

Installing a java application such as TOPCAT is usually pretty straightforward, and there's not too much that can go wrong - at least compared to applications written in compiled languages such as C. If it's not starting, or failing to complete even very simple actions, the most likely explanation is that the version of Java you are using is broken or incomplete. The most likely cause of this is that you are using GNU Java, which at time of writing is not nearly complete enough to run TOPCAT. Unfortunately it is distributed as the default java in some Linux distributions. Running "java -version" should enable you to work out if this is the case. If you are using GNU java, then download a JRE from java.sun.com and make sure the java from that is first on your path. Apparently on Debian and Ubuntu, you can install Sun's JRE directly simply by doing "sudo apt-get install sun-java6-jre". Other distributions may have their own shortcuts.

If that doesn't seem to be the problem, please mail me and ask for help.



3 Using TOPCAT

Q3.1 What table format should I use?

The most efficient table format to use is FITS (either basic or "FITS-plus" - see Q3.2) specifically FITS BINTABLE extensions in uncompressed files on disk. If your tables are large, using FITS rather than some other format will both save memory (enabling you to work with larger datasets) and speed up table loading. If your data is initially in some other format (e.g. an ASCII-based one) it is often a good idea to convert once to FITS and subsequently use the FITS file within TOPCAT.

If your tables are small then the efficiency doesn't matter so much. VOTable and FITS-plus, and to a lesser extent basic FITS, can store rich metadata (units, UCDs, column descriptions etc) which may be something you want.

If you want to do processing of a table in some other language such as IDL or Python or Perl, a number of ASCII-based formats are available. Of these, CSV (Comma-Separated Values) is usually the best choice - the others can sometimes truncate long string values.


Q3.2 What's FITS-plus?

TOPCAT can read and write two variants of FITS binary tables, "FITS-basic" and "FITS-plus" (these are just terms used by STIL/TOPCAT to distinguish the two variants). Both of these are perfectly legal FITS tables, but TOPCAT squirrels away some extra metadata in the primary HDU when using FITS-plus. In most cases, TOPCAT writes FITS-plus by default, and reads either. Nearly all of the time, you shouldn't need to worry about the difference.

There is more detailed information in the manual.


Q3.3 How do I read Multi-Extension FITS files?

If you load a FITS file, any table extensions it contains (BINTABLE or TABLE extensions) will be loaded as separate tables by default.

Note that this behaviour has changed since version 3.5. In previous versions, by default only the first table extension was loaded, and any others were silently ignored.


Q3.4 How do I read multi-table VOTable documents?

If you load a VOTable document, any tables it contains (<TABLE> elements) will be loaded as separate tables by default.

Note that this behaviour has changed since version 3.5. In previous versions, by default only the first TABLE element was loaded, and any others were silently ignored.


Q3.5 What size tables can TOPCAT cope with?

This depends a bit on one thing and another, but as a rule of thumb a few million rows and a few tens of columns should be quite usable while maintaining quite good performance. If you're having trouble with much smaller tables than this, there is almost certainly something you can do about it. For more details see Q3.6.

Larger tables are not out of the question; there are reports of interactive use with 300 million rows.


Q3.6 Help! I've run out of memory.

A lot of effort has gone into making TOPCAT scalable. Some data sets may simply be too large to process, but up to a few million rows it should be possible to use the program with good performance and without running out of resources. You may have to take some steps to ensure that this is so however. Some good tips are:

See the manual for more discussion.

Some operations which would run out of memory in TOPCAT can be done using STILTS, since it can stream data in some circumstances.


Q3.7 Can I save a Row Subset?

Since version 3.6 (August 2010), the Session save option in the Save Window can be used to save your entire session (for some or all of the loaded tables). Unlike the Current Table save option, this saves things like lists of subsets, hidden columns, etc, in addition to the usual table data and metadata. It can be used to save a single table or several in the same file. To reload the session data, simply reload the saved file into TOPCAT like a normal table. Note however that this saves information about which rows are in which subsets as flags, rather than the rule for subset inclusion (such as an algebraic expression).

In previous versions, the only way you could do this was to create a new boolean column representing the subset(s) to save, and save the table with that column. You could create the new column from the Subset Window using the Subsets|To Column menu item or equivalent toolbar button; this column then contained true/false according to whether each row is/is not in the subset in question. When the table was reloaded into TOPCAT, new subsets will be created for any such boolean columns.


Q3.8 TOPCAT won't read my VOTable

It is possible that there is a bug in TOPCAT, but it's much more likely that there is an error in the VOTable. The best thing to do in this case is to diagnose the problem by using the STILTS votlint tool. Do not be afraid, this is very easy to do, something like:

   % stilts votlint foo.xml
or
   % java -jar stilts.jar votlint foo.xml
This will print WARNING or ERROR messages to the terminal describing questionable or illegal aspects of the file. You may be able to see what's wrong, and possibly fix it with careful use of a text editor from this information. Either way, if there are errors it would be a good idea to contact the originator of the VOTable with the votlint output and tell them that they are generating broken VOTables.


Q3.9 Where has my VOTable metadata gone?

TOPCAT's data access is format-neutral - it has its own model of what constitutes table data and metadata rather than being tied to a particular data format. This is good in that it means it can work with tables in various different formats. However, it also means that data/metadata structure which is very specific to a particular data format may not be accommodated. Things like VOTable <GROUP> and <RESOURCE> elements fall into this category.

To put it another way, both load and save operations in TOPCAT result in a sort of format translation (to and from TOPCAT's internal data model). Some things can get lost or changed in both of these stages, so unfortunately loading a table into TOPCAT and saving it without making any explicit changes can result in loss of data.

To put it a third way, TOPCAT is (amongst other things) a table editor, but if you think of it as a VOTable editor you may be disappointed.


Q3.10 Can I indicate missing upper/lower limits when plotting error bars?

There is no specific facility for displaying missing limits, but you can probably generate a plot that looks like you want it to. You can represent missing limits using the error bar style which resembles little arrow heads. Set up two subsets in the plot, one for points with error limits and one for points without. You can set up two data sets in the plot with the same point data but different error data, one containing the subset of points with limits and one containing the points without. For the points without limits, provide fake error values and use a suitable error plotting style, such as the error bars with the little arrow heads.


Q3.11 How can I join two tables with the same row numbers side by side?

TOPCAT's crossmatching facilities are very smart and very flexible. So much so that they can do stupid, easy things as well as clever, hard ones. If you have tables that want joining together without reordering the rows, choose the Exact Value algorithm and use the magic value "$0" for the Matched Value Column for each input table. $0 evaluates to the row index in each case, so that all the row 1's will be joined together, as will all the row 2's etc.


Q3.12 The crossmatching is giving the wrong results!

The crossmatching code in TOPCAT/STILTS is quite well tested, and reports of actual bugs are rare. However, I take the correctness seriously, and if you think you've found an anomaly, please report it.

A bug was reported on 23 May 2011 by Yannick Roehlly concerning sky matches near the line RA=0. If all the points in one table are on one side of this line, but some are within an error radius of it, the matcher may fail to identify the corresponding matches on the other side of the line. A related bug means that matches may be missed if tables in a match do not use a single normalised range for RA (e.g. 0<=RA<=360). This bug is fixed at TOPCAT v3.9 and STILTS v2.3-1.

Another minor bug was reported by Martin Hennemann in Feb 2012 that could result in some unwanted single-table output rows in multi-table group-mode matches; this was fairly harmless since it was obvious from the output table what was wrong.

No other crossmatching bug has been reported in recent years.

One thing which can be confusing is matching objects in crowded regions, since in crowded regions (where the typical distance between positions is similar to or less than the match radius) the results depend strongly on the details of the matching criteria.

Note in particular that TOPCAT's "Best Match Only" mode is symmetric between tables. If you select "Best Match Only" for the Match Selection toggle (or equivalently "find=best" in STILTS tmatch* tasks), then each row from either table will only appear a maximum of once in the output table. That means if object A in table 1 is a match for object B in table 2, it cannot be a match for any other objects in table 2. This may lead to fewer matches than you were expecting.

If what you want to do is to find, for each object in table 1, the closest match in table 2, you should use one of the asymmetric options: "Best match for each Table 1 row" (or "Table 2 row") in TOPCAT's Match Selection chooser, or "find=best1 (or "best2") in STILTS. Note that these options were only introduced in TOPCAT v3.9/STILTS v2.4, prior to that only symmetric best and all options were available. So, if you have an earlier version and want to do this, you should upgrade.

If you've got doubts about the correctness of a crossmatch, it's a very good idea to plot the results to see whether it's doing what you think or hope it is. Since v4.1 this is usually fairly easy; see the manual.


Q3.13 Can I crossmatch more than 2 tables at once?

Yes. The multi-table crossmatch options are available from the Joins menu in the control menu. They don't appear on the toolbar just because they would take up too much space. Currently a 4-way match is the most you can do at once. But that's only because nobody has asked me for more. If you want the option to match more than that, let me know and I will add higher-numbered options.

Somewhat more flexible multi-table crossmatches are available using the STILTS tmatchn command. Equivalent functionality will be added to TOPCAT at some point.


Q3.14 Can I concatenate more than 2 tables at once?

No. You can do this using the STILTS tcat or tcatn commands however.


Q3.15 Can TOPCAT read SExtractor output?

It cannot read the ASCII files that SExtractor outputs by default. If you want to use TOPCAT with the output from SExtractor, set it to write its output as FITS tables. You can do this by setting SExtractor's CATALOG_TYPE parameter to "FITS_1.0".


Q3.16 Can I script TOPCAT?

No. The only ways you can control TOPCAT itself are

  1. by giving arguments (including tables to load at startup) on the command line
  2. by pointing and clicking
  3. using SAMP (or PLASTIC) messages from other applications

However TOPCAT's sister application STILTS is a set of command-line tools which provide much of the same functionality as TOPCAT. You can write scripts using these applications if you want to automate the kind of table processing that TOPCAT can do.


Q3.17 Can I save/restore the state of a TOPCAT session?

Since version 3.6 (August 2010), yes you can. Use the Session option in the Save Window to save, and load the table that outputs in the normal way to restore.

Note however that this saves (most of) per-table state, and not everything, so for instance currently displayed plots will be lost. This may be improved in a future release.


Q3.18 Can I see a log of the actions TOPCAT has performed during a session?

No, you can't do that either.


Q3.19 Can TOPCAT talk to other tools/processes?

Yes, by using SAMP or PLASTIC.


Q3.20 What's SAMP and/or PLASTIC all about then?

SAMP is the Simple Application Messaging Protocol. PLASTIC is the PLatform for AStronomical Tool InterConnection. The two protocols do basically the same job in basically the same way, but with some differences - SAMP is an evolution of PLASTIC, and can in many ways be thought of as PLASTIC "version 2". PLASTIC is being deprecated - you should use SAMP instead unless you need to interoperate with tools which only have PLASTIC support.

The job they do is to allow different tools, usually running on the same desktop, to talk to each other, for instance by sending tables from one to another. This topic is discussed in more detail in the manual.

Briefly, if a SAMP/PLASTIC "hub" is running then some of TOPCAT's controls allow you to send instructions or data to other tools. Usually these options are controlled from an Interop menu on the relevant window.

A SAMP/PLASTIC hub may already be running as part of another application (such as the AstroGrid VODesktop). If it is not, TOPCAT usually starts one internally when it starts up.


Q3.21 Can I select the image viewer for TOPCAT to send images to?

Currently it is possible to choose the image viewer only for the View URL as Image option in the Activation Window by using the Image Viewer selector in that panel - SAMP/PLASTIC and non-SAMP/PLASTIC options are available. You cannot currently select the image viewer for the Display Cutout Image option or else where in TOPCAT, a non-SAMP/PLASTIC viewer is chosen automatically.

It is hoped that this situation will be rationalised and improved in a future version.


Q3.22 Can I calculate the difference between column values in adjacent rows?

The expression language used by TOPCAT works entirely within a single row, so doing things like adding a new column B whose value in row n is the difference between the values of another column A in row n and its value in row n+1 is not directly possible.

There is a hack which you can use to achieve this however. If you join (match) the table to itself so that the same column appears twice in the resulting table, but offset by one row, you can then do calculations involving both values. The join can be done in the Pair Match Window, by using the Exact Value match algorithm, using the same table for both inputs, with the Matched Value Column set to "index" and "index+1" respectively. The "index" (or equivalently "$0") token is a magic column which evaluates to the row number in the table. You can then add a column B with an expression like "A_2-A_1" to get the difference. If required, you could then match the part of the table containing the new column back to the original table.

Yes this is a bit of an effort. Here is a STILTS command which effectively does what is described above:

   stilts tmatch2 in1=table.fits in2=table.fits \
                  matcher=exact values1=index+1 values2=index join=1or2 \
                  ocmd='addcol b a_2-a_1'



4 Known Bugs and Issues

Q4.1 I can't find a resource in the NVO/VAO registry from the Cone/SIA/SSA/TAP window

Between mid-2010 and mid-2014 searching the NVO (a.k.a. VAO) registry at STScI did not work. This turned out to be the result of bugs in both TOPCAT and the STScI registry, as finally uncovered by the estimable Menelaus Perdikeas. In version 4.1 and later, these searches should work.


Q4.2 It crashes immediately on startup, or when I open the Load window

There is a bug which may be seen by people who are set up to use Starlink applications as well as TOPCAT. It may provoke messages like:

   no such constant AST__KYCIR
or
   #
   # An unexpected error has been detected by HotSpot Virtual Machine:
   #
   #  SIGSEGV (0xb) at pc=0xf79af28b, pid=5153, tid=3888896928

You should be able to fix it by unsetting the LD_LIBRARY_PATH environment variable, for instance if you use a C-like shell:

   % unsetenv LD_LIBRARY_PATH
or for bash/sh:
   $ unset LD_LIBRARY_PATH
   $ export LD_LIBRARY_PATH
If you continue to have problems, please contact me and I'll try to provide more help.


Q4.3 3D error bars extending outside the box are invisible

In the Cartesian/Spherical 3D plots, if an error bar would extend outside of the box/sphere then it is simply not plotted, so that even the part which is within the box/sphere is not visible.


Q4.4 Overwriting mapped FITS file causes Java crash

If a FITS file has been mapped for reading (this is the normal way that uncompressed FITS files on disk are read) and it is overwritten, either by TOPCAT or by some other program, it can result in a crash of the Java Virtual Machine (ugly error message to the console and instant death of TOPCAT). TOPCAT will not normally perform such an overwrite, but it can't stop you doing it from outside the program.

The best thing is: don't overwrite FITS files currently in use. However, you can work around this by deleting the file before you overwrite it, or by loading it into TOPCAT using a file:-type URL, e.g "file://localhost/foo/bar/cat.fits" rather than "/foo/bar/cat.fits". This will load without mapping (and as a result will be much slower for large files).


Q4.5 Scisoft 7.0 invocation script broken

TOPCAT is included in the ESO Scisoft software bundle. The "topcat" invocation script included with that bundle does not have the same functionality as the one documented in the manual, so for instance command-line arguments cannot be supplied to it. If it's not doing what you want, you can run using java -jar topcat-full.jar.

This problem should be fixed in Scisoft releases after 7.0.


Q4.6 Expressions containing self-references cause trouble

If an algebraic expression for a synthetic column or row subset contains a reference to itself via a $id identifier, the program may hang. It might recover - if you can edit the expression so it doesn't contain such a reference any more, do so. In any case, try to avoid doing this.


Q4.7 SIAP load dialogue doesn't work

The SIAP load dialogue has a bug in it at TOPCAT version 3.4 - it always fails and reports a NullPointerException. This is fixed in more recent versions.


Q4.8 This IPAC table won't load

There is a bug in the table output from some IPAC services including GATOR which means that some table data type declarations are truncated (e.g. type "doub" rather than "double" or "d"). Before version 3.5 TOPCAT treated these as invalid tables and refuses to load them, issuing an error like

Unknown IPAC data type doub

This bug may be fixed at the IPAC end one day; in the mean time versions of TOPCAT from 3.5 will work round it.

Of course in order to load IPAC format tables you must specify that they are in IPAC format, e.g. by selecting "IPAC" from the Format selector box; you knew that anyway.


Q4.9 I get an error about a SizingScrollPane on my Java 1.7 Mac

On some Macs running java 1.7 (Mountain Lion??), doing things like trying to open the table or statistics windows can give you a stack trace like:

   Exception in thread "AWT-EventQueue-0" java.lang.NullPointerException
           at uk.ac.starlink.topcat.SizingScrollPane.getConfig(SizingScrollPane.java:81)
           at uk.ac.starlink.topcat.SizingScrollPane.getPreferredSize(SizingScrollPane.java:40)
           ...
I can't tell if this is my fault or OSX's, but it is fixed in version 4.0 and later.


Q4.10 VizieR keyword searches don't work

The output of the VizieR keyword search web service changed in July 2012 in a backwardly incompatible way, and TOPCAT couldn't make sense of the new output. The people at CDS made a partial fix, so that if the service recognises a query from TOPCAT it serves the old format, but this doesn't always work. In particular if you are running from WebStart there seem to be problems.

This problem should go away in versions later than v3.9. Either upgrade, or use a non-WebStart version.


Q4.11 On MacOS Mountain Lion the .dmg file is reported "damaged" or "can't be opened"

On the Mountain Lion release of Mac OS X, attempting to install the topcat-full.dmg file apparently can result in the message:

"TOPCAT is damaged and can't be opened, you should eject the disk image"
or possibly (not so far reported, but looks like it might happen):
"TOPCAT can't be opened because it is from an unidentified developer"

It seems that this is in Mountain Lion and later, security settings (a.k.a. GateKeeper) default to not letting you run anything from anywhere other than the Mac App Store or from signed applications. Unfortunately, this results in it reporting java-based non-signed applications as "damaged".

There are two workarounds for this. One is to eliminate the quarantine flag for the file from the command line. If TOPCAT is in the current path, you can type

   xattr -d com.apple.quarantine TOPCAT.app

Alternatively, you can (perhaps temporarily) change one of the security settings by going to the Apple menu | "System Preferences..." | "Personal" section | "Security & Privacy" | "General" pane' and changing the setting Allow applications downloaded from: to Anywhere. You can run TOPCAT, and a dialog will be shown telling you the application is not signed. Once you accept it, you can revert, if you want, your security settings to the default setting, Allow applications downloaded from: Mac App Store and identified developers. Here is a more detailed explanation (for a different application, but you get the idea).

I may be able to fix this one day if I can work out how to sign an OS X application (I think I have a respectable certificate, but don't know how to use codesign). Offers of help welcome.

Thanks to Juan de Dios Santander Vela for help with this question.


Q4.12 My plot axis labels come out backwards on OS X!

It seems that in recent versions (java 1.7?) of MacOS X, the text written sideways up the Y axis of plots sometimes comes out written backwards, e.g. it appears to read "EDUTINGAM_B" rather than "B_MAGNITUDE". In fact any text written non-horizontally is placed wrong (e.g. sky axis labels). I am pretty sure this is a Mac Java bug, which I hope will fix itself at some point (does anybody know how to report this kind of thing?).

The new-style plotting windows in v4.1 can work around this issue (I believe) which seems to go away if you use antialiased text. As well as getting the text the right way round it does look a bit nicer like this, but it is also perceptibly slower. TOPCAT attempts to determine if its running in MacOS, and if so it sets antialiasing on by default. However, it doesn't always guess right (should be improved in future versions). You can turn antialiasing on/off by using the Antialias checkbox in the Grid tab of the Axes control in the various plot windows, so try that if your labels are backwards.

Note in versions after v4.1 this will work slightly differently - text antialiasing will be set using the Antialias text type in the Font selector instead. By then, it ought to be better at guessing OS, so you hopefully will not need to worry about it.

If you're using the old-style plot windows or an older topcat version, running on java 1.6 seems to make it work OK. Although apparently Oracle has removed the ability to select Apple's Java 1.6 from it's Java preference pane, you may be able to launch topcat using Java 1.6 by typing:

   export JAVA_HOME=`/usr/libexec/java_home -v 1.6`
   java -jar /path/to/topcat-full.jar

Thanks to Juan de Dios Santander Vela for help with this question.



5 Last Resorts

Q5.1 I still don't know...

Try consulting the manual. Or at least the relevant part - click the Help button in the window you're in for some context-specific documentation.


Q5.2 It's not in/I'm too lazy to read the manual.

The best thing is to subscribe to, then send a message to, the topcat-user mailing list. I am usually monitoring this, and you will probably get a reply quite soon.

If for some reason you don't want to do that, then by all means contact me by mail. I'd rather answer questions on usage than have people assume something doesn't work when it does.