TOPCAT is an interactive graphical tool for analysis and manipulation of tabular data. Its manifesto is:
Does what you want with tables
A table, for TOPCAT's purposes, is a data structure with columns, so that all the values in a given column are of the same kind. The types of value in each column are usually, but not necessarily, numbers, strings, or arrays of these. The most important example in astronomy of a table is an object or source catalogue. Two- or multi-dimensional arrays, such as images or data cubes are not tables - TOPCAT will not be able to help you with these.
Q1.3 What are TOPCAT's strengths?
Some of these capabilities (limited scriptability, large tables) are however provided by TOPCAT's sister package, STILTS.
Q1.5 Is TOPCAT just for astronomy?
The large majority of TOPCAT's user base is currently in astronomy. However many of its facilities are not astronomy-specific; it is quite suitable for use in other contexts, and has been so used. I'm very open to approaches from other scientific domains who are interested in using it, and possibly customising it accordingly, but the amount of effort to publicise the software effectively in another user community would be considerable, and I'm not currently doing that.
Q1.6 Is TOPCAT just for the Virtual Observatory?
No. In many ways, TOPCAT is a traditional data analysis tool, in that it will perform most of its functions quite happily with no network connection. It does however provide a number of VO-friendly facilities.
Q1.7 Why is TOPCAT written in Java?
Several reasons, some of the important ones being:
No.
Q1.9 Gosh what a terrific bit of software. How can I express my appreciation?
Why thank you. The best thing you can do is to help publicise its use by telling your colleagues how you've found it useful. Acknowledging its use in published papers and talks is also very helpful.
Feedback is also great. Telling me about your positive experiences with it can (a) give me a warm fuzzy feeling inside and (b) come in useful when it comes to justifying development effort, e.g. to funding agencies. Telling me about your negative experiences with it can help me to make improvements.
Q1.10 How should I acknowledge TOPCAT?
It is much appreciated if you can acknowledge TOPCAT in your publications and presentations. The preferred method is to cite one or both of these papers:
2005ASPC..347...29T (TOPCAT)2006ASPC..351..666T (STILTS)Alternatively, you can cite one or both of the following permanent URLs:
I keep an eye on the ADS links above for citations of these papers. If you publish anything that cites or otherwise uses this software outside of the normal astronomy literature, I'd be most interested to hear about it.
Q1.11 What other papers are there relating to TOPCAT?
The following papers relate to TOPCAT and STILTS development and functionality:
2005ASPC..347...29T
TOPCAT & STIL: Starlink Table/VOTable Processing Software
2006ASPC..351..666T
STILTS - A Package for Command-Line Processing of Tabular Data
2008ASPC..394..422T
Column-Oriented Table Access Using STIL:
Fast Analysis of Very Large Tables
2009ASPC..411..510T
STILTS Plotting Tools: Scriptable graphics generation
from small and large tables
2014ASPC..485..257T
Visualising Large Datasets in TOPCAT v4
2015ASPC..495..177T
External Use of TOPCAT's Plotting Library
2017ASPC..512..589T
TOPCAT's TAP Client
10.3390/informatics4030018
TOPCAT: Desktop Exploration of Tabular Data for Astronomy and Beyond
2020ASPC..522...67T
TOPCAT: Working with Data and Working with Users
2019ASPC..523...43T
TOPCAT and Gaia
2022ASPC..532....3T
TOPCAT Visualisation Over the Web
2024ASPC..535..389T
Taplint, the TAP Service Validator
taplint
(Conference paper, 4pp)
2022arXiv221116913T
Improving Performance in Java: TOPCAT and STILTS
2024arXiv240101156T
TOPCAT Corner Plot
Q1.12 Where can I find old/new versions?
The most recent public release is available from the web page. Previous versions can be found in the release archive. New releases are notified on both the mailing lists.
There will sometimes be a bleeding-edge version in the pre-release directory - this may contain fixes and new features not yet available in the public release which you can experiment with at your own risk.
Q1.13 How do I get the source code to look at?
The source code for TOPCAT and friends is available in the Starlink/starjava repository on Github.
If you just want all the source code just to look at,
an alternative may be to grab the
topcat_src.zip file from the web page,
which contains all the java source code for TOPCAT, STILTS and the
uk.ac.starlink.* classes it uses.
Third party library code is not included.
See also Q1.14.
Q1.14 How can I compile TOPCAT from source code?
TOPCAT can be built from the sources at https://github.com/Starlink/starjava/. The README.md file there explains how to go about it. The only requirement is a JDK >=8 (currently tested for JDKs 7, 11 and 21). Note that considerable tidying up of the sources and build system were done in June 2024; prior to that there were various other requirements and pitfalls when attempting to build it.
Since the standalone jar file
topcat-full.jar
contains all the library classes, a quick and dirty alternative
is to compile from the sources in the
topcat_src.zip file from the web page
by putting topcat-full.jar on your classpath at compile time.
Note that most users will not need to do any compilation; the standalone jar files (topcat-full.jar etc) are platform-neutral and can be used with any compatible Java Runtime Environment (>= version 8)
Q1.15 I think I've found a bug.
If there's something wrong which is not noted in Section 4, please mail me and let me know. I may be able to give you a workaround or updated copy, and more importantly it gives me a chance to fix it in future versions.
Q1.16 Why doesn't TOPCAT display interactive sky images/show movies/speak my weight?
I have a long list of features that I'd like to add eventually, some of which will probably get done and some of which will probably not. However, I'm always interested in new suggestions. The existing feature set is very largely due to suggestions and requests from existing users.
Having said that, there are some things which I am unlikely to add, in particular things which are not basically to do with manipulation of tables, and things which would require non-Java code to be introduced.
Q1.17 How can I find out when new versions are released?
Subscribe to the topcat-announce mailing list - see
list information here.
Q1.18 What license is TOPCAT released under?
The application is released under the GNU General Public Licence (GPL). However, many of the underlying libraries are available under more permissive licenses such as the LGPL. There is some more information in the STILTS LICENSE.txt file.
Q1.19 What related software is there?
The following items are related to TOPCAT:
Q1.20 Who has developed TOPCAT?
Most of the code has been written by Mark Taylor. Much help has been supplied by other members of various user and developer communities, and some external libraries are used. More details are available in the Acknowledgements section of the manual.
Q1.21 What project(s) support(s) TOPCAT?
TOPCAT has been supported by a number of grants and projects over its lifetime, nearly all ultimately funded by UK or European astronomy research organisations. In chronological order:
The work has mostly been done in the Astrophysics Group of the School of Physics at the University of Bristol in the UK (though some was done in various senses at the Institute of Astronomy in Cambridge and ARI in Heidelberg).
Q1.22 What are its future prospects?
TOPCAT has no assured long term funding, but has managed for the last decade or two to hop from one funding source to another without falling between the cracks. At time of writing things are looking positive for the near future: funding is secured until mid 2027 by grants from STFC, the UK's main funding body for astronomical research.
If because of terminated funding or for some other reason I cease work on it, another project or individual could take on some level of maintenance, support and/or development in my absence. I like to think that the code is sufficiently well organised and documented that this is a realistic possibility. Feel free to take a look to see whether you agree.
Q2.1 What do I need to run TOPCAT?
You need a Java SE (Java 2 Standard Edition) runtime environment version 8 or above (see Q2.2) and, usually, one of the TOPCAT jar files (topcat-full.jar or topcat-extra.jar, though it's available in other forms too). Since Java is available for almost all modern machines, TOPCAT will run on almost any computer with a screen, for instance Linux, Solaris, Mac OSX and MS Windows.
If you are using the MacOS DMG file you don't need anything else, since that bundles its own Java runtime.
TOPCAT will not run on a phone or iPad, or within a web browser.
Q2.2 What version of Java do I need?
Current versions of TOPCAT/STILTS require Java SE 8 or later. This Java version has been available since 2014, so it is likely to be installed already, and if not should be available, for any but the most ancient platforms.
Development and testing is done using Oracle Java, but OpenJDK or whatever else is easy to install on your system is probably OK, as long as it's Java 8 or later.
If you are unable or unwilling to upgrade your Java installation, you can use an older TOPCAT version: v4.6-3 (2019) or earlier will run on Java 6, v4.3-2 (2015) or earlier will run on Java 5, and v3.5-2 (2010) or earlier will run on Java 4.
Q2.3 I get an error like "java.lang.UnsupportedClassVersionError"
This means that the Java version you have installed is too old to run TOPCAT. Since TOPCAT v4.7 (STILTS v3.2) you will need at least Java 8 ( Q2.1). Upgrade your java to a more recent version Q2.2 and try again.
Q2.4 What's the best way to download and run TOPCAT?
The recommended method is to grab one of the jar files
from the
TOPCAT web page:
if you may want to read/write parquet files then get
http://www.starlink.ac.uk/topcat/topcat-extra.jar,
otherwise the smaller
http://www.starlink.ac.uk/topcat/topcat-full.jar
will do.
On a Un*x system you can also get (and chmod +x) the
http://www.starlink.ac.uk/topcat/topcat
shell script and put it in the same directory.
Then just invoke "topcat" (if you have the shell script)"
or "java -jar topcat-full.jar" and you're off.
If you have a Mac, you may prefer to get the topcat-all.dmg file instead (which bundles Java so does not need an existing Java installation), but note items Q4.4 and Q2.6 in this case. With homebrew on a Mac, the following should do the trick:
brew install --cask topcat --no-quarantine
If you are running an Ubuntu-derived Linux distribution, see the question about Debian Astro.
Since early 2017, the TOPCAT and STILTS packages have been included in the Debian Astro suite (thanks to the hard work of Ole Streicher). That means that those packages are available in Debian-derived Linux distributions such as Ubuntu (since Debian 10 and Ubuntu 17.10). In such distributions, you should be able to just do
# apt install topcat stiltsto get both packages. This is not however guaranteed to give you the most recent versions.
Note however that these versions are not identical to the "official" releases available from http://www.starlink.ac.uk/topcat/. They should work OK, but the version may lag behind the official release, some of the external library versions are a bit different, a few of the more obscure features may be missing, etc. Version skew apart, differences in behaviour between the two are possible, but not expected to be significant.
Q2.6 How do I set command-line flags from Mac OS X?
If you have installed the topcat-all.dmg file
using homebrew or from the web page,
you will be able to run it in a
Mac-like way by pointing and clicking. However, this does not permit
you to specify important command-line flags such as
-Xmx8192M, not to mention specifying tables on the
command line to load at startup etc
(see the manual for
more information on command-line arguments).
To set command-line flags for use with the DMG file, since TOPCAT v4.9, you can edit the file
/Volumes/topcat/TOPCAT.app/Contents/vmoptions.txt
Note that in earlier versions the advice was different
(also more complicated and less reliable).
Alternatively, run the program from the command line.
If you have installed via homebrew, the topcat
and stilts scripts should be on your path.
If not, depending where you have installed the .dmg file,
the topcat script should be present at a location like:
/Volumes/topcat/TOPCAT.app/Contents/Resources/app/topcatso invoking the script at that location or putting that directory on your path and invoking
topcat with the relevant
flags should do the trick.
You can run STILTS in a similar way.
Failing that, you should be able to find the topcat-extra.jar
file somewhere. In that case, something like this ought to work:
java -Xmx8192M -jar /Volumes/topcat/TOPCAT.app/Contents/Resources/app/topcat-extra.jar
You don't have to use the DMG file of course; you can just download the topcat-full.jar (or topcat-extra.jar) jar file and optionally the topcat script directly and use them in the same way that other Un*x users do. This may be a good idea if you have trouble getting the DMG file working.
Thanks to Frank Primini, Juan de Dios Santander Vela, Klaus Rübke, Nicholas Seymour and Ming Yang for help with this question.
Q2.7 I have a Retina hi-res display on my Mac and TOPCAT looks blurry
Thanks to Christophe Deil for supplying this answer.
TOPCAT does work on Macbook Pro with Retina display, although some plots are drawn as bitmaps by the application and then scaled by 2x by the java implementation before being rendered to the Retina screen, resulting in unsharp image and text. Everything else (text, buttons, other plots) will look Retina-sharp if you start TOPCAT in high-resolution mode.
From the command line TOPCAT will start in high-resolution mode automatically. To make TOPCAT start in high-resolution mode when double-clicked on the icon in Applications or the Dock, you have to manually declare TOPCAT to be high-resolution capable. Add
<key>NSHighResolutionCapable</key> <true/>toward the end of TOPCAT's
Info.plist file with the text editor of
your choice, right before the closing </dict></plist>
lines.
See the previous question more information
about finding and editing the Info.plist.
Finally you can check by right-clicking on TOPCAT and selecting "Get Info" in Applications, you should see the "Open in Low Resolution" checkbox disabled. A simpler check is to just start TOPCAT, if low-resolution mode is enabled you will see it right away because then everything in TOPCAT (text, buttons, plots) will look unsharp.
I may decide to change the default in the future, but for now there seems a question about whether setting hi-res might cause problems. Any feedback about positive or negative experiences with this is very welcome.
Q2.8 How do I set up TOPCAT to run on Microsoft Windows?
The following instructions have kindly been supplied by Clive Page for use with Windows XP:
C:\WINDOWS\System32\java.exe
and they should be modified in the obvious way if
this is not the case.
topcat-extra.jar or
topcat-full.jar
to a suitable location;
the following instructions assume topcat-full.jar
and that you put it in your
own "My Documents" folder, but any folder will do.
topcat-full.jar in your own "My Documents"
folder:
C:\WINDOWS\System32\java.exe -jar "C:\Documents and Settings\your name\My Documents\topcat-full.jar"
If you don't get this right first time it can be altered later,
using the Properties window as described below.
Note that you need quotation marks around the string after the
-jar switch if it contains spaces, as it does in the
example above.
If you want to provide a switch for example to specify more memory,
such as -Xmx100M for 100 Megabytes,
then this switch should be inserted before the -jar switch.
Click NEXT,
then you get prompted for a name: enter "Topcat" or whatever seems
appropriate.
Click FINISH and a new icon should appear on your
desktop (with a Java logo).
java.exe file.
You will probably find it convenient to alter this to some other folder,
such as the one where your FITS files are generally located,
or where your browser downloads them from the web.
After editing it, finish by clicking Apply
and OK.
Christopher Bishop adds some notes relating to Vista and Firefox downloads:
If you are downloading Topcat using Firefox on Vista it is important that
you make sure that you set the Firefox download to the folder you want to
save it in. If you don't then the topcat-full.jar will end up by default in
a temporary file.
If you then follow steps 3 and 4 above a Topcat icon will be created,
but you may find Vista will have deleted your download in the temp file.
At least it did in mine!
The solution I found was to go to Firefox Tools - Options - Main and then
look at the downloads window. In the downloads window there is a box that
enables you to specify where you want the download to go. So I put
C:\user\yourname\documents
(I would check in your documents folder that
topcat-full.jar is really there and that Vista hasn't deleted it!)
Q2.9 Can I run it using Java WebStart?
WebStart is a Java technology which enables one-click download, installation, updating and invocation of Java applications over the web. It can be quite convenient, but it sometimes doesn't work properly and in more recent versions of Java it is deprecated (Java 9+) or no longer provided (Java 11+).
From early versions of TOPCAT, it was possible to run it using
WebStart, if you have WebStart installed, from a URL like
http://www.starlink.ac.uk/topcat/topcat-full.jnlp.
But newer Java installations don't support WebStart
and enabling some functionality requires signing the topcat jar files,
which is complicated and expensive.
Therefore from TOPCAT 4.8-8 (April 2023) WebStart invocation was provisionally deprecated, and the JNLP files removed at v4.10 (August 2024).
If people complain I may reinstate this for a while, but WebStart is on the way out, so it's likely to disappear eventually.
Q2.10 It doesn't seem to work!
Installing a java application such as TOPCAT is usually pretty straightforward, and there's not too much that can go wrong - at least compared to applications written in compiled languages such as C. If it's not starting, or failing to complete even very simple actions, the most likely explanation is that the version of Java you are using is broken or incomplete. You should try to install or reinstall Java for your system; see Q2.1 and Q2.2.
If you have a Mac, see also Q4.4.
If neither of those seems to be the problem, please mail me and ask for help.
Q3.1 What table format should I use?
The most efficient table format to use is FITS (either basic or "FITS-plus" - see Q3.2) specifically FITS BINTABLE extensions in uncompressed files on disk. If your tables are large, using FITS rather than some other format will both save memory (enabling you to work with larger datasets) and speed up table loading. For very large, and especially very wide tables (bigger than available RAM), the colfits format may be a better choice. If your data is initially in some other format (e.g. an ASCII-based one) it is often a good idea to convert once to a FITS-based format and subsequently use the FITS file within TOPCAT.
If your tables are small then the efficiency doesn't matter so much. VOTable and FITS-plus, and to a lesser extent basic FITS, can store rich metadata (units, UCDs, column descriptions etc) which may be something you want.
If you want to do processing of a table in some other language such as Python, IDL or Perl, a number of other formats are available. ECSV is good for interoperating with Python, Feather is efficient for interoperating with R, and CSV (Comma-Separated Values), while inefficient and having various problems, is generally a workable lowest-common-denominator solution.
TOPCAT can read and write two variants of FITS binary tables, "FITS-basic" and "FITS-plus" (these are just terms used by STIL/TOPCAT to distinguish the two variants). Both of these are perfectly legal FITS tables, but TOPCAT squirrels away some extra metadata in the primary HDU when using FITS-plus. In most cases, TOPCAT writes FITS-plus by default, and reads either. Nearly all of the time, you shouldn't need to worry about the difference.
There is more detailed information in the manual; see also SUN/252.
Q3.3 How do I read Multi-Extension FITS files?
If you load a FITS file, any table extensions it contains (BINTABLE or TABLE extensions) will be loaded as separate tables by default.
Q3.4 How do I read multi-table VOTable documents?
If you load a VOTable document, any tables it contains (<TABLE> elements) will be loaded as separate tables by default.
Q3.5 What size tables can TOPCAT cope with?
This depends a bit on one thing and another, but as a rule of thumb a few million rows and a few tens of columns should be quite usable while maintaining quite good performance. If you're having trouble with much smaller tables than this, there is almost certainly something you can do about it. For more details see Q3.6.
Larger tables are not out of the question; there are reports of interactive use with 300 million rows, and 20 thousand columns (not at the same time).
Q3.6 Help! I've run out of memory.
A lot of effort has gone into making TOPCAT scalable. Some data sets may simply be too large to process, but up to a few million rows it should be possible to use the program with good performance and without running out of resources. You may have to take some steps to ensure that this is so however. Some good tips are:
-Xmx flag.-disk flag.
But since v3.5 TOPCAT uses an adaptive storage policy,
and now this is rarely necessary.Some operations which would run out of memory in TOPCAT can be done using STILTS, since it can stream data in some circumstances.
The Session save option in the Save Window can be used to save your entire session (for some or all of the loaded tables). Unlike the Current Table save option, this saves things like lists of subsets, hidden columns, etc, in addition to the usual table data and metadata. It can be used to save a single table or several in the same file. To reload the session data, simply reload the saved file into TOPCAT like a normal table.
An alternative way, for instance if you want to communicate the subset contents to a non-TOPCAT application, is to create a new boolean column representing the subset(s) to save, and save the table with that column. You could create the new column from the Subset Window using the Subsets|To Column menu item or equivalent toolbar button; this column then contained true/false according to whether each row is/is not in the subset in question. When the table was reloaded into TOPCAT, new subsets will be created for any such boolean columns.
Q3.8 TOPCAT won't read my VOTable
It is possible that there is a bug in TOPCAT, but it's
much more likely that there is an error in the VOTable.
The best thing to do in this case is to diagnose the problem by using
the STILTS votlint
tool. Do not be afraid, this is very easy to do, something like:
% stilts votlint foo.xmlor
% topcat -stilts votlint foo.xmlThis will print WARNING or ERROR messages to the terminal describing questionable or illegal aspects of the file. You may be able to see what's wrong, and possibly fix it with careful use of a text editor from this information. Either way, if there are errors it would be a good idea to contact the originator of the VOTable with the
votlint output and tell them that they are generating
broken VOTables.
Q3.9 Where has my VOTable metadata gone?
TOPCAT's data access is format-neutral - it has its own model of what constitutes table data and metadata rather than being tied to a particular data format. This is good in that it means it can work with tables in various different formats. However, it also means that data/metadata structure which is very specific to a particular data format may not be accommodated. Things like VOTable <GROUP> and <RESOURCE> elements fall into this category, except in some specific cases (e.g. where they related to TIMESYS/COOSYS elements).
To put it another way, both load and save operations in TOPCAT result in a sort of format translation (to and from TOPCAT's internal data model). Some things can get lost or changed in both of these stages, so unfortunately loading a table into TOPCAT and saving it without making any explicit changes can result in loss of data.
To put it a third way, TOPCAT is (amongst other things) a table editor, but if you think of it as a VOTable editor you may be disappointed.
Q3.10 Can I indicate missing upper/lower limits when plotting error bars?
There is no specific facility for displaying missing limits, but you can probably generate a plot that looks like you want it to. You can represent missing limits using the error bar style which resembles little arrow heads. Set up two subsets in the plot, one for points with error limits and one for points without. You can set up two data sets in the plot with the same point data but different error data, one containing the subset of points with limits and one containing the points without. For the points without limits, provide fake error values and use a suitable error plotting style, such as the error bars with the little arrow heads.
Q3.11 How can I join two tables with the same row numbers side by side?
TOPCAT's crossmatching facilities are fairly smart and flexible.
So much so that they can do stupid, easy things as well as clever, hard ones.
If you have tables that want joining together without reordering the
rows, choose the Exact Value algorithm and use the
magic value "$0" for the Matched Value Column
for each input table. $0 evaluates to the row index in
each case, so that all the row 1's will be joined together, as will all
the row 2's etc.
In STILTS it's easier, there is a command that does just this:
tjoin.
Q3.12 The crossmatching is giving the wrong results!
The crossmatching code in TOPCAT/STILTS is quite well tested, and reports of actual bugs are rare. However, I take the correctness seriously, and if you think you've found an anomaly, please report it.
The following bugs affecting correctness of crossmatching have been reported in recent years:
<n>d_err / <n>-d Cartesian with Errorsskyerr / Sky with Errors2d_ellipse / 2-d Cartesian Ellipsesskyellipse / Sky EllipsehealpixRingIndex function, which resulted from a bug
in the vect2pix_ring function of the underlying
PixTools HEALPix library.
This function is used during sky crossmatching, so seems possible
that it could have resulted in some missed associations.
If so, the problems would have been for associations with a point on
each side of the longitude=0 line in the equatorial half of the
celestial sphere. Tests have failed to show up any actual match
failures, so it looks like this did not affect match results,
but it's hard to say for sure that it could not have happened.
This bug is fixed in any case at TOPCAT v4.5 and STILTS v3.1-1.
match
subset created in some matching modes by TOPCAT's Pair Match window
(the same issue affected the match<N> subsets
created by the Multiple Match window) contained completely the wrong rows.
This affected TOPCAT v4.8-4 to v4.8-8, but is fixed in later versions.
The rest of the matching was unaffected (the output tables contained
the right row and column content) and it did not affect STILTS.
No other crossmatching correctness bug has been reported in recent years.
One thing which can be confusing is matching objects in crowded regions, since in crowded regions (where the typical distance between positions is similar to or less than the match radius) the results depend strongly on the details of the matching criteria.
Note in particular that TOPCAT's "Best Match Symmetric" mode
is symmetric between tables.
If you select "Best Match Symmetric" for the Match Selection toggle
(or equivalently "find=best" in STILTS tmatch*
tasks), then each row from either table will only appear a maximum
of once in the output table. That means if object A in table 1 is a
match for object B in table 2, it cannot be a match for any other objects
in table 2. This may lead to fewer matches than you were expecting.
If what you want to do is to find, for each object in table 1, the
closest match in table 2, you should use one of the asymmetric options:
"Best match for each Table 1 row" (or "Table 2 row") in TOPCAT's
Match Selection chooser, or "find=best1 (or "best2")
in STILTS.
Note that these options were only introduced in TOPCAT v3.9/STILTS v2.4,
prior to that only symmetric best and all options were available.
So, if you have an earlier version and want to do this, you should upgrade.
If you've got doubts about the correctness of a crossmatch, it's a very good idea to plot the results to see whether it's doing what you think or hope it is. Since v4.1 this is usually fairly easy; see the manual.
Q3.13 Can I crossmatch more than 2 tables at once?
Yes. The multi-table crossmatch options are available from the Joins menu in the control menu. They don't appear on the toolbar just because they would take up too much space. Currently a 4-way match is the most you can do at once. But that's only because nobody has asked me for more. If you want the option to match more than that, let me know and I will add higher-numbered options.
Somewhat more flexible multi-table crossmatches are available
using the STILTS
tmatchn command.
Equivalent functionality will be added to TOPCAT at some point.
Q3.14 Can I concatenate more than 2 tables at once?
No. You can do this using the STILTS
tcat or
tcatn
commands however.
Q3.15 Can TOPCAT read SExtractor output?
It cannot read the ASCII files that SExtractor outputs by default. If you want to use TOPCAT with the output from SExtractor, set it to write its output as FITS tables. You can do this by setting SExtractor's CATALOG_TYPE parameter to "FITS_1.0".
No. The only ways you can control TOPCAT itself are
However TOPCAT's sister application STILTS is a set of command-line tools which provide much of the same functionality as TOPCAT. You can write scripts using these applications if you want to automate the kind of table processing that TOPCAT can do. For generating STILTS plots corresponding to TOPCAT plots, the Stilts Control in TOPCAT's plot windows may be useful.
Q3.17 Can I save/restore the state of a TOPCAT session?
Since version 3.6 (August 2010), yes you can. Use the Session option in the Save Window to save, and load the table that outputs in the normal way to restore.
Note however that this saves (most of) per-table state, and not everything, so for instance currently displayed plots will be lost. This may be improved in a future release.
Q3.18 Can I see a log of the actions TOPCAT has performed during a session?
No, I'm afraid not.
Q3.19 Can TOPCAT talk to other tools/processes?
Yes, by using SAMP or PLASTIC.
Q3.20 What's SAMP and/or PLASTIC all about then?
SAMP is the Simple Application Messaging Protocol.
It allows different tools, usually running on the same desktop, to talk to each other, for instance by sending tables from one to another. This topic is discussed in more detail in the manual.
Briefly, if a SAMP "hub" is running then some of TOPCAT's controls allow you to send instructions or data to other tools. Usually these options are controlled from an Interop menu on the relevant window.
A SAMP hub may already be running as part of another application (such as Aladin). If it is not, TOPCAT usually starts one internally when it starts up.
PLASTIC is the PLatform for AStronomical Tool InterConnection; it was an earlier protocol that did effectively the same thing as SAMP, but is now more or less obsolete.
Q3.21 Can I calculate the difference between column values in adjacent rows?
The expression language used by TOPCAT works entirely within a single row, so doing things like adding a new column B whose value in row n is the difference between the values of another column A in row n and its value in row n+1 is not directly possible.
There is a hack which you can use to achieve this however.
If you join (match) the table to itself so that the same column appears
twice in the resulting table, but offset by one row, you can then
do calculations involving both values. The join can be done
in the Pair Match Window,
by using the Exact Value match algorithm,
using the same table for both inputs, with the
Matched Value Column
set to "$index" and "$index+1" respectively.
The "$index" (or equivalently "$0") token is
a magic column which evaluates to the row number in the table.
You can then add a column B with an expression like "A_2-A_1"
to get the difference. If required, you could then match the part of the
table containing the new column back to the original table.
Yes this is a bit of an effort. Here is a STILTS command which effectively does what is described above:
stilts tmatch2 in1=table.fits in2=table.fits \
matcher=exact values1='$index+1' values2='$index' join=1or2 \
ocmd='addcol b a_2-a_1'
Q4.1 I can't see my Documents/Downloads/etc folders on MacOS Catalina+
It seems that in Catalina (and later versions?) of MacOS, certain directories are effectively invisible from the various file choosers, e.g. the Filestore Browser used in the Load and Save windows.
This seems to be related to new security restrictions introduced in this version of MacOS; I think it might be to do with the way that TOPCAT is launched from the .dmg file (further reading for enthusiasts here and here). I currently can't work out how to solve it, but some workarounds are possible: from the Load and Save windows, there is the option to use the System Browser as an alternative to the Filestore Browser. If you do that, you should get the standard MacOS Chooser, which ought to be able to see all the directories. Alternatively, if you start topcat using
java -jar topcat-full.jarrather than by clicking on the icon, it seems to work OK.
Something else that might work is accessing System Preferences/Security & Privacy/Privacy and giving topcat Full Disk Access.
Apparently, if you use the System Browser to look at these folders once, then subsequent access with the Filestore Browser works as well, maybe even in different TOPCAT invocations; see this report.
Thanks to Davide Lena (SRON) Marina Kounkel (U.Michigan), and Jarle Brinchmann (Leiden) for help investigating this.
Q4.2 Overwriting mapped FITS file causes Java crash
If a FITS file has been mapped for reading (this is the normal way that uncompressed FITS files on disk are read) and it is overwritten, either by TOPCAT or by some other program, it can result in a crash of the Java Virtual Machine (ugly error message to the console and instant death of TOPCAT). TOPCAT will not normally perform such an overwrite, but it can't stop you doing it from outside the program.
The best thing is: don't overwrite FITS files currently in use.
However, you can work around this by deleting the file before you overwrite it,
or by loading it into TOPCAT using a file:-type URL, e.g
"file://localhost/foo/bar/cat.fits" rather than
"/foo/bar/cat.fits". This will load without mapping
(and as a result will be much slower for large files).
Q4.3 Expressions containing self-references cause trouble
If an algebraic expression for a synthetic column or row subset
contains a reference to itself via a $id identifier,
the program may hang. It might recover - if you can edit the expression
so it doesn't contain such a reference any more, do so.
In version 4.6-3 and later, TOPCAT will usually prevent you from doing
this.
Q4.4 On MacOS the .dmg file is reported "damaged" or "can't be opened"
On the Mountain Lion release (and later?) of Mac OS X, attempting to install the topcat-all.dmg file apparently can result in the message:
"TOPCAT is damaged and can't be opened, you should eject the disk image"or maybe
"TOPCAT.app can't be opened because Apple can't check it for malicious softwareor
"TOPCAT can't be opened because it is from an unidentified developer"
It seems that in Mountain Lion and later, security settings (a.k.a. GateKeeper) default to not letting you run anything from anywhere other than the Mac App Store or from signed applications. Unfortunately, this results in it reporting java-based non-signed applications as "damaged".
There are several workarounds for this (though that may be MacOS-version dependent). Try one of the following:
curl -OL http://www.starlink.ac.uk/topcat/topcat-all.dmg",
then install TOPCAT with the downloaded .dmg file
(this works because using curl instead of the browser bypasses
the MacOS Gatekeeper quarantine).
xattr -d com.apple.quarantine TOPCAT.app".
sudo spctl --master-disable
sudo spctl --master-enable on the command line, or
"Allow applications downloaded from:
Mac App Store and identified developers" in the GUI)
and the application should continue to run.
I think the underlying problem is that I'm not a trusted Apple developer, because I'm not keen to pay the non-negligable amount of money that costs. If anybody has other/better ways to fix it, advice is welcome.
Thanks to various people who have helped with answering this question, including Juan de Dios Santander Vela, Jarle Brinchmann, Marina Kounkel, Alexander Mayer and Andy Lawrence.
Q4.5 Tables are scrambled when scrolling
In some cases apparently the table viewer window and other JTables displayed can get their pixels scrambled when they are scrolled. This must be some kind of bug in Java or maybe some display drivers. It seems to have more than one possible cause.
If the scrambling happens when you scroll to the right, the issue may be the one described here; to fix it try running with one or other of these system properties set:
sun.java2d.xrender=true
sun.java2d.opengl=true
Q4.6 The text on my high-resolution screen is too small
Some users report that running TOPCAT on high-resolution (4k, HiDPI, ??) screens results in a GUI that comes out too small, e.g. including fonts that are hard to read. The details of this problem seem to depend on exactly what hardware and software you have. Here are some measures that might fix it:
GDK_SCALE environment variable, which is set to
an integer scaling factor.
(Does this only work in Gnome?
If so, does setting GDK_SCALE by hand
in non-Gnome environments help?)
sun.java2d.uiScale,
e.g. java -Dsun.java2d.uiScale=2 -jar topcat-full.jar
(see system properties).
In principle this might accept fractional values such as 1.5,
but at least in some (most?) cases it seems to be restricted to
integer values.
GDK_DPI_SCALE be set in some circumstances as well??
)
If you have any experience on what does and doesn't work in different environments, please get in touch so I can improve the advice here.
Q4.7 The TOPCAT window is unexpectedly dark/black on my M1 Mac
Under some circumstances, the TOPCAT windows can apparently go black or enter some kind of dark mode and make it hard to see what's going on, when running on (new M1 architecture?) Macs. I don't know whether this is a single issue or several different ones, but here is some relevant information and things which have been found to fix it:
-Dsun.java2d.xrender=false", e.g.
"java -Dsun.java2d.xrender=false /path/to/topcat-full.jar".
Possibly "-Dsun.java2d.pmoffscreen=false" might help too?
defaults write org.xquartz.X11 enable_render_extension 0".
It may be necessary to uninstall XQuartz,
then issue this defaults command,
then reinstall XQuartz (e.g. v2.8.5) for this to take effect;
or maybe you just need to restart it, reports differ.
Thanks to Ben Maughan (Bristol) and Ariane Lançon (Strasbourg) for investigating this. If you have more/different/better information about what works, please get in touch.
Q4.8 MacOS v14 reports a WARNING about Secure coding
MacOS v14 (Sonoma), and possibly later versions, may issue the following warning when TOPCAT is started:
WARNING: Secure coding is not enabled for restorable state!
Enable secure coding by implementing
NSApplicationDelegate.applicationSupportsSecureRestorableState:
and returning YES.
This has to do with a potential security vulnerability in
application code related to save/restore of windows;
see here for more detail.
It doesn't look to me like this is going to cause actual security
problems to TOPCAT users, but I have to admit I don't understand the
issue very well. I believe that avoiding save/restore of TOPCAT windows
(which I don't think will really work anyway) ought to avoid any
potential vulnerability.
The fix to this has to be done in the Java Runtime that TOPCAT
executes in.
I believe it has been fixed at OpenJDK 8u421, 11.0.24, 17.0.12 and 21.0.4
(see issue JDK-8318854).
If you are using a topcat-all.dmg file
downloaded before 17 July 2024 you may see this message,
but for downloads from
http://www.starlink.ac.uk/topcat/topcat-all.dmg
since 18 July 2024 (? or possibly 27 July 2024) it should go away,
since the DMG file now bundles Corretto OpenJDK >=11.0.24.
Q5.1 I can't find a resource in the NVO/VAO registry from the Cone/SIA/SSA/TAP window
Between mid-2010 and mid-2014 searching the NVO (a.k.a. VAO) registry at STScI did not work. This turned out to be the result of bugs in both TOPCAT and the STScI registry, as finally uncovered by the estimable Menelaus Perdikeas. In version 4.1 and later, these searches should work. In version 4.2 and later, the RegTAP protocol is used by default, and the problem ought not to arise.
Q5.2 It crashes immediately on startup, or when I open the Load window
There is a bug which may be seen by people who are set up to use Starlink applications as well as TOPCAT. It may provoke messages like:
no such constant AST__KYCIRor
# # An unexpected error has been detected by HotSpot Virtual Machine: # # SIGSEGV (0xb) at pc=0xf79af28b, pid=5153, tid=3888896928
You should be able to fix it by unsetting the LD_LIBRARY_PATH environment variable, for instance if you use a C-like shell:
% unsetenv LD_LIBRARY_PATHor for bash/sh:
$ unset LD_LIBRARY_PATH $ export LD_LIBRARY_PATHIf you continue to have problems, please contact me and I'll try to provide more help.
Q5.3 3D error bars extending outside the box are invisible
In the Cartesian/Spherical 3D plots, if an error bar would extend outside of the box/sphere then it is simply not plotted, so that even the part which is within the box/sphere is not visible.
Q5.4 Scisoft 7.0 invocation script broken
TOPCAT is included in the ESO
Scisoft software bundle.
The "topcat" invocation script included with that bundle
does not have the same functionality as the one documented in
the manual,
so for instance command-line
arguments cannot be supplied to it.
If it's not doing what you want, you can run using
java -jar topcat-full.jar.
This problem should be fixed in Scisoft releases after 7.0.
Q5.5 I get an error about a SizingScrollPane on my Java 1.7 Mac
On some Macs running java 1.7 (Mountain Lion??), doing things like trying to open the table or statistics windows can give you a stack trace like:
Exception in thread "AWT-EventQueue-0" java.lang.NullPointerException
at uk.ac.starlink.topcat.SizingScrollPane.getConfig(SizingScrollPane.java:81)
at uk.ac.starlink.topcat.SizingScrollPane.getPreferredSize(SizingScrollPane.java:40)
...
I can't tell if this is my fault or OSX's, but it is fixed in version 4.0
and later.
Q5.6 My plot axis labels come out backwards on OS X!
It seems that in recent versions (java 1.7?) of MacOS X,
the text written sideways up the Y axis of plots sometimes
comes out written backwards, e.g. it appears to read
"EDUTINGAM_B" rather than "B_MAGNITUDE".
In fact any text written non-horizontally is placed wrong
(e.g. sky axis labels).
I am pretty sure this is a Mac Java bug, which I hope will fix itself
at some point (does anybody know how to report this kind of thing?).
The new-style plotting windows in v4.1 can work around this issue (I believe) which seems to go away if you use antialiased text. As well as getting the text the right way round it does look a bit nicer like this, but it is also perceptibly slower. TOPCAT attempts to determine if its running in MacOS, and if so it sets antialiasing on by default. However at version 4.1, it doesn't always guess right. You can turn antialiasing on/off by using the Antialias checkbox in the Grid tab of the Axes control in the various plot windows, so try that if your labels are backwards.
In version 4.2 and later, text antialiasing will be set using the Antialias text type in the Font selector instead. By then, it ought to be better at guessing OS, so you hopefully will not need to worry about it.
Q5.7 When I load an LDAC FITS file I can't see my data
Files in the LDAC FITS format are organised in an unusual way with the first (LDAC_IMHEAD) table extension containing a single-cell table where the cell's content is an array of FITS headers. This confuses TOPCAT into thinking that the file is in colfits format, which in turn means it ignores the second (LDAC_OBJECTS) table extension, and that's the one you probably want to look at.
A hack is present in TOPCAT v4.2-2 and later to avoid this,
but there are various ways to work around it anyway.
One is to select "FITS" (rather than "(auto)")
as the input file format.
Another is to use the
Hierarchy Browser
to examine the file and select the table you want.
You can also select the correct table by name or HDU index as, e.g.,
"D1_r_sex.cat#LDAC_OBJECTS" or
"D1_r_sex.cat#2", either on the command line or
using the Position in file field at the bottom right of the
Filestore Browser
window.
Try consulting the manual. Or at least the relevant part - click the Help button in the window you're in for some context-specific documentation. The Find field in the Help Window is also good for locating the right part of the documentation.
Q6.2 It's not in/I'm too lazy to read the manual.
You can subscribe to, then send a message to, the topcat-user mailing list, if it's a question you think others might be interested in too. Otherwise, or if you prefer not to use the mailing list for any reason, you can email the author at m.b.taylor@bristol.ac.uk. Either way, you can usually expect a reply quite soon.