EXTREME cribsheet: Converting for extreme data sets in 99 easy steps

This section gives a list of what you should to to a Starlink package to enable it to cope with large images. Many of the steps can be applied equally to modifying non-USSC code which makes use of the USSC. The steps here give an indication of how to proceed, but for more discussion of what's going on, refer to other sections in this document. Some examples of the commands you could use are given, but obviously you can change these to personal taste.

1.

Identify all C source files and all Fortran source files (the listing of package files given by the Source Code Browser can be useful for this).

% tar xf package_source.tar
% echo >ffiles *.f *.gen pkg1_par pkg1_err
% echo >cfiles *.c *.h

2.

Start up the EXTREME package

% extreme
 
   EXTREME commands are now available -- Version 0.1-0

3.

Run do-inscnf on Fortran files to wrap %VAL arguments in CNF_PVAL calls, if this has not already been done.

% do-inscnf `cat ffiles`
pkg1_sub.f:                Constant %VAL arg in PKG1_ADD?
5 new dependencies on include file 'CNF_PAR'.
5/25 modified files written in ./inscnf.changed.
Logfile is ./inscnf.log.

4.

Check CNF_PVAL insertions which generated warnings, and remove them if they wrap arguments which are not registered pointers.

% vi inscnf.changed/pkg1_sub.f

5.

Rebuild source archive, rebuild package, and test.

% mkdir retar
% cd retar
% tar xf ../package_source.tar
% cp ../inscnf.changed/* .
% tar cf ../package_source.tar *
% cd ..
% rm -r retar inscnf.changed
   ...

6.

Ensure that you have 64-bit copies of all the Starlink libraries on which the package depends (these are not required for making the changes, but will be needed for building and testing the modified package).

7.

Run do-frepint on Fortran files to replace INTEGER declarations by INTEGER * 8.

% tar xf package_source.tar `cat ffiles`
% do-frepint `cat ffiles`
pgk1_mult.f:               INTEGER*2 declaration not changed                 
pkg1_divid.f:              No IMPLICIT NONE in module PKG1_DIVID
pkg1_misc.f:               Nowhere to declare INT__'s in module PKG1_MISC

22/25 modified files written in ./frepint.changed.
Logfile is ./frepint.log

8.

Check constructs which led to warnings and make any necessary edits, following the discussion in section 3.2:

Missing IMPLICIT NONE:

An IMPLICIT NONE statement should really be inserted into the module in question and explicit declarations made for all the variables used. If this is impractical, the only essential change is to make sure that all the implicit INTEGER variables are explicitly declared as INTEGER * 8. Note it is not sufficient to make an IMPLICIT INTEGER*8 (I-N) declaration, since the resulting declarations would not be edited at build time by the makefile machinery described in section 3.4.

EQUIVALENCE statements:

Ensure that none of the memory associations are between INTEGER and some other type. If they are, careful recoding will probably be required.

Use of INTEGER specific intrinsic functions:

If possible these should be changed to the corresponding generic names of the intrinsic functions. If this is not possible some other workaround may be required.

INTEGER*n declarations left unchanged:

These probably require no further modification.

Nowhere to declare INT__'s:

This indicates that literal integer constants were found as (apparently) actual arguments to subroutine calls and replaced by constants, but frepint couldn't find a suitable place in the module to put the PARAMETER statements (it looks for an INCLUDE statement). In this case you will have to insert the lines yourself, but running frepint will tell you what it would have inserted, so that you can paste it in by hand.

% frepint pkg1_misc.f >/dev/null
frepint: Nowhere to declare INT__'s in module PGK1_MISC

*  Local constants for use as actual arguments:
      INTEGER * 8 INT__0
      INTEGER * 8 INT__12
      PARAMETER ( INT__0 = 0 )
      PARAMETER ( INT__12 = 12 )

9.

If there is any tricky storage association involving integers, such as use of COMMON or EQUIVALENCE, or a formal argument of different type to an actual argument, you may need to recode this. Hopefully such things should be few and far between.

10.

If the package calls any unconverted libraries (e.g. non-standard intrinsic functions), change the types of actual arguments in affected calls back to INTEGER.

11.

It is now possible, though not normally worth the effort, to change any private INTEGER*8 variables back to INTEGERs for efficiency or other reasons.

12.

Prepare for C source conversion: ensure that all functions are properly declared using ANSI C-style formal argument type declarations. If you think this may not be the case, compiler warnings may be of help.

% gcc -fsyntax-only -Wimplicit-function-declaration -Wstrict-prototypes *.c
tmp.c:160: warning: implicit declaration of function `pipe'
% man pipe
    ...

13.

Make sure that communication with Fortran routines is done properly using CNF as described in SUN/209. In particular it is important to use the TRAIL macro for passing character string lengths.

14.

Run do-crepint on C files to replace references to int type by INT_BIG.

% tar xf package_source.tar `cat cfiles`
% do-crepint `cat cfiles`
input.c:               2 x Format string implies int * (comment inserted)
output.c:              5 x Format string has %[cdiouxX*] (comment inserted)  
output.c:                  Format string non-literal (comment inserted)
toplevel.c:                Non-stdlib system header file <unistd.h>    
toplevel.c:                Type of main not changed from int
toplevel.c:                Type of argc not changed from int

8/8 modified files written in ./crepint.changed.
Logfile is ./crepint.log.

15.

Check constructs which led to warnings and make any necessary edits, following the discussion in section 3.3:

main or argc declaration unchanged:

Check that the items in question are supposed to be int and not just called main or argc by coincidence.

Format strings:

These will mostly require edits as shown in the examples and discussion in section 3.3. Positions of the warned-about calls are marked in the source code by `crepint:' comments.

signal, bsearch and qsort calls:

These calls do not need modification, but make sure that the functions which they pass are declared and defined using int type not INT_BIG.

Inclusion of non standard-library system header files:

These presumably indicate use of non standard-library system libraries; the calls to such functions should be identified, and any which take int * arguments or variable argument lists should be treated in a similar way to scanf and printf. You can see which calls are made to the functions declared in a header file by removing the include directive and checking compiler warnings, something like:

% grep -v '<unistd.h>' toplevel.c >tmp.c
% gcc -fsyntax-only -Wimplicit-function-declaration tmp.c

16.

Check that you've attended to all the Format string warnings and removed the flagging comments.

% grep 'crepint:' crepint.changed/*

17.

Have a look for implicit function or argument declarations and replace them by explicit INT_BIG declarations. Certain flags of gcc can be used to pick this up:

% gcc -fsyntax-only -Wimplicit-int -Wstrict-prototypes *.c

or the -proto flag of the Tru64 Unix C compiler:

% cc -noobject -protois -DINT_BIG=long crepint.changed/*.c
% grep '[^a-z]int[^a-z]' *.H

18.

It is now possible, though not normally worth the effort, to change any private INT_BIG variables back to ints for efficiency or other reasons.

19.

Rebuild the source archive.

% mkdir retar
% cd retar
% tar xf ../package_source.tar
% cp ../frepint.changed/* .
% cp ../crepint.changed/* .
% tar cf ../package_source.tar *
% cd ..
% rm -r retar frepint.changed crepint.changed

20.

Use extmk to make the necessary modifications to the mk file.

% mv mk mk.orig
% extmk < mk.orig > mk
% chmod 755 mk

21.

If extmk generated any warnings, edit the mk file by hand accordingly.

22.

If there are platform-dependent files in the package you must either add

      SOURCE_VARIANT='alpha_OSF1'

and

      SOURCE_VARIANT='sun4_Solaris'

to the new alpha_OSF1_64 and sun4_Solaris_64 stanzas of the mk file respectively, or generate new platform-dependent files for the two new platforms.

23.

Use extmakefile to make some of the necessary modifications to the makefile, and check the changes made.

% mv makefile makefile.orig
% extmakefile < makefile.orig >makefile
% diff makefile.orig makefile

24.

If extmakefile indicated that it failed to make any of the changes that it wanted to, insert the relevant bits of code by hand with reference to section 3.4.

25.

Edit the makefile by hand to make the rest of the necessary changes as discussed in 3.4:

• Ensure that all, and only, extractions of Fortran source files from source archives are processed using the REPLACE_INTEGER8 macro (replace the new dummy target $(FORTRAN_FILES) with macros representing the Fortran files to be extracted, and check for any other use of TAR_OUT to extract fortran files, e.g. in the `do_test' target). (search for uses of TAR_OUT).
• Add rules for building any new include files required by the changes (CNF_PAR, EXT_PAR or extreme.h).
• Add any new dependencies of object files on these include files.
• Fix, or remove, any private archive-type targets which have been broken by the fact that source files in the build directory are no longer original copies.

26.

Export the source, and build the package. Look out especially for the following warnings:

• Fortran variable misaligned in COMMON block.
• C pointer type mismatch.
• Incompatible arguments.

If there are compilation errors or new warnings fix them. In general, sun4_Solaris_64 is more sensitive to problems than alpha_OSF1_64, since the sizes of long and pointers are actually different between the 32-bit and 64-bit compilations.

27.

Test! Bear in mind that the changes may have broken the package on the old platforms as well as the new ones.