Chapter 11

Running scsh

Scsh is currently implemented on top of Scheme 48, a freely-available Scheme implementation written by Jonathan Rees and Richard Kelsey. Scheme 48 uses a byte-code interpreter for good code density, portability and medium efficiency. It is R5RS. It also has a module system designed by Jonathan Rees.

Scsh's design is not Scheme 48 specific, although the current implementation is necessarily so. Scsh is intended to be implementable in other Scheme implementations. The Scheme 48 virtual machine that scsh uses is a specially modified version; standard Scheme 48 virtual machines cannot be used with the scsh heap image.

There are several different ways to invoke scsh. You can run it as an interactive Scheme system, with a standard read-eval-print interaction loop. Scsh can also be invoked as the interpreter for a shell script by putting a ``#!/usr/local/bin/scsh -s'' line at the top of the shell script.

Descending a level, it is also possible to invoke the underlying virtual machine byte-code interpreter directly on dumped heap images. Scsh programs can be pre-compiled to byte-codes and dumped as raw, binary heap images. Writing heap images strips out unused portions of the scsh runtime (such as the compiler, the debugger, and other complex subsystems), reducing memory demands and saving loading and compilation times. The heap image format allows for an initial #!/usr/local/lib/scsh/scshvm trigger on the first line of the image, making heap images directly executable as another kind of shell script.

Finally, scsh's static linker system allows dumped heap images to be compiled to a raw Unix a.out(5) format, which can be linked into the text section of the vm binary. This produces a true Unix executable binary file. Since the byte codes comprising the program are in the file's text section, they are not traced or copied by the garbage collector, do not occupy space in the vm's heap, and do not need to be loaded and linked at startup time. This reduces the program's startup time, memory requirements, and paging overhead.

This chapter will cover these various ways of invoking scsh programs.

11.1  Scsh command-line switches

When the scsh top-level starts up, it scans the command line for switches that control its behaviour. These arguments are removed from the command line; the remaining arguments can be accessed as the value of the scsh variable command-line-arguments.

11.1.1  Scripts and programs

The scsh command-line switches provide sophisticated support for the authors of shell scripts and programs; they also allow the programmer to write programs that use the Scheme 48 module system.

There is a difference between a script, which performs its action as it is loaded, and a program, which is loaded/linked, and then performs its action by having control transferred to an entry point (e.g., the main() function in C programs) that was defined by the load/link operation.

A script, by the above definition, cannot be compiled by the simple mechanism of loading it into a scsh process and dumping out a heap image -- it executes as it loads. It does not have a top-level main()-type entry point.

It is more flexible and useful to implement a system as a program than as a script. Programs can be compiled straightforwardly; they can also export procedural interfaces for use by other Scheme packages. However, scsh supports both the script and the program style of programming.

11.1.2  Inserting interpreter triggers into scsh programs

When Unix tries to execute an executable file whose first 16 bits are the character pair ``#!'', it treats the file not as machine-code to be directly executed by the native processor, but as source code to be executed by some interpreter. The interpreter to use is specified immediately after the ``#!'' sequence on the first line of the source file (along with one optional initial argument). The kernel reads in the name of the interpreter, and executes that instead. The interpreter is passed the source filename as its first argument, with the original arguments following. Consult the Unix man page for the exec system call for more information.

Scsh allows Scheme programs to have these triggers placed on their first line. Scsh treats the character sequence ``#!'' as a block-comment sequence,15 and skips all following characters until it reads the comment-terminating sequence newline/exclamation-point/sharp-sign/newline (i.e., the sequence ``!#'' occurring on its own line).

In this way, the programmer can arrange for an initial

#!/usr/local/bin/scsh -s
!#
header appearing in a Scheme program to be ignored when the program is loaded into scsh.

11.1.3  Module system

Scsh uses the Scheme 48 module system, which defines packages, structures, and interfaces.

More information on the the Scheme 48 module system can be found in the file module.ps in the doc directory of the Scheme 48 and scsh releases.

Programming Scheme with a module system is different from programming in older Scheme implementations, and the associated development problems are consequently different. In Schemes that lack modular abstraction mechanisms, everything is accessible; the major problem is preventing name-space conflicts. In Scheme 48, name-space conflicts vanish; the major problem is that not all bindings are accessible from every place. It takes a little extra work to specify what packages export which values.

It may take you a little while to get used to the new style of program development. Although scsh can be used without referring to the module system at all, we recommend taking the time to learn and use it. The effort will pay off in the construction of modular, factorable programs.

11.1.3.1  Module warning

Programmers who open both the scheme and scsh structures in their own packages should make sure to always put the scsh reference first.

Do this: Not this:
    

(define-structure web-server
  (open scsh
        scheme
        net-hax
        )
  (file web))
    

(define-structure web-server
  (open scheme
        scsh
        net-hax
        )
  (file web))
Open scsh before scheme. Not scsh after scheme.
Ordering the two packages like this is necessary because scsh overrides some of the standard R4RS Scheme definitions exported by the scheme package with its own definitions. For example, scsh's versions of the R4RS I/O functions such as display and write take integer file descriptors as arguments, as well as Scheme ports. If you open the scheme structure before the scsh structure, you'll get the standard Scheme 48 definitions, which is not what you want.

11.1.4  Switches

The scsh top-level takes command-line switches in the following format:

scsh [meta-arg] [switchi ...] [end-option arg1 ... argn]
where
meta-arg: \ script-file-name
switch: -e entry-point Specify top-level entry-point.

-o structure Open structure in current package.

-m structure Switch to package.

-n new-package Switch to new package.

-lm module-file-name Load module into config package.

-l file-name Load file into current package.

-dm Do script module.
-ds Do script.
end-option: -s script
-sfd num
-c exp
--
These command-line switches essentially provide a little linker language for linking a shell script or a program together with Scheme 48 modules. The command-line processor serially opens structures and loads code into a given package. Switches that side-effect a package operate on a particular ``current'' package; there are switches to change this package. (These switches provide functionality equivalent to the interactive ,open ,load ,in and ,new commands.) Except where indicated, switches specify actions that are executed in a left-to-right order. The initial current package is the user package, which is completely empty and opens (imports the bindings of) the R4RS and scsh structures.

If the Scheme process is started up in an interactive mode, then the current package in force at the end of switch scanning is the one inside which the interactive read-eval-print loop is started.

The command-line switch processor works in two passes: it first parses the switches, building a list of actions to perform, then the actions are performed serially. The switch list is terminated by one of the end-option switches. The argi arguments occurring after an end-option switch are passed to the scsh program as the value of command-line-arguments and the tail of the list returned by (command-line). That is, an end-option switch separates switches that control the scsh ``machine'' from the actual arguments being passed to the scsh program that runs on that machine.

The following switches and end options are defined:

11.1.5  The meta argument

The scsh switch parser takes a special command-line switch, a single backslash called the ``meta-argument,'' which is useful for shell scripts. If the initial command-line argument is a ``\'' argument, followed by a filename argument fname, scsh will open the file fname and read more arguments from the second line of this file. This list of arguments will then replace the ``\'' argument -- i.e., the new arguments are inserted in front of fname, and the argument parser resumes argument scanning. This is used to overcome a limitation of the #! feature: the #! line can only specify a single argument after the interpreter. For example, we might hope the following scsh script, ekko, would implement a simple-minded version of the Unix echo program:

#!/usr/local/bin/scsh -e main -s
!#
(define (main args)
  (map (lambda (arg) (display arg) (display " "))
       (cdr args))
  (newline))
The idea would be that the command
ekko Hi there.
would by expanded by the exec(2) kernel call into
/usr/local/bin/scsh -e main -s ekko Hi there.
In theory, this would cause scsh to start up, load in file ekko, call the entry point on the command-line list
(main '("ekko" "Hi" "there."))
and exit.

Unfortunately, the Unix exec(2) syscall's support for scripts is not very general or well-designed. It will not handle multiple arguments; the #! line is usually required to contain no more than 32 characters; it is not recursive. If these restrictions are violated, most Unix systems will not provide accurate error reporting, but either fail silently, or simply incorrectly implement the desired functionality. These are the facts of Unix life.

In the ekko example above, our #! trigger line has three arguments (``-e'', ``main'', and ``-s''), so it will not work. The meta-argument is how we work around this problem. We must instead invoke the scsh interpreter with the single \ argument, and put the rest of the arguments on line two of the program. Here's the correct program:

#!/usr/local/bin/scsh \
-e main -s
!#
(define (main args) 
  (map (lambda (arg) (display arg) (display " "))
       (cdr args))
  (newline))
Now, the invocation starts as
ekko Hi there.
and is expanded by exec(2) into
   
/usr/local/bin/scsh \ ekko Hi there.
When scsh starts up, it expands the ``\'' argument into the arguments read from line two of ekko, producing this argument list:
-e main -s ekko Hi there.
        
^
|

Expanded from \ ekko
With this argument list, processing proceeds as we intended.

11.1.5.1  Secondary argument syntax

Scsh uses a very simple grammar to encode the extra arguments on the second line of the scsh script. The only special characters are space, tab, newline, and backslash.

You have to construct these line-two argument lines carefully. In particular, beware of trailing spaces at the end of the line -- they'll give you extra trailing empty-string arguments. Here's an example:

#!/bin/interpreter \
foo bar  quux\ yow

would produce the arguments
("foo" "bar" "" "quux yow")

11.1.6  Examples

Note that the sort example can be compiled into a Unix program by loading the file into an scsh process, and dumping a heap with top-level top. Even if we don't want to export the sort's functionality as a subroutine library, it is still useful to write the sort program with the module language. The command line design allows us to run this program as either an interpreted script (given the #! args in the header) or as a compiled heap image.

11.1.7  Process exit values

Scsh ignores the value produced by its top-level computation when determining its exit status code. If the top-level computation completed with no errors, scsh dies with exit code 0. For example, a scsh process whose top-level is specified by a -c exp or a -e entry entry point ignores the value produced by evaluating exp and calling entry, respectively. If these computations terminate with no errors, the scsh process exits with an exit code of 0.

To return a specific exit status, use the exit procedure explicitly, e.g.,

scsh -c \
  "(exit (status:exit-val (run (| (fmt) (mail shivers)))))"

11.2  The scsh virtual machine

To run the Scheme 48 implementation of scsh, you run a specially modified copy of the Scheme 48 virtual machine with a scsh heap image. The scsh binary is actually nothing but a small cover program that invokes the byte-code interpreter on the scsh heap image for you. This allows you to simply start up an interactive scsh from a command line, as well as write shell scripts that begin with the simple trigger

#!/usr/local/bin/scsh -s

You can also directly execute the virtual machine, which takes its own set of command-line switches.. For example, this command starts the vm up with a 1Mword heap (split into two semispaces):

scshvm -o scshvm -h 1000000 -i scsh.image arg1 arg2 ...
The vm peels off initial vm arguments up to the -i heap image argument, which terminates vm argument parsing. The rest of the arguments are passed off to the scsh top-level. Scsh's top-level removes scsh switches, as discussed in the previous section; the rest show up as the value of command-line-arguments.

Directly executing the vm can be useful to specify non-standard switches, or invoke the virtual machine on special heap images, which can contain pre-compiled scsh programs with their own top-level procedures.

11.2.1  VM arguments

The vm takes arguments in the following form:

scshvm [meta-arg] [vm-options+] [end-option scheme-args]
where
meta-arg: \ filename
vm-option: -h heap-size-in-words
-s stack-size-in-words
-o object-file-name
end-option: -i image-file-name
--

The vm's meta-switch ``\ filename'' is handled the same as scsh's meta-switch, and serves the same purpose.

11.2.1.1  VM options

The -o object-file-name switch tells the vm where to find relocation information for its foreign-function calls. Scsh will use a pre-compiled default if it is not specified. Scsh must have this information to run, since scsh's syscall interfaces are done with foreign-function calls.

The -h and -s options tell the vm how much space to allocate for the heap and stack. The heap size value is the total number of words allocated for the heap; this space is then split into two semi-spaces for Scheme 48's stop-and-copy collector.

11.2.1.2  End options

End options terminate argument parsing. The -i switch is followed by the name of a heap image for the vm to execute. The image-file-name string is also taken to be the name of the program being executed by the VM; this name becomes the head of the argument list passed to the heap image's top-level entry point. The tail of the argument list is constructed from all following arguments.

The -- switch terminates argument parsing without giving a specific heap image; the vm will start up using a default heap (whose location is compiled into the vm). All the following arguments comprise the tail of the list passed off to the heap image's top-level procedure.

Notice that you are not allowed to pass arguments to the heap image's top-level procedure (e.g., scsh) without delimiting them with -i or -- flags.

11.2.2  Inserting interpreter triggers into heap images

Scheme 48's heap image format allows for an informational header: when the vm loads in a heap image, it ignores all data occurring before the first control-L character (ASCII 12). This means that you can insert a ``#!'' trigger line into a heap image, making it a form of executable ``shell script.'' Since the vm requires multiple arguments to be given on the command line, you must use the meta-switch. Here's an example heap-image header:

#!/usr/local/lib/scsh/scshvm \
-o /usr/local/lib/scsh/scshvm -i
... Your heap image goes here ...

11.2.3  Inserting a double-level trigger into Scheme programs

If you're a nerd, you may enjoy doing a double-level machine shift in the trigger line of your Scheme programs with the following magic:

#!/usr/local/lib/scsh/scshvm \
-o /usr/local/lib/scsh/scshvm -i /usr/local/lib/scsh/scsh.image -s
!#
... Your Scheme program goes here ...

11.3  Compiling scsh programs

Scsh allows you to create a heap image with your own top-level procedure. Adding the pair of lines

#!/usr/local/lib/scsh/scshvm \
-o /usr/local/lib/scsh/scshvm -i
to the top of the heap image will turn it into an executable Unix file.

You can create heap images with the following two procedures.

(dump-scsh-program main fname)     --->     undefined         (procedure) 
This procedure writes out a scsh heap image. When the heap image is executed by the Scheme 48 vm, it will call the main procedure, passing it the vm's argument list. When main returns an integer value i, the vm exits with exit status i. The Scheme vm will parse command-line switches as described in section 11.2.1; remaining arguments form the tail of the command-line list that is passed to main. (The head of the list is the name of the program being executed by the vm.) Further argument parsing (as described for scsh in section 11.1.4) is not performed.

The heap image created by dump-scsh-program has unused code and data pruned out, so small programs compile to much smaller heap images.

(dump-scsh fname)     --->     undefined         (procedure) 
This procedure writes out a heap image with the standard scsh top-level. When the image is resumed by the vm, it will parse and execute scsh command-line switches as described in section 11.1.4.

You can use this procedure to write out custom scsh heap images that have specific packages preloaded and start up in specific packages.

Unfortunately, Scheme 48 does not support separate compilation of Scheme files or Scheme modules. The only way to compile is to load source and then dump out a heap image. One occasionally hears rumours that this is being addressed by the Scheme 48 development team.

11.4  Statically linking heap images

The static heap linker converts a Scheme 48 bytecode image contained in a .image file to a C representation. This C code is then compiled and linked in with a virtual machine, producing a single executable. Some of the benefits are:

The static heap linker takes arguments in the following form:

scsh-hlink image executable [option ...]
It reads in the heap image image, translates it into C code, compiles the C code, and links it against the scsh vm, producing the standalone binary file executable.

Each C file represents part of the heap image as a constant C long vector that looks something like this:

const  long p116[]={0x882,0x24,0x19,
                    0x882,(long)(&p19[785])+7,(long)(&p119[125])+7,
                    0x882,(long)(&p119[128])+7,(long)(&p119[131])+7,
                    0x882,(long)(&p102[348])+7,(long)(&p3[114])+7,
                    0xfc2,0x2030200,0x7100209,0x1091002,0x1c075a,
                    0x882,(long)(&p29[1562])+7,(long)(&p119[137])+7,
                    0x882,(long)(&p78[692])+7,(long)(&p119[140])+7,
                        .
                        .
                        .
                    };

Translating to a C declaration gives us freedom from the various object-file formats.16 Note that the const declaration allows the compiler to put this array in the text pages of the executable. The heap is split into parts because many C compilers cannot handle multi-megabyte initialised vector declarations.

The allowed options to the heap linker are:

Be warned that the current heap linker has many shortcomings.

In spite of these many shortcomings, we are providing the static linker as it stands in this release so that people may get some experience with it.

Here is an example of how one might use the heap linker:

        scsh-hlink scsh.image fastscsh

We'd love it if someone would dive into the source and improve it.

11.5  Standard file locations

Because the scshvm binary is intended to be used for writing shell scripts, it is important that the binary be installed in a standard place, so that shell scripts can dependably refer to it. The standard directory for the scsh tree should be /usr/local/lib/scsh/. Whenever possible, the vm should be located in

/usr/local/lib/scsh/scshvm
and a scsh heap image should be located in
/usr/local/lib/scsh/scsh.image
The top-level scsh program should be located in
/usr/local/lib/scsh/scsh
with a symbolic link to it from
/usr/local/bin/scsh

The Scheme 48 image format allows heap images to have #! triggers, so scsh.image should have a #! trigger of the following form:

#!/usr/local/lib/scsh/scshvm \
-o /usr/local/lib/scsh/scshvm -i
... heap image goes here ...


15 Why a block-comment instead of an end-of-line delimited comment? See the section on meta-args.

16 This idea is due to Jonathan Rees.