% A typical header for a latex file
% also sets up lozenges for abort resume and end
\documentstyle[twoside,12pt]{report}   
\pagestyle{headings}
\topmargin = 0cm
\oddsidemargin = 0cm
\evensidemargin = 0cm
\headheight = 5mm
\headsep = 3mm
\textheight = 8in
\textwidth = 5.9in
\makeindex
\begin{document}
\input{ekberg$disk:[ekberg.tex]pretxt.tex}
 
% titlepage set up
\begin{titlepage}
\begin{center}
{\tiny .} \\
\vspace{2in}
{\huge Performance Measurement Software} \\
\vspace{3in}
{\large Very Rough Draft} \\
\vspace{1in}
\today{} \\
\end{center}
\end{titlepage}
\clearpage
\setcounter{page}{1}
\pagenumbering{roman}
\tableofcontents
\clearpage
\setcounter{page}{0}
\pagenumbering{arabic}

\chapter{Introduction}

This document describes the performance software available for running
benchmark suites, manipulating the collected data, and reporting
information from the collected data.  These topics are initially
discussed with reguard to the three standard benchmark suites, Gabriel,
Zippel, and Interactive, and existing report formats, in order to
provide general, but essential information for using the performance
software.  The section on performance testing methodology should also be
read, before running any suite, in order to ensure accuracy of the
collected data.  Additional topics discuss details of the performance
software data structure, the different approaches to defining your own
suites as exemplified by the three standard suites, and how to define
personalized reports.

\chapter{Standard Benchmark Suites}

\section{Introduction}

This chapter will discuss the functionality of the benchmarks that
comprise each of the standard benchmark suites, GABRIEL, ZIPPEL, and
INTERACTIVE.  The benchmarks in each of the suites were designed to
place the system under specifically controlled conditions and the
performance software records system performance under these conditions.
Each of the suites represents a unique class of benchmarks, each with
its own method of definition.  The methods for defining benchmark suites
of each type will be discussed in the chapter, Defining New Benchmark
Suites. 
   
\section{The Gabriel Benchmarks}   

The GABRIEL benchmark suite, a subset of the Richard P.  Gabriel
collection, includes the following benchmarks: Boyer, Browse, Deriv,
Dderiv, Fdderiv, Fft, Traverse, Destru, Puzzle, Triang, Tak, Takr, Stak,
Ctak, Takl, Fprint, Fread, Nprint, Tprint, Frpoly, and Div2.  Each of
these is a self-contained function or group of functions which performs
some specific task while the performance software meters the function's
performance.

The following descriptions of the Gabriel benchmarks are excerpts from
the the book {\underlinePerformance and Evaluation of Lisp Systems} by
Richard P. Gabriel, published by the MIT press in 1985.

Boyer, essentially a theorem-proving benchmark, is a good mixture of
typical list-structure-manipulating operations with a proportional
number of function calls.  Boyer performs about three quarters of a
million CARs and a half million each of CDRs, NULLs, and ATOMs.  

Browse performs many of the list-structure-manipulating operations
performed by Boyer, but the mixture of operations is proportionally
similar to that of an expert system.  

Deriv is a symbolic derivative benchmark, which uses a simple subset of
Lisp and does a lot of Consing.  Dderiv is a variant of deriv that is
table-driven, using property lists rather than Cond.  Fdderiv is a
variant of Dderiv which optimizes Funcall by using a Funcall-like
function that assumes it is being handed compiled code. 

Fft is a benchmark written by Harry Barrow which tests a variety of
floating point operations, including array references. 

Traverse is a benchmark that creates and traverses a tree structure.  

Destru benchmarks the destructive operations Rplaca and Rplacd.  

Puzzle is Forest Baskett's benchmark program which solves a
block-packing puzzle invented by John Conway, using a two dimensional
array.  Triang is a program which solves a board game using a one
dimensional array.  

The ''Tak'' set of Gabriel benchmarks is a family of heavily recursive
functions, called the Takeuchi functions.  Takr is a version of Tak that
was designed to thwart the effectiveness of cache memories.  Stak uses
special binding to pass arguments rather than the normal argument
passing technique.   Ctak is a version of Tak that uses CATCH and THROW
to return values instead of using the function-return mechanism.  Takl
is like Tak but does not perform any explicit arithmetic. 

Fprint is a file print benchmark that prints to a file, a test pattern
which is a  six level tree with a branching factor of six.  Nprint
prints the same test pattern as Fprint to a null stream in order to
measure the test pattern format time.  Tprint is a benchmark which tests
terminal output to a 10 inch by 10 inch window.  Fread measures reading
from a file.  It requires the existence of Fprint.tst which is created
by Fprint.

Div2 is a benchmark which contains a recursive as well as an iterative
test that divides by 2 using lists of n Nil's.

Frpoly is a benchmark from Berkeley that computes powers of particular
polynomials represented as lists.

\section {The Interactive Benchmarks}       

The INTERACTIVE benchmark suite was designed to simulate the real world
environment of a typical development session.  In this suite, each
benchmark is the execution of a predefined system function or an
application.  The basis for the benchmark suite, the text file SCRIPT,
is read one top-level form at a time by the performance software.  Each
form defines a benchmark containing its name and the function or
application to execute.  The benchmark is executed by stuffing the cdr
of this form, into the keyboard buffer of the active window.  The
performance software times each benchmark's execution and records the
results in the benchmark's history structure.

The first series is the ''select-create'' benchmarks, which select
various applications in order to measure initial environment creation.
The following applications are included in this series : converse,
inspector, lisp listener, peek, backup, font editor, telnet, vt100, and
zmacs editor.  

The next series is the ''select'' benchmarks which execute the same
applications as the select-create benchmarks.  The purpose of this
series is to measure the loading performance of previously instantiated
windows.  Once loading performance has been measured, the ''break''
benchmarks moniter entering the break mode in the editor and lisp
listener.  Then performance for entering the error handler is measured
by benchmarks that force an error and make a call to ''ferror.'' 

The next series is designed to test ZMACS performance.  One benchmark
edits a nonexistent file.  Others consist of variations of file copy,
executed in order to measure remote to remote, remote to local, and
local to local file copy performance.  Another edits a new file and
enters text from the script file into the new buffer.  This benchmark is
followed by an apropos on the word ''comment''.  The following group
measure performance of a cycle of file load, compile and save.  This
cycle is executed for a remote file and then for a local file.  Then a
group designed to test source compare is executed.  Benchmarks in this
group include: a mark region / kill region, a source compare, a source
merge compare by form, a source merge compare by text, and finally an
edit buffers followed by a yank region.  Next, some additional file
benchmarks are executed that measure local and remote file deletions,
expunges and dired's.  The final group of edit benchmarks measure
scrolling, meta-point, setting major and minor modes, and listing and
editing buffers.  

The final series of Interactive benchmarks is executed in the Lisp
Listener.  First a simple function, Tak, is typed into the buffer,
evaluated and then compiled and executed.  Finally hostat, print-disk-
label and print-herald benchmarks are executed.

\section {The Zippel Benchmarks}

The ZIppel Benchmark Suite consists of two subsuites, Zippels and
Extensions.  The Zippels consist of eight groups of benchmarks including
Instructions, Calls, Arithmetics, Arrays, Lists, Strings, Flavors, and
Pathnames.  The Extensions consist of seven groups of benchmarks
including Resource, Hash, Graphics, Array-Push, Eval, Branch, and
Binding.  

The Instruction Benchmarks perform basic push, pop, load, and store
functions.  These basic operations include : push an immediate constant,
push a special variable, push an instance variable, pop a value off the
stack, store into a special variable, and store into an instance
variable.  The timings for these benchmarks are used as normalization
values for the other zipple benchmarks.  One or more of these timings
are inherent in various benchmarks and are subtracted from their results
to improve accuracy.  This issue is discussed futher in section 8.2,
Zippel Benchmark Definition.

The Call Benchmarks perform basic function calling operations with
variations on the number and type of arguments supplied, as well the
number of values returned and whether they are stored or ignored.  Each
basic function call defines a Zipple benchmark.  Variations in this
group include: benchmarks that perform function calls with zero to seven
arguments and \&rest arguments, those that return zero to three values
which are ignored and one to four values which are stored, those that
make calls with one to three optional arguments, some of which are
supplied and others ignored, and finally one that does a three keyword
argument function call.

The Arithmetic Benchmarks perform timings of arithmetic functions.  This
group includes basic addition, subtraction, multiplication, and division
of fixed, float, and big numbers.  More complex operations include
Wallis, Fibonacci, and Factorial functions.  There are also polynomial
operations such as exponentiation of univariate and multivariate
polynomials, and greatest common divisor of a multivariate polynomial.

The Array Benchmarks perform basic array access and store operations.
There are benchmarks that perform references and stores for one and two
dimensional arrays and array leaders, and for one dimensional displaced
arrays, bit arrays, 8 bit arrays, and flonum arrays.  The final array
benchmarks perform reference, store, and setf on an array leader.

The List Benchmarks perform storage allocation and list access
operations.  The first benchmarks include a cons of two integers, and
the list operation on 5,10,15 and 20 elements.  Other benchmarks include
car, cdr, cadr, rplaca, and rplacd on a list of 3 elements, nth 10,
nthcdr 10, last, nreverse, and reverse on a list of 26 elements, and
nconc, append, memq, member, assq, assoc, equal, and mapcar on lists of
sizes 4 and 26. 

The next group of Zipples, the String Benchmarks measure the performance
of string manipulative functions.  This group includes three subgroups
of benchmarks, string, print, and read.  The string benchmarks include
string-search, string-equal, string=, string-lessp, string<,
string-search-char, string-search-not-char, substring, nsubstring,
string-up-down-case (which performs a string-down-case followed by a
string-up-case), and string and intern of a symbol.

The Print Benchmarks measure the performance of various print operations
and are all executed to a null stream.  The first benchmarks execute
format using both princ and prin1.  The next series execute prin1 for
the following data types : symbol, symbol with package, fixnum, flonum,
bignum, rational, string, and list.  The final benchmark does a
grind-top-level on a defun.

The Read Benchmarks, read (via the read function) various s-expressions
from a stream whose output is bound to a string.  These s-expressions
include the following types: string, symbol, fixnum, flonum, list, and
defun. 

The flavor benchmarks contain several groups that measure the
performance of various flavor mechanisms.  The first group is made up of
normalization benchmarks which compute normalizations for function
calls, get-handler-for, and Deamons.  The second group contains message
passing and method combination benchmarks.  The third group contains
benchmarks which measure accessing instance variables.  The fourth group
contains the Defflavor benchmarks.  These include simple Defflavors,
Defflavors varying the number of mixins, and Defflavors with gettable,
settable, and initiable instance variables.  The fifth group measure
instantiating the various flavors defined by the Defflavor benchmarks.
The final group of flavor benchmarks measure the performance of
Defmethod, defining primary, before, and after methods.

The final group of the Zipples category, pathname benchmarks, measures
the performance of various operations which use logical pathnames.  The
benchmarks in this group include Parse-pathname, Set-default-pathname,
Merge-pathname-defaults, Complete-pathname, open local file for output,
open local file for input, close a local input file, and close a local
output file. 
   
The first group of the Extension category, resource benchmarks, includes
benchmarks that measure Using-resource, Deallocate-resource, and
Allocate-resource.  

The Hash benchmarks includes benchmarks that measure Remhash, Swaphash,
Gethash, and  Puthash for a hash table of size 1000.

The Graphic benchmarks measure the performance of various graphic
operations by sending the operation to a window bound to terminal-io.
The benchmarks in this group include Draw-String, Draw-Char, Draw-Line,
Draw-Filled-Circle, Draw-Hollow-Circle, Draw-Triangle, Draw-Rectangle,
and Bitblt, a bit block transfer operation.

The Array-Push benchmarks measure the performance of the functions
Array-Push-Extend and Array-Pop.

The next group, the Eval benchmarks, measure the performance of the
functions: Eval, Symeval, Fsymeval, eval of 0,1,2 and 7 argument
functions, and eval of 2,3 and 7 functions.

The Branch benchmark measure the performance of Branch not nil and
Branch fallthrough.

The Bind benchmarks measure the performance of Bind and Closure.

\chapter {Performance Data Structure}

\section {Introduction}
   
The performance data structure facilitates running benchmark suites and
saving the results, massaging selective benchmark results in order to
reduce noise errors, and comparing suites ran on different machine
configurations.  Each benchmark code and results is organized into one
or more classes as specified by the benchmark definitions.  The
benchmarks in each class contain the results for all known machines and
each machine contains descriptive environment information for that
configuration.

The top level of the data structure is the special variable {\bf
*ALL-BENCHMARK-CLASSES*} whose value is the names of all known benchmark
classes.  (Classes become known through {\bf Make-System} or loading a
results file.)  The value of each class is the list of benchmark names
belonging to that class.  Finally, the property list of each benchmark
name contains pairs of the property : machine name - benchmark history.
The benchmark history associated with benchmark B and machine M is the
data set collected when running B on M.  A new member is added to the
set for each iteration of B.  The complete data set is known as the
benchmark history.  This property is the heart of the data structure and
is discussed in the next section.  An additional property that exists
for Gabriel and Zipple benchmarks is : benchmark - benchmark-code.  The
value of the keyword benchmark is the compiled benchmark code.  This
property is also discussed in an additional section.

\section {Machine Name - Benchmark History}

In addition to serving as the indicator for the Machine Name - Benchmark
History property, Machine Name's property list contains all of the
environment information collected for that machine.  The property list
currently consists of one property, which has the keyword {\bfHerald}.
The value of Herald is an a-list with the seven following keys: User,
Time, Herald, System, Machine, Room, and Disk-Label.  The respective
values of the seven keys are as follows: the logged-in user when the
benchmarks were ran, the universal time the benchmarks were ran, the
Print-Herald results, the software-version for the environment in which
the benchmarks were ran, the machine-version for the environment in
which the benchmarks were ran, the Room results, and the
Print-Disk-Label results for each known unit in the disk system.

The value of Machine Name, Benchmark History, is a structure that
contains the performance data collected when running this particular
benchmark on this particular machine.  The structure is the list
consisting of the following items: Name, Pretty-Name, Count,
Un-Normalized-Time, History, and Plist.  {\bfName} is the symbol name of
the benchmark and is used in all references to the benchmark - both code
and data.  {\bfPretty-Name} is the name used when printing benchmark
results.  {\bfCount} is the number of iterations the benchmark has
executed.  {\bfUn-Normalized-Time} is the total realtime the benchmark
executed, without any normalization values subtracted.  {\bfHistory} is
a structure consisting of the following items, by default:  Real-Time,
Disk-Time, Page-Faults, Consing, Cpu-Time, and Paging-Time.  

The history items discussed thus far are the results of the meters
collected by default by the support software.  Additional values for
meters specified by the user will also be stored in this history list.
They come after the default meter values, in the order and format
specified by the user.  Each user-specified item in History must be
preceeded by a identifier keyword.  Not only does this facilitate
interpreting the data structure but it is needed by the reporting
software to identify the items.  The software knows the position of the
default meter values, therefore those value always have Nil keywords.

{\bfPlist} is a list that contains keyword - value pairs.  The value of
keyword Normalization is the list of normalization benchmarks that apply
to this particular benchmark.  (Normalization is discussed in the
chapter ''Defining New Benchmark Suites'', in the section ''Zippel
Benchmarks''.)  The value of keyword Classes is the list of benchmark
classes to which this benchmark belongs. 


\section {benchmark - benchmark-code}


Another property that exists on the property list of benchmark name for
Gabriel and Zippel benchmarks is benchmark - benchmark-code.  The
keyword is benchmark and its value is a dtp-closure.  This property is
where the compiled benchmark code resides; it is common to all machines
known in the system.  Notice that each machine name resides on this
property list also, as the designator of its history for this benchmark.
Since the Interactive benchmarks are created from a script file, they do
not consist of compiled code; therefore they do not possess this
property.

  
\chapter {Performance Testing Methodology}

This section is intended to provide a sound starting point for doing
serious performance anaysis of the system.  It is not possible to cover
all variables that could affect the results of a test, therefore anyone
doing testing must take great care to control any additional factors
that are particular to the environment being tested.

At all times keep in mind that test results are {\underline extremely}
sensitive to the environment in which they are run.  Careful following
of these guidelines is necessary to ensure repeatability of
measurements, which in turn allows for a productive discussion of the
results.  The exact methodology will change as the system evolves,
therefore only general guidelines are appropriate in this document.

Start from as clean an environment as possible, preferably a cold boot,
which will cause the fewest number of possible side-effects.  Since it
is not always possible to start from a cold boot, load the minimum
amount of changes to the system to support the test, and carefully
document the details about your modifications.  In addition, archive the
following information with the data:

\begin{itemize}
\item (PRINT-HERALD) to document patch level of all software pieces,
	 total amounts of physical and virtual memory, and
	 machine on which the test was run.
\item (ROOM)  Additional virtual memory information.
\item (PRINT-DISK-LABEL) for every disk pack on the system.  This 
	 documents exact system, paging, and file system
	 layout.
\end{itemize}

In addition, document detailed information about any customizations
included in login files, or their equivalents.

The timing function used should wrap the reading of clocks and meters as
well as the test function in a (WITHOUT-INTERRUPTS) form.  Whenever
possible tests should be run with the network disabled (CHAOS:DISABLE)
in order to eliminate the effect of random network traffic processing
from the results.

\chapter {Running Benchmark Suites} 

This chapter will discuss the method for running the standard benchmark
suites, Gabriel, Zippel, and Interactive.  Although the procedure is
very similar for each of the suites, there are some minor differences;
therefore the general method will be explained first, and then the
issues unique to each suite.  Before running a benchmark suite on your
system, it is most important follow the guidelines presented in the
chapter, Performance Testing Methodology, in order to ensure data
integrity. 
 
Once you have created the environment in which you wish to run a
benchmark suite, load the main defsystem from its host system by
executing {\bf(Load ''host:bench;defsystem'').} This is the only file
required to get you started.  The defsystem defines the
logical-pathname-host, specifying the host as {\bf''Bench:''}, the
performance software directory as {\bf''Bench:bench;''} and the
benchmark result files directory as {\bf''Bench:br;''.}

Once the defsystem is loaded, the easiest way to run any of the
benchmark suites is to execute {\bf(Do-Suite\_Name 'machine)} where
Suite\_Name is the name of the suite to be run, and machine is a
descriptive name for the environment which has been booted.  For
example, executing (Do-Gabriel 'fan16-lru) would run the Gabriel
benchmark suite on a system booted with fan16 load band and lru
microcode band.  

The Do-Suite\_Name function performs a Make-System on the benchmark
suite and runs the benchmarks specified within.  Once the final
benchmark has finished running, all of the collected data is written to
the file {\bfMachine-Suite\_Name.Bench.} For example, if the previous
Gabriel example were run, the results would be stored in the file
''fan16-lru-gabriel.bench''. 
 
The Make-System call loads (compiling if necessary) the benchmarks and
other performance support files listed in the file {\bfSuite\_Name-
defsystem.} For the three standard suites, Gabriel, Zippel, and
Interactives, the defsystem file name would be Gabriel-defsystem,
Instruction-defsystem, and Interactive-defsystem, respectively.  
															
Once the Make-System has completed, Do-Suite\_Name performs
{\bf(Run-Suite\_Name machine)} which executes the benchmarks in batch
mode, looping a fixed number of times.  Data is collected from meters
for real time, disk time, paging time, number of page faults, consing
time, and cpu time.  These are the standard meters which are collected
by default in the performance data structure.  If desired, you can
specify additional meters that should be collected while the benchmarks
are executing.  For specific details on how to accomplish this, refer to
chapter 8, Defining New Benchmark Suites.  

The results file, created by the Run-Suite\_Name function call to
Save-Results, consists of a machine property designating the type of
Lisp machine on which the suite was run, information documenting the
environment in which the suite was run, and the test results.  The
documenting information consists of the user identification, the
universal time the benchmarks ran, the print-herald information, the
software-version and machine-version for the environment in which the
benchmarks were run, the Room results (called with no arguments), and
the disk label.  Test results consist of a record for each benchmark in
the suite, containing the data collected for that benchmark.  The result
record is discussed in detail in chapter 3, Performance Data Structure.

\chapter {Data Handling Techniques}

This chapter will discuss the performance routines that are available
for improving the accuracy of the data that has been collected through
running a benchmark suite in a specific environment; there are several
situations in which you may need to do this.  Often data obtained from
one or more of the benchmarks are questionable and you may want to rerun
those particular benchmarks again, replacing the questionable data with
the newly obtained results.  Other times you may feel the standard
deviation of the data collected for a given benchmark is too large and
that you need to collect additional data in order to stabilize results
for that benchmark.  Finally, you may want to run some additional
bench-marks, appending those results to previously collected data.  Part
of the {\bf Run-Benchmarks file}, the routines available for
accomplishing these tasks include {\bfPerform-Benchmarks, Save-Results,
Update-Results}, and {\bfRestore-Results.} Another routine,
{\bfGenerate-Benchmark}, can be used to create compatible benchmark
result records from data available from hard copy only.

When you want to run one or more selected benchmarks, the performance
routine to use is {\bfPerform-Benchmarks.} Perform-Benchmarks has four
parameters which are defined as follows : (Benchmarks \&key (Repcount
10.)  (Report-P t) (Clear t)).  {\bfBenchmarks} is the list of names of
the benchmarks you want to run.  {\bfRepcount} is the number of times to
run each benchmark before the next one is run.  (More accurately, the
number of times a given benchmark is run will be the difference between
repcount and the count of the number of runs previously recorded for
that benchmark.)  Once the final benchmark has completed, {\bfReport-P}
if true will cause Report-Benchmarks to print to the terminal the
results of all the benchmarks named in Benchmarks.  If true, {\bfClear}
will forget any result data it currently knows about for all the
benchmarks named in Benchmarks. 

Once the desired benchmarks have been rerun, you can use the function
{\bfUpdate-Results} to replace the old results with the new results.
The function has three parameters which are defined as follows :
(Pathname \&optional (Benchmark-List *All-Benchmarks*) (Name
My-Benchmark-Property)).  {\bfPathname} is the name of the file where
the old results are stored; the new results will be stored in a
higher-version-numbered file with the same name.  {\bfBenchmark-List} is
the list of names of benchmarks for which results are to be updated.
This will generally consist of the names of the benchmarks that were
rerun, however the default, *All-Benchmarks* consists of all the
benchmarks that have been run (or loaded via Restore-Results) in the
current environment.  {\bfName} is the benchmark property of the machine
on which the benchmarks were run.  The default, My-Benchmark-Property,
for an Explorer would be Explorer-Benchmark. 

{\bfUpdate-Results} writes the environment documentation from the old
result file and for the current environment to the new result file.
Next the function reads in the benchmark records of the old result file
and writes them to the new file, discarding any record for a benchmark
named in Benchmark-list.  Once the old result file has been written, the
results of each benchmark named in Benchmark-List are appended to the
end of the new result file.  

{\bfSave-Results} is similar to Update-Results except there is no old
result file from which previous benchmark records are read.  The
parameters are the same as those for Update-Results and are used in the
same fashion.  Save-Results writes the current environment documentation
to the result file and then the results of each benchmark named in
Benchmark-List.  

{\bfRestore-Results} is the function to use for loading benchmark result
files.  It has two parameters which are defined as follows: (Pathname
\&optional Machine-Name).  The function loads each benchmark record in
the result file specified by {\bfPathname.} and stores it under its
benchmark name, a field from the benchmark record.  All of the benchmark
names are stored under {\bfMachine-Name.}  If the machine name is
omitted, it defaults to the name stored in the result file.  If previous
benchmark records exist for that machine name, restored records replace
old records with the same name, but old records that do not get replaced
still exist after Restore-Results completes.

{\bfGenerate-Benchmark} is a routine that enables you to create a
benchmark structure from benchmark records in a text file and optionally
save the results to an output file.  This routine facilitates comparing
your own benchmarks to those from other sources, because it puts raw
data into a compatible result form.  Once this is accomplished, you can
use the reporting software to compare these new benchmarks to your own.
The parameters for the routine are as follows : (Machine Class-List
Input-Pathname \&optional Output-Pathname).  {\bfMachine} is the name of
the machine under which the benchmark records are to be stored.
{\bfClass-List} is the name of the benchmark suite to which the raw data
belongs, for example, Gabriel-Benchmarks.  Input-Pathname is for the
text file containing the raw benchmark data.  The records in this file
must be in the format : benchmark-name cpu-time-in-seconds.  If
Output-Pathname is supplied, the benchmark records generated from the
raw data will be saved to this file.  You can use Save-Results to save
the results at a later time, if so desired.

\chapter {Reporting Results}

\section {Introduction}

The performance reporting sofware provides three levels of flexibility
in generating detailed reports from previous runs of benchmark suites.
At the top level, there are standardized reports for the standard
benchmark suites, which report details of collected meters for
individual machines and comparisons between two or three machines.  At
the second level, you can define and arrange personalized reports of
factual and comparison information from a multitude of predefined
columns.  (The standard reports were defined in this manner.)  At the
lowest level, you can define your own columns for a given report, which
can be mixed with any previously defined columns.  You may need to
define new columns in two situations.  The data is there but you want to
report it differently.  You have defined new meters and now need to
report the newly collected data.  

There are several analyze functions that the reporting software provides
for reporting data.  All of the default meter values and the user
defined meter values can be accessed and reported in a variety of
statistical formats through the reporting software.  These issues are
also discussed in this chapter.

\section {Initial Setup}

The report software gets loaded on two occasions: in conjunction with
loading the runtime software for executing a benchmark suite, and
independently when the runtime software is not part of the environment.
If you have already ran a benchmark suite in the current environment,
then the report software has been loaded.  Otherwise you must load the
report software by loading the defsystem and doing a Make-System.  The
specific steps are as follows: 

\begin{itemize} 
\item (Load
''lm:bench;defsystem'') 
\item (Make-System 'Reports) 
\end{itemize}

The directory where the report software expects the benchmark data files
to be is Bench:br; where Bench is defined as the logical host name.  All
of the reports are printed to the terminal so it is necessary to dribble
them to an output file if you want to save them for later reference.
Try to pick a descriptive dribble filename that will indicate the type
of benchmark suite ran and the machines that were compared.  For
example, a good filename for a Gabriel report of two machines, Symbolics
with system 6.0, and Explorer with fan20 might be,
''Sym6-ExpFan20-Gabriel.Report''.  Often the report format will be wider
than 80 columns, so it is best to use a printer that supports
landscaping or compressed print.  The Imagen font 5X5 can easily print
ten columns of information in portrait mode.

\section {Standardized Reports}

The correct syntax for executing one of the standardized reports, is
{\bf(Report-Suite\_Name Machines Reports).} For the Gabriel and
Interactive suites, {\bfSuite\_Name} is Gabriel and Interactive,
respectively; for the Zippel suite, there is a report for each subsuite,
Zippel and Extensions, with corresponding Suite\_Name.  {\bfMachines} is
the required list of one, two, or three machine names, the name
associated with the data file that contains the results of running a
given suite on that machine.  (Perhaps ''system'' would have been a
better nomenclature.)  For example, if Report-Gabriel is given the list
'(Exp172) , it knows to read the data file
''Bench:Br;Exp172-Gabriel.bench''.  

{\bfReports}, an optional parameter, is a list of names of reports that are
to be generated for two machines.  Report-Suite\_Name uses a fixed set of
reports for one or three machines.  Report-Gabriel for example, uses the
default reports Gabriel-Compare Realtime-Compare Cpu-Compare and
Paging-Compare for two machines.  For one machine it uses the reports
Gabriel-Report and for three machines, the reports Gabriel-Compare3,
Realtime-Compare3, and Paging-Compare3.  Each of these reports is
comprised of standard report columns which provide an easy and flexible
method for defining and arranging your own reports.  

The standard categories made available by the report software include
the following : cpu time, real time, disk response time, paging time,
number of page faults, and number of consings.  For each of these
categories it is possible to report, based on the data stored for each
repetition of the given benchmark, mean, median, standard deviation,
minimum, and maximum.  You can also report percentage changes and ratios
of these statistical items for any two machines.  The cross product of
the standard categories and the statistical items listed above,
benchmark name, and number of repetitions performed, makeup the
predefined columns available for building reports.

\section {Defining Reports From Predefined Columns}

The second level of flexibility, designing reports using predefined
columns, can best be explained by way of example.  Let's take a closer
look at a dummy report called My-Compare.  The report prints factual and
comparison information for two machines and is defined as follows:

\begin{verbatim}
   (DEFREPORT My-Compare
    	  short-pretty-name 1 count
	  1 cpu-min 2 cpu-min 
	  (compare-1-2 cpu-min) 
  	  blank-3 1 time-min 2 time-min) 
\end{verbatim}

{\bfDefreport} is a macro that puts the name 'My-Compare in the list
*All-Reports*, the list of all known reports.  It then defines a
constant, {\bfMy-Compare}, whose value is the list of symbols and lists
appearing in the Defreport.  The symbols and symbols in each list are
defined by entries in the constant {\bfReport-Types.} A symbol that is
referenced in a defreport but not defined in Report-Types consitutes an
error.  Each entry (symbol or list) in the Defreport defines one column
of the report, except for numbers which are used to switch machines
being reported on.  A list references two entries in Report-Types and
defines a column for the car of the list based on the value of the cdr
of the list.  In this example, (compare-1-2 cpu-min) would define a
column comparing machine one to machine two on minimum cpu time.

The report that would be generated by the call
(Report-Gabriel '(Exp172 Exp200) 'my-compare) would be as follows
(assuming that the only known Gabriel benchmarks were boyer and fft) :

\begin{verbatim}
Benchmark Results    TI internal data    4/07/86 14:46:46    Page1    

EXP172          EXP EXP172    EXP200             EXP172    EXP200    
                Num   Min       Min     2 rel     Min       Min     
Benchmark Name  Run   Cpu       Cpu      to 1   RealTime  RealTime  
GABRIEL-BOYER    5  26.62 s   26.78 s    0.99   26.62 s   26.78 s   
GABRIEL-FFT      5  23.21 s   23.23 s    1.00   23.21 s   23.23 s   
\end{verbatim}

If you wanted to produce a new type of report for the Gabriel suite, the
above procedure would work just fine.  However, it is more likely that
you will want to produce new report types for your own benchmark suites.
In addition to writing the Defreports you will need a procedure very
similar to Report-Gabriel in order to make use of your Defreports.  For
this reason, we need to look at the Report-Gabriel function. 
  
Report-Gabriel calls Load-Reports with Machine\_names and Suite\_name
which will load the data files defined by the two parameters.  For
example, if the call (Load-Reports '(Exp172) 'Gabriel) is made, the
function will read the data file ''Bench:Br;Exp172-Gabriel.bench''.
Load-Reports calls Restore-Results for each machine in Machine\_names.
Restore-Results loads the results for each machine, updating the global
variable *gabriel-benchmarks*, which will contain a list of the names of
all known Gabriel benchmarks.  Report-Gabriel then calls Run-Reports
with the parameters (*gabriel-benchmarks* Machine\_names Report1
\&optional Report2 Report3).  The first two parameters have been
discussed.  Report1, Report2, and Report3 are lists of Defreports to be
produced for each machine in Machine\_names.  When there are one, two,
or three, machines, Run-Reports will call Report-Benchmarks for each
report in Report1 Report2 or Report3, respectively.  Report-Benchmarks
will print each report for each benchmark whose name appears in
*Gabriel-Benchmarks* if every machine has a record for that benchmark.
If one or more machines had no data for a given benchmark, that
benchmark is excluded from the report and a message appears at the end
of the report. 


\section {Defining New Report Columns}

	   
The lowest level of flexibility allows you to define new columns in
addition to the standard columns available for building report formats.
Once the new columns have been defined, you can use them in a Defreport
as you would standard columns.  

Each Defreport entry's underlying format and structure is defined
through an entry by the same name in Report-Types.  Each entry in
Report-Types is an alist of the form : (Name Format-string Top-header
Bottom-header Analyze-Function .  Args).  

{\bfName} is the report-type name used to reference the report type in a
Defreport.  {\bfFormat-string} is a format control string used for
printing the headers and results.  {\bfTop-header} is a string used as
the top line of the two line label.  {\bfBottom-header} is a string used
as the bottom line of the two line label.  {\bfAnalyze-Function} is the
function whose result is printed with Format-string as the report value.
Function is called with arguments Benchmark .  Args, where Benchmark is
the benchmark structure object being reported.  If an item in a
Defreport is a list, for example (compare-1-2 cpu-min), then the cdr of
the list is added to Args.

For example, two entries in Report-Types, cpu-min and compare-1-2, which
were used in the previous example, MyCompare, were defined as follows:

\begin{verbatim}
DefConstant Report-Types
   '((cpu-min '' ~10a'' ''  Min  '' ''  Cpu   '' analyze-time CPU Min)
     (compare-1-2 '' ~5,2f'' ''2 rel'' ''to 1'' compare-machines 1 2))
\end{verbatim}

Reporting items in the benchmark history that were additional meter
values specified by the user, is similar to reporting default meter
values; however they are a little different.  The major difference is
that the meter label must be specified so that the reporting software
will know which item you are referencing.  The item still has to be
defined in the Defconstant Report-Types, and referenced in one or more
DefReports before it can become a column of output.

There are three generic report types that were designed to facilitate
reporting user specified meter values: Get, Time, and Fix.  Get returns
a flonum value, Time returns a raw microsecond time in appropriate
units of ns, us, ms, or seconds, and  Fix returns a rounded fixnum.  These
DefConstant entries are adequate in most instances which means generally
all you have to do is make the DefReport references.

The DefReport entry must be a list whose Car is the Report-Type name and
whose Cdr is the arguments to be passed to the print function.  For the
three generic report types, the arguments are Type and Need.  {\bfType}
is the meter label that is used to specify the meter value in the
history list.  {\bfNeed} is one of the following: mean mode median
std-dev min max total size samples count real-count un-normalized-time.
It is the computation that is to be performed on this data item.

The following example shows the report specification of a meter value
that was user-defined.  The reference is to Report-Type Fix.  The meter
label in the benchmark history is :extra-pdl.  The desired Need is mean.

\begin{verbatim}
(defreport gabriel-report
	med-pretty-name count cpu-mean
	cpu-dev cpu-min cpu-max disk-max
	(fix :extra-pdl mean))
\end{verbatim}


\section {The Analyze Functions}


The reporting software includes several general Analyze-Functions that
are used by the standard reports and should cover 99 percent of all
additional user needs.  These functions are:  Analyze-History,
Average-Disk-Response, Analyze-Time, Analyze-History-Fix,
Compare-Machines, and Compare-Percent.  When using the standard
Report-Types, these functions are of no concern to the user since they
get called automatically.  However, when defining new Report-Types, it
is necessary for you to specify the analyze function.  The following
descriptions should facilitate this process.

{\bfAnalyze-History} is the heart of the analyze functions; all of the
others listed above use it as well.  Analyze-History returns a value
based on Need and Type.  {\bfNeed} is the desired statistical analysis
and can be one of the following: total mean mode median std-dev min max
samples size count real-count un-normalized-time.  To understand the
explanation of these statistical items remember that there is a history
entry for each iteration of a given benchmark on a given machine.  Each
history entry contains a value for each of the meters collected on that
iteration.

\begin{itemize} 

\item {\bfTotal} is the total of each recorded value of the specified
meter type.  For example, if there were five iterations of benchmark X,
then there will be five Cpu-Time meter values in the history structure
and these five values will be summed as Total. 
\item {\bfMean} is Total divided by the number of history items
(iterations.)
\item By definition, {\bfMode} is the meter value appearing most
frequently for the meter history.  As defined by the reporting software,
it is one-half of the sum of the smallest and largest history items.
\item {\bfMedium} is the meter value below and above which there is an
equal number of values.  
\item {\bfStd-Dev} is the statistical standard deviation of the meter
item.  
\item {\bfMin} is the minimum value for the meter item.  
\item {\bfMax} is the maximum value for the meter item.  
\item {\bfSamples} is the number of iterations the benchmark has
currently been executed on a given machine.  
\item {\bfSize} is the benchmark count divided by the number of recorded
history items.  When the benchmark is first created, the benchmark count
is specified.  Size is the ratio of this count to Samples.  
\item {\bfCount} is the product of Samples and benchmark count divided
by the number of history items.
\item {\bfReal-Count} is Count expressed as a decimal integer.
\item {\bfUn-normalized-timed} is the accumulated Real Time for all
iterations of the benchmark, without any normalization.  This number of
iterations is the product of benchmark count and Samples.

\end{itemize} 

{\bfType} is the meter type from the benchmark history to which the
statistical analysis is to be applied.  It can be one of the following:
time disk-time paging-time faults cons cpu or a keyword.  The
enumeration of each of the record values for the default types, plus
each of the keyword types with its recorded value, constitutes one
history item.  This item corresponds to one iteration of the benchmark.
If the repetition count specified when the benchmark was defined is
greater than one, then each of the default type values is divided by the
repetition count.

\begin{itemize} 

\item {\bfTime} is the Real Time in microseconds, calculated from the
counter si:%microsecond-time, that was required to execute the benchmark
one iteration.  
\item{\bfDisk-Time} is the Disk Wait Time in microseconds, calculated
from the counter si:%disk-wait-time, that was recorded while the
benchmark was executing one iteration.
\item {\bfPaging-Time} is the Total Page Fault Time in microseconds,
calculated from the counter si:%total-page-fault-time, that was recorded
while the benchmark was executing one iteration. 
\item {\bfFaults} is the Total Page Fault Count, calculated as the
sum of counters %count-disk-page-reads and si:%count-fresh-pages, while
the benchmark was executing one iteration.
\item {\bfCons} is the Total Cons Count, calculated from the function
Area-Size, that was recorded while the benchmark was executing one
iteration.
\item {\bfCpu} is the Total Cpu Time, calculated as the difference
between Time and Disk-Time, that was recorded while the benchmark was
executing one iteration.
\item{\bfKeyword} is a meter label for the meter that was specified by
the user when defining the benchmark.  The meter value immediately
follows the Keyword in the history structure.   

\end{itemize} 

{\bfAverage-Disk-Response} returns the dividend of disk-time and faults
in appropriate units of ns, us, ms, or seconds.  Disk-time and faults
are calculated by Analyze-History as described above.

{\bfAnalyze-Time} returns any microsecond time in appropriate units of
ns, us, ms, or seconds.  It gets the microsecond time from
Analyze-History based on type and need as described above.

{\bfAnalyze-History-Fix} uses the Round function to return as a fixnum,
the value it receives from Analyze-History.

{\bfCompare-Machines} returns the ratio of two recorded meter values it
receives from Analyze-History, based on type and need as described above.
Normally this function is used for comparing the same history item of
the same benchmark for two different machines.

{\bfCompare-Percent} returns the value of Compare-Machines as a
percentage.


\chapter{Defining New Benchmark Suites}

\section{Introduction}

This chapter will explain in greater detail the three standard benchmark
suites, Gabriel, Zippel, and Interactive.  The purpose of the
explanation is to provide an understanding of how each suite was defined
so that additional suites can be defined using the same methods.  Each
suite was defined from a collection of code segments that differed in
format from the others.  For this reason each type of collection
required a different method of defining it as a benchmark suite.  

The Gabriel benchmarks, which consist of self-contained functions, were
defined using the function {\bf Define-Gabriel-Benchmarks}.  The Zippel
benchmarks consist of single operations that require certain
preconditions in order to execute.  They were defined using the macro
{\bf Define-Benchmark}.  The Interactive benchmarks consist of
expressions that are stuffed into the keyboard buffer and executed.
They are defined "on-the-fly".  As each expression is stuffed in the
buffer and completes execution, the benchmark name and meter results are
recorded.  Hopefully these defining methods will be general enough to
facilitate defining most any collection of code segements into a
benchmark suite.

Reguardless of the type of benchmark definition that is used, the
support software will collect the default meters, (Disk-Wait-Time,
Paging-Time, Hard-Page-Fault-Count and Area-Size), and store the results
in the benchmark's data structure.  The default meters are collected for
each benchmark type and additional meters may be specified when using
the macro Define-Benchmark.  The data structure is identical for each
type of benchmark and is discussed in detail in the chapter, Performance
Data Structure.

\section{Gabriel Benchmarks}

The Gabriel benchmarks consist of one or more main functions and a
start-up function whose sole purpose is to execute the benchmark.
Defining them requires writing the basic code for the benchmark,
including a call to the start-up function as one of the arguments of the
macro {\bf Timer}, and specifying the benchmark to the function
{\bf Define-Gabriel-Benchmark}.  The general procedure is to write all
of the basic benchmarks first, including the Timer calls for each
benchmark's initial function, and then make one call to
Define-Gabriel-Benchmark with a list of all benchmark names and
specifications.  Since there are approximately 45 Gabriel benchmarks, it
was decided to put each group in a separate package and file.  If you
are defining a few benchmarks only, you may wish to put all of them in
the same package.

The following example is part of the code that defines the benchmark
Gabriel-Div2.  The function test1 is the top-level function for the
benchmark.  (It calls div2 which is not shown here.)  Following test1 is
the call to the macro Timer, with the macro expansion shown next to it.
Timer simply compiles the start-up function, timit1, whose sole task is
calling test1.  The macro's parameters are Name, Form and Args, where
{\bf Name} is the name of the start-up function, {\bf Form} is the call
to the benchmark's top-level function, and {\bf Args} is the arguments
the start-up function might need.  The start-up function's name is
insignificant; it is only used to specify the benchmark to
Define-Gabriel-Benchmark. 


\begin{verbatim}
 ;;;-*- Mode:LISP; Package:BENCH-DIV2; Base:10 -*-   

    (defun test1 (l)
        (do ((i 300. (1- i)))
	       ((= i 0))
	       (div2 l)
	       (div2 l)
	       (div2 l)
	       (div2 l)))
	   

  (timer timit1                       (PROGN 'COMPILE
         (test1 l))      --->           (DEFUN TIMIT1 ()
                                          (TEST1 L)))
\end{verbatim}

Once the caller function has been created by Timer, the benchmark has to
be specified to Define-Gabriel-Benchmarks which performs the remaining
housekeeping tasks, associating the benchmark code with its true symbol
name and assigning the benchmark name to the specified class.  The
following example shows the call to Define-Gabriel-Benchmarks for the
benchmark Div2 used in the previous example, and an additional benchmark
Frpoly.  This example is a little simplistic because the entire Gabriel
suite was defined with only two calls to Define-Gabriel-Benchmarks, one
for each class of Gabriels, *Gabriel-Benchmarks* and *Gabriel-Print*.

The parameters to Define-Gabriel-Benchmarks are Bench-Name Repeat-Count
and Benchmarks.  {\bfBench-Name} is the name of the class of benchmarks
to which the benchmarks will be assigned.  Benchmarks belonging to more
than one class must be specified by additional calls.  For example, the
Gabriel print benchmarks were assigned to the sub-class *Gabriel-Print*
by a second call to Define-Gabriel-Benchmarks with Bench-Name, *Gabriel-
Print*.  {\bfRepeat-Count} is stored on the benchmark symbol name's
property list as the value of property Benchmark-Repeat-Count.  The
value is not currently used by the performance software.  

The final parameter to Define-Gabriel-Benchmarks, {\bfBenchmarks}, is a
list containing three members: Repeat-Count, Function-Definition, and
GC- Flag.  {\bfRepeat-Count} applys to all of the benchmarks being
defined is this list.  The value is optional and not currently used by
the performance software.  {\bfGC-Flag} is the garbage collection flag
and has possible values of gc, no-gc, and nil.  If gc is specified, when
the benchmark has executed, a fast and dirty garbage collection will be
performed by resetting the temporary consing area.  The other two values
no-gc, and nil will prevent garbage collection from being done.  

{\bfFunction-Definition} is a list containing three members as well.
The first member is the true name of the benchmark; the name by which
all references to the benchmark are made.  This benchmark name is added
to the class name for the group of benchmarks (the parameter
Bench-Name.) The second member is the function name that was given to
the macro Timer when the benchmark code was written.  The third member
is the parameters to be passed to this function, when it is executed.


\begin{verbatim}
    (define-gabriel-benchmarks '*gabriel-benchmarks* 1
      '(5
	   (gabriel-div2-1 bench-div2:timit1) gc
	   (gabriel-frpoly-1-10 bench-frpoly:timit1 10) no-gc))
	
\end{verbatim}

\section{Zippel Benchmarks}

The benchmarks in each of the Zippel groups consist of single operations
(such as function calls or flavor instantiations) that need additional
code to control the environment in which they execute.  For this reason
each benchmark is defined using the more versatile macro, {\bf
Define-Benchmark}.  The macro expands each benchmark into a
self-contained function, providing code for controlling page-faults,
executing normalization functions, determining which meters are
collected while the benchmark is running, and cleaning up after the
benchmark terminates.  In addition it provides code for saving the
collected meter values. 

The parameter list for Define-Benchmark is (name pretty-name repcount
\&body others).  {\bf Name} is the name of the symbol containing the
expanded benchmark.  The benchmark code is stored on Name's property
list as the value of property Benchmark.  Name is used in all references
to the benchmark, both for the execution and the report functions.
{\bfPretty-name} is the print name associated with the benchmark and is
the name printed with the benchmark results.  

{\bf Repcount} is the number of times that the benchmark code, defined
in :body or :real-body, is to be executed.  This can be confusing
because you also specify the number of iterations at the top level, when
executing the benchmarks.  Repcount defines the number of ''internal''
iterations, which is executed for each iteration specified at the top
level.  All of the default meter values collected are the sums over all
of the internal iterations.  The values recorded in the history
structure are first divided by Repcount to get the most accurate value
for a single iteration.  In the following example, Fixnum-Addition,
Repcount is specified as 3000.  This means that if you execute
Run-Benchmarks on this benchmark with a loop count of 1, it will execute
3000 times.  The meter values recorded for this one iteration will first
be divided by 3000.

{\bf Others} consists of one or more keyword/value pairs, where the
legal keywords are: :classes :body :bindings :declarations :real-body
:cleanup-form :normalization :meters and :allow-page-faults.These
keyword/value pairs provide the flexibility for executing the benchmark
in a specifically controlled environment.  To better explain the format
and purpose of these pairs, let's look at the definition of the
benchmark Fixnum-Addition.

\begin{verbatim}
(define-benchmark Fixnum-Addition ''Fixnum Addition'' 3000
  :classes (*zippel-benchmarks* *fixnum-arithmetic* *arithmetic*)
  :allow-page-faults nil
  :normalization (+(normalization push-local)(normalization store-local)) 
  :bindings ((v1 val1) (v2 val2) result)
  :body (setq result (+ v1 v2)))
\end{verbatim}

The value for keyword {\bf :classes} is a list of global names to which
the symbol Name will be added.  Classes make it easy to organize the
benchmarks into one or more groups for convenience in running and
reporting results for related benchmarks.  In the example,
fixnum-addition belongs to three classes which makes it easy to run or
report the entire suite of Zippel benchmarks or only one of the
subgroups.

When the value of keyword {\bf :allow-page-faults} is true,
Define-Benchmark adds no additional code for controlling page faults.
However, when the value is nil the macro inserts code to keep executing
the benchmark until no paging occurs during the current execution.  Only
the meter values recorded during this final execution are returned.  
	  
Normalizations are benchmarks themselves, and are run in the same
environment as the rest of the benchmarks.  The purpose of normalization
is to measure operations that are inherent in certain types of
benchmarks.  For example, pushing a local value onto the stack is an
inherent operation in all arithmetic operations.  Its performance is
measured in the normalization benchmark, push-local, and the result is
used to normalize the results of all arithmetic benchmarks.  In our
example, the normalization benchmark store-local is also specified,
because the code contains a setq which is an inherent operation.  We are
not interested in the setq time, only in the fixnum addition time. 

Normalization benchmarks are defined using the same macro as the other
Zippels, Define-Benchmark.  By convention, they are designated as such
by adding ''-normalization'' to the symbol name.  For example, the
store-local normalization benchmark is designated
Store-Local-Normalization.  All of the normalization benchmarks should
be ran before the other benchmarks are executed.  As each benchmark
completes execution, the normalization value will be subtracted from the
benchmark's recorded real-time and cpu-time.

The value of keyword {\bf :bindings} is a list of variables and initial
binding that will be needed in the benchmark code.  The Define-Benchmark
macro simply binds them using a Let statement.

The value of keyword {\bf :declaration} is a list of declarations to aid
in compilation or optimization.  The Define-Benchmark macro simply uses
these as arguments of a Declare statement.

The argument {\bf :meters} provides a method of specifying meters that
you want to collect in addition to the default ones.  The default meters
include Disk-Wait-Time, Paging-Time, Hard-Page-Fault-Count and
Area-Size.  It is often desirable to collect additional information,
such as :extra-pdl data.  There are approximately 40 meters that are
defined in the Explorer system and can be collected by the performance
software.  See the comments concerning the DefSysConst
A-Memory-Counter-Block-Names in Run-Benchmarks for a listing and
description of these meters.

The following example, Slinghash, shows the format for specifying the
:meters argument.  Each sublist, which specifies one meter, consists
of three members: meter-label, meter-form and difference-function.  The
meter list may consist of as many sublist as required.  

{\bfmeter-label} is a required keyword for identifying the value that is
saved for this meter.  The default meters always use nil as the label
for their values, because their position in the history list is known at
compile time.  The example shows the specification of two meters.
Labels not only facilitate looking at the data structure, but are needed
by the reporting software to distinguish multiple meter values.

{\bfmeter-form} is the form that should be used for collecting the
meter.  For the meters known in A-Memory-Counter-Block-Names, the form
will always be (si:read-meter counter-name).  The Define-Benchmark macro
expands the code to collect the measurement once before and once after
the :body code.  

The latter collected value will be subtracted from the previous value
once the benchmark has executed, using the third argument,
{/bfdifference-function}.  The default one is minus (-).  For time
values, microsecond-time-difference is used to compensate for the
potential clock wrap around, as shown for the second meter in the
example.  The result of the subtraction is the value that is saved in
the history list along with the optional meter-label. 


\begin{verbatim}
(define-benchmark slinghash nil 500
  :classes (*extended-zippels* *hash-benchmarks*)
  :bindings ((var 0) (hash-table test-hash-table))
  :meters ((lru (si:read-meter 'si:%least-used-page)
	         nil)
           (nil (si:read-meter 'si:%total-page-fault-time)
	        microsecond-time-difference))
  :body
  (GETHASH (SETQ var (1+ var)) hash-table))
\end{verbatim}


If there is a meter that you plan to include for all benchmarks, you may
wish to set it as a default meter.  This will prevent you from having to
specify it with the :meter argument for each benchmark definition.  You
can change the default collected meters by modifying the global constant
{\bfBenchmark-Meter-List}.  Each element of Benchmark-Meter-List is a
sublist with three members : Meter-Label, Form, and Difference-Function.
The format and functionality of the constant is identical to that of the
:Meters argument used in Define-Benchmark. 

The chapter, Performance Data Structure, gives more information on the
format of the history list.  The chapter, Reporting Results, explains how
to access user specified meter values using the performance reporting
software.
	
The value for the keywords {\bf :body} and {\bf :real-body} contains the
section of code or operation which Define-Benchmark is to expand into a
benchmark.  The two keywords are mutually exclusive; only one of them
should be specified in a given definition.  The value of either of the
two keywords can only contain a single top level form (which could be a
Progn if necessary.)  

{\bf :body} is used when defining less complex benchmarks that need no
initialization code.  With this keyword specified, Define-Benchmark will
expand the benchmark to execute Repcount times, collecting the default
and additionally specified meters.

{\bf :real-body} is used when you need greater flexibility than :body
allows.  Using this keyword allows you to specify initialization code
that you want to be executed prior to the actual benchmark code, and not
to be included in the metering.  The following example of the Bitblt
benchmark shows how :real-body is commonly used.  The call to
tv:prepare-sheet is initialization code that will be executed before,
and excluded from the metered code.  Benchmark-body is the
performance macro that provides code to execute the benchmark (in this
example, the call to Bitblt) the number of specified times, and collect
the default and additional meters.  Define-Benchmark uses this macro
when :body is specified.

\begin{verbatim}	   
(define-benchmark bitblt "Bitblt a 100 x 100 square" 200
  :classes (*extended-zippels* *graphics-benchmarks*)
  :bindings ((scrn-ary (SEND tv:main-screen :screen-array)))
  :cleanup-form (SEND terminal-io :set-char-aluf alu)
  :real-body
  (tv:prepare-sheet (tv:main-screen)
    (benchmark-body (200)
      (BITBLT tv:alu-xor 100 100 scrn-ary 10 10 scrn-ary 200 200)))) 
\end{verbatim}

The value of the final keyword, {\bf :cleanup-form}, is the form to be
executed once the benchmark has executed, and all meters have been
collected.  In the previous example the clean-up form will set the alu
argument for terminal-io back to the value of alu.  

Once the code section has been expanded, Define-Benchmark puts the
benchmark on the property list of the symbol Name as the value of
property Explorer-Benchmark.  This data structure is discussed in detail
in the section, Performance Data Structure.  When Perform-Benchmarks is
called with the list of benchmarks to be executed, it gets the benchmark
code from each benchmark name's property list.  

\section{Interactive Benchmark Definition}

Unlike the Gabriel and Zippel benchmarks which are compiled code, each
Interactive benchmark is an s-expression which is stuffed into the
active keyboard buffer and interpreted.  The s-expressions are stored in
a script file which is read one form at a time.  Once the benchmark has
completed execution, the results are stored on the benchmark name's
property list.  The code for the benchmark is not stored as it is for
the other suites.

Each s-expression in the script file defines an Interactive benchmark
and simulates a single action a user would perform, such as ''meta-x
compile buffer''.  Each one must be of the following list form:
(Test-Name Strings \#/Char Symbols).  {\bfTest-Name} is the name to be
associated with the benchmark.  The meter results are stored on
Test-Name's property list.  The remaining three form items can be in any
number and any order that makes sense for the current operation.  Each
of them will be stuffed into the keyboard buffer and executed.
{\bf\#/Char} designates using the reader macro, \#/ to specify the
character Char.  (Refer to chapter 21 of the Lisp Reference Manual for
more information on reader macros.)  {\bfStrings} are just strings that
will be entered into the buffer, such as ''compile buffer''.  Finally,
the characters of {\bfSymbols}' print-names will be entered into the
buffer.  The following sample benchmark which measures the performance
of deleting local files will make the explanation more clear.

\begin{verbatim}

    (Edit-Delete-Local-File #\m-x "delete file" #\Return
	(String local-dir) "tree.*.*" #\Return #\Y)		

\end{verbatim}

The value of Test-Name is Edit-Delete-Local-File which is the name
associated with the benchmark.  The remainder of the expression
determines what will be stuffed into the buffer and executed.  In this
example three actions will occur:

\begin{itemize}
\item Meta-X delete file RETURN is stuffed and executed.
\item filename-to-be-deleted RETURN is stuffed and executed.
      File-name is the concatenation (String local-dir) and "tree.*.*".
\item Y is stuffed and executed.  This is in response to the delete files
      confirmation prompt.  
\end{itemize}

Everything in the list form except for Test-Name is included in the
meter timings.  

The special symbol {\bfWait} can be used between form objects as needed
to prevent losing characters stuffed into the buffer.  When the symbol
Wait is read, the stuff routine will wait until the buffer is again
available before stuffing the next form or object.  In the previous
example it may have been better to have put a Wait symbol before the
filename designation.  The special Test-Name symbol {\bfIgnore} can be
used to execute a form that is not a benchmark and for which you do not
want to save the meter results.  The form will be executed normally but
the results discarded.  Ignore facilitates clean-up between iterations
so the next loop will start in the same environment as did the previous
one. 

The following example shows the use of ignore and wait symbols.  This
benchmark which is named edit-clean is not really a benchmark; its
purpose is to clean up the ZMACS environment before executing the next
benchmark.  The ignore prevents the timing results recorded during
edit-clean from being saved.  The first wait symbol ensures that this
form will not execute until the previous benchmark has completed.  The
second wait symbol ensures that the return is not missed.

\begin{verbatim}
edit-clean
(select-editor #\system #\e)
(ignore " " wait #\c-x #\k wait #\return "no" #\return #\return #\abort)
\end{verbatim}

The script file execution is begun by the function Junior, which has
parameters (Pathname \&optional Clear-p Name (Repcount 1)).
{\bfPathname} is the pathname of the script file to be executed.
{\bfClear-p} when true will cause the performance software to forget any
known history of benchmarks ran for Name.  {\bfName} is the system name
to be associated with this benchmark run and result data file as
described in the chapter Running Benchmark Suites.  {\bfRepcount} is the
number of times the benchmark suite is to be executed.  Junior's main
task is to create the process Stuff which is the driver that executes
the benchmarks.

Process Stuff opens the script file for input, then reads each form and
processes it until end of file.  As each form is read, Stuff saves
Test-Name in the special variables, *all-benchmarks* and
*interactive-benchmarks* and creates a benchmark structure for the
current benchmark.  Next it calls Stuff-Input on each element in the cdr
of the script form.  Finally it calls Record-Benchmark to save the meter
results for the current benchmark. 
	
There is a debug mode for the Interactive benchmarks which can be useful
when testing new script files.  The third parameter to Junior,
{\bfName}, in normal mode is a symbol indicating the environment and the
result file.  However, supplying this parameter as a list causes
Stuff-Input to go into a debug mode.  In debug mode Stuff waits after
each character is stuffed, slowing the interaction so it's easier to see
what's going on.  

The script file may contain top-level symbols instead of the lists
describe above.  These symbols act as labels that can be used for
including or excluding portions of the script file during debug.  Name
list is a list of labels, optionally preceeded by a minus (-) sign.
Stuff skips over sections of script that are not preceeded by one of the
labels in the Name list, or, if the first thing in the name list is a
minus (-), it skips sections of the script that are in the Name list.
This feature can be used to speed up debug by making it easy to skip
over portions of the script that are already debugged.

The following example should clarify the previously made points.  It
contains the call to the function Junior and a subsection of the script
file, Debug.Script.  Notice that the parameter Name is a list of labels
as described above.  Finding the Name parameter specified as a list,
Stuff will switch to debug mode.  Notice also the matching label
''edit-misc'' that appears before the series of Edit benchmarks.  When
process Stuff reads in this top-level label, it will search the name
list parameter for a match.  Finding the label in the name list, it will
begin stuffing and executing the script forms in debug mode.  When Stuff
reads the next top-level label, Create, it will not find a match in Name
list and will not execute the Select series of benchmarks.  If the first
symbol in Name list had been a minus sign, then Stuff would have skipped
over the Edit series and executed the Create series in debug mode.

\begin{verbatim}
(junior "lm:bench;debug.script" T '(edit-misc edit-dired) 1)

edit-misc
(edit-list-buffers #\c-x #\c-b wait " ")
(edit-select-buffer #\c-x #\b wait "tree" #\escape wait #\return)  
(edit-scroll-forward #\c-v)
(edit-scroll-backward #\m-v)
(edit-select-previous-buffer #\c-m-l)

create
(select-converse-create #\system #\control-c)
(select-inspector-create #\system #\control-i)
(select-lisp-create #\system #\control-l)
\end{verbatim}

\end{document}
























































































