This page gives a quick how-to for writing new CEAL benchmarks.

Please refer to the CEAL Manual for instructions on

Writing a New CEAL Program

Ordinarily, a CEAL program is embedded into a larger application. During development and testing, however, it is useful to work with the CEAL program in isolation. The CEAL Test Harness is a library used to test for correctness, stability and performance of CEAL programs. The test harness is generic---it is reused across all such programs.

To use the harness, each CEAL program provides an implementation to the interface described below. Conceptually, the interface provides an implementation of the following three aspects of the program:

Each aspect is implemented by implementing several functions. We describe each aspect in turn.

Part 1: The Computation (Core & Verifier)

void cealtesthook_run_core();
void cealtesthook_run_verf();

To test for input/output correctness, the test harness runs both a (self-adjusting) core version of the computation, as well as a (non-self-adjusting) verifier version on the same input. For computations whose input fully determines the output (e.g., where non-determinism does not affect the output, given a fixed input), this comparison helps to verify that the self-adjusting version is consistent with non-self-adjusting code.

Avoiding code duplication. To avoid writing two versions of every program, CEAL provides a way to synthesize both the core and verifier versions from a single piece of code. For example, if the program's computation is a function compute, the following implementations suffice for the interface above:

   1 void compute() { ... }
   3 void cealtesthook_run_verf() {
   4    compute();
   5 }
   7 void cealtesthook_run_core() {
   8   core(compute)();
   9 }

/!\ Note the use of the "core" keyword: conceptually, it yields a self-adjusting version of compute. For the verifier version, no special keyword is used: the verifier version is ordinary meta-level code.

Part 2: Input

void cealtesthook_input_generate(long size);
void cealtesthook_input_print(FILE* file);
void cealtesthook_input_iter_begin();
void cealtesthook_input_iter_next();
int  cealtesthook_input_iter_isdone();
void cealtesthook_input_iter_change();
void cealtesthook_input_iter_revert();

At a first approximation, the test harness calls these functions in a "change-propagate-revert-propagate" loop like the following:

   1 cealtesthook_input_generate(input-size);
   2 cealtesthook_input_iter_begin();
   3 while( ! cealtesthook_input_iter_isdone() ) {
   4   cealtesthook_input_iter_change();
   5   ceal_propagate();
   6   cealtesthook_input_iter_revert();
   7   ceal_propagate();
   8   cealtesthook_input_iter_next();
   9 } 

Part 3: Output

In addition to printing, this aspect to the interface provides a way to test the program for input/output correctness: it compares the core output with a (non-self-adjusting) verifier output.

typedef enum cealtesthook_output_version_e {
} cealtesthook_output_version_t ;

void cealtesthook_output_print(FILE* file, cealtesthook_output_version_t v);
int  cealtesthook_output_check(); /* returns non-zero if equal, zero otherwise */

Building Your New Program

Say you just wrote a program named myprog, and say that myprog.c implements the test harness interface described above. It's best if myprog is a unique name that is unused among the existing applications. Assuming that you're in the root of the CEAL development tree,

  1. Create directory ./src/apps/myprog
  2. Move myprog.c to ./src/apps/myprog/myprog.c.
  3. cd ./build
  4. run ./ myprog

If the compilation succeeds, you've built one (or several) versions of myprog.

  1. cd ../bin/apps/myprog/
  2. ls

Each subdirectory here corresponds to a distinct binary for myprog.

Testing Your New Program

The test harness runs a sequence of phases after generating the program input. By default, it runs all of the following:

  1. fromscratch (i)
    • In this phase, the core program is run on the input.
  2. allinsrem (p)
    • In this phase, the change-propagate-revert-propagate loop is run (see above). The phase is so-named for historical reasons: for many benchmarks, changes consist of insertions into the input, and reversions consist of deletions of the inserted elements.

  3. verifier (v)
    • In this phase, the verifier program is run on the input.
  4. verifier-check (V)
    • In this phase, the outputs of the core and verifier programs are compared.

Though this order is fixed (you cannot swap the order of any of the phases listed above), you skip some and instead only run a subset (e.g., just the core, or just the core and verifier, without propagation or checking) using the switch -phases phase-list (e.g., -phases i, or -phases iv, respectively).

Testing Every Change

By default, the harness only checks the final core output against the verifier output. This is only a very minimal sanity check. To check the core program's change-propagation behavior more exhaustively, use the -verf-all-updates switch, which tells the test harness to test the core output against a verifier run each time change propagation completes.

CEAL_programming_howto (last edited 2011-12-06 10:52:15 by hammer)