Including external results

Very often, you are running experiments in order to compare your work with existing results: is my algorithm running faster? Does my procedure provide better precision/recall? Etc.

LabPal provides a convenient way to include existing results and process them side by side with the new results that are generated by your experiments. The principle is simple: it suffices to create new experiments that do not run anything, and rather directly output pre-recorded results.

By hand

This can be done by hand. For example, suppose you found this table that advertises sorting times for a (fictional) new algorithm called the "Blabla Sort":

Array size	Time (ms)
1000	14
2000	24

You would like to plot these results along with the ones computed by your existing sort experiments. To this end, we can create a new experiment class:

public class BlablaSort extends Experiment {
  public BlablaSort(int size, int time) {
    setInput("Algorithm", "Blabla Sort");
    setInput("Size", size);
    write("Time", time);
    setStatus(Status.DONE);
  }

  public void execute() {
  }
}

Note how this experiment "hard-codes" its output parameter "Time" directly from a value passed to the constructor, instead of performing an actual sorting and measuring its duration. In consequence, there is nothing to do in method run, as the output results for this experiment are already generated. (As a side note, the constructor also already sets the experiment's state to DONE.)

From this point on, instances of BlablaSort can be added to SortingLab, included into tables and plots, like any other experiment.

Automatically from a file

A more systematic way of creating "pre-executed" experiments is to use the ExperimentBuilder. First, instead of hard-coding the data in the lab's setup method, let us move it into a text file called blabla.txt:

# Sorting times for BlaBla Sort
# Fetched from the following paper: ...

Size*   Time
---------------------
1000    14
2000    24

Let us now change the class BlablaSort so that it implements the interface CloneableExperiment. A cloneable experiment must implement a method called newExperiment, which is expected to return a new, empty instance of this class.

public class BlablaSort extends Experiment implements CloneableExperiment<BlablaSort> {
  public class BlablaSort() {
    setInput("Algorithm", "Blabla Sort");
    setStatus(Status.DONE);
  }
  public void execute() {
  }
  public BlablaSort newExperiment() {
    return new BlablaSort();
  }
}

We are now ready to use the ExperimentBuilder to automatically create instances of BlablaSort by filling their data from the contents of the text file. In our lab:

public void setup() {
  ExperimentBuilder<BlablaSort> builder = new ExperimentBuilder<BlablaSort>(new BlablaSort());
  Scanner scanner = new Scanner(FileHelper.internalFileToStream(this.getClass(), "blabla.txt"));
  try {
    Set<BlablaSort> experiments = builder.buildExperiments(scanner);
  } catch (ParseException e) {
  }
  ...
}

Let us examine what this code does:

• The first line creates a new ExperimentBuilder, which will create instances of experiments of class BlablaSort

• The second line opens a scanner to the text file we created

• The third line passes this scanner to the builder. In response, the builder returns a set of two BlablaSort experiments. These experiments can then be manipulated like all the others: added to the lab, to a table, etc. (An exception is thrown if the file is incorrectly formatted.)

Where do these two experiments come from? They have been instantiated from the contents of blabla.txt.

• Blank lines, lines that start with # and ---- are ignored

• The first non-ignored line is expected to contain a tab-separated list of parameter names

• All remaining non-ignored lines are interpreted as parameter values for one new experiment instance. For example, line "1000 14" will produce a new BlablaSort experiment with parameter "Size" set to 1000, and parameter "Time" set to 14.

Since BlablaSort "simulates" sorting by an algorithm, "Size" should be interpreted as an input parameter, while "Time" is rather an output parameter. This distinction can be made in the text file by putting a star at the end of input parameter names.

As we have seen, a possible use of the ExperimentBuilder is to include experiments made from existing results fetched from some external source (research paper, database). In such a case, the execute method of the experiment is empty. However, the ExperimentBuilder can be used to instantiate any experiment, including experiments that need to be run. For example, one can use a text file to define the set of input parameters to be given to experiments, instead of creating them manually in the lab's setup method.