Easily run experiments on a computer
If you are doing research in Computer Science, it is very likely that every now and then, you need to run experiments on your computer. LabPal is a Java library that allows you to quickly setup an environment for running these experiments, collating their results and processing them in various ways: generating tables, data files, plots, etc.
As a matter of fact, running experiments and collecting data is a fundamental aspect of Computer Science research. A recent study by Oliveira et al. has shown that roughly 85% of publications in a sample of top-tier conferences provide empirical results generated from a set of experiments. These experiments generally involve the repeated execution of instructions for varying sets of input parameters, the collection of experimental data, and its processing in various forms (tables, plots, etc.). From this point of view, CS can be likened to any other experimental science, with the computer as the primary "environment" where experiments are taking place.
Reproducibility in Computer Science
The possibility for a third party to reproduce results published by someone else is a cornerstone of experimental science. Unfortunately, in this respect, Computer Science lags behind. In a recent article in the Communications of the ACM, Collberg and Proebsting reveal that published computer systems research is not always accompanied by the code that supports the research. In their study, out of a sample of 601 research papers that describe experiments, 407 had results that could not be reproduced, mostly due to code and data not being made publicly available. In the large majority of cases, this was caused by the fact that the corresponding source code could not be found for various reasons (code not publicly available, authors unwilling/unable to provide it or simply not responding).
To address this problem, several publication outlets have started putting incentives for authors of experimental research to make their data, and more importantly their code, available to others. For example, the ACM SIGSOFT International Symposium on Foundations of Software Engineering (FSE) encourages submissions that facilitate reproducibility and follow-up research by using available data sets, publicly providing data sets, or making tools publicly available; this is formalized through the Research Artifact Committee, where authors can submit their research data and code for external evaluation. Other conferences in various fields of Computer Science have started to adopt measures in the same spirit.
However, merely making data and code available only solves part of the problem. The process of running custom-made programs and collecting their data is not standardized; this often results in researchers cooking up what Crick et al. described as a bunch of "hack-together, use-once, throw-away scripts". This way of working presents several disadvantages.
First, this code is often extremely environment-specific, requiring precise versions of many pieces of software, or files to be present in hard-coded locations on the experimenter's machine; hence simply re-creating the proper environment for running the experiments becomes a challenge. As a matter of fact, in the Collberg and Proebsting study, 32 of of the 226 papers for which source code was obtained could not be compiled or run by the authors.
Second, the one-off nature of these scripts is such that no time is invested to implement advanced functionalities or care for user-friendliness. In particular, most scripts of this kind generate undocumented data files, whose content and meaning can be next to impossible to interpret by an external eye. The practice can be detrimental for the original researchers themselves; since most of this code is hardly reusable from one paper to the next, many menial tasks (parsing text files, generating graphs) are re-coded on every new project, resulting in a waste of time. The situation is probably aggravated by the fact that there is little academic credit to be obtained for developing tools in support of research, but which do not bring any new contribution in terms of direct results.
The need for an experimental environment
These observations highlight the need for software that could streamline the execution of experiments and make the collecting and processing of their data easier. Ideally, a researcher would have at her disposal a set of tools that would minimize the amount of boilerplate code required for conducting common experimental tasks. This code, being reusable across projects, would be worth investing time in features seldom seen in typical scripts, such as user-friendly means of controlling the execution of each experiment and visualizing the collected data. With added support for packaging the whole in a stand-alone bundle, this would result in an easy way for external researchers to repeat experiments independently and comment on their results.
To address the gap in the current supply of support tools for conducting repeatable CS experiments, we developed LabPal, a Java library for coordinating the execution of experiments on a computer and collecting their results.
Features
All your experimental setup (including your code, its input files and library dependencies) can be bundled into a
single, runnable JAR file, making it easy for anybody to download and re-run your experiments.
The runnable JAR acts as a web server; when launched, a user can see all the experiments and control their execution using a web browser.
For a few more lines of code, the input/output parameters of an experiment can come with a small textual description
that is displayed in the web interface, so that a user can understand the meaning of each data element. You can do the same with the lab itself, and each experiment it contains.
Each running experiment can update a visual progress bar; LabPal can even estimate and show the time remaining before they complete.
A set of partially executed experiments can be saved to disk, then reloaded and resumed at a later time (or even on a different machine).
I want to use LabPal!
Just download the latest release and include it in your classpath.
Or download our template project that includes a few extra features (like a boilerplate Readme and Ant build script).
If you want to know more about LabPal's features:
Read the short tutorial
Look at our lab examples to learn how to use various features
Look at our slightly longer instructions
Consult the online API documentation
About LabPal
LabPal was developed by Sylvain Hallé, Associate Processor at Université du Québec à Chicoutimi, Canada. Pr. Hallé is also the head of LIF, a research lab, where LabPal is extensively used for the processing of experimental results.
Last updated