Marco A. Janssen

Will you be able to run your computational models in the future? Even with well-documented code, this can be difficult due to changes in the software frameworks and operating systems that your code was built on. In this paper we discuss the use of containers to preserve code and their software dependencies to reproduce simulation results in the future. Containers are standalone lightweight packages of the original model software and their dependencies that can be run independent of the platform. As such they are suitable for reuse and sharing results. However, the use of containers is rare in the field of modeling social-environmental systems. We provide an introduction to the basic principles of containerization, argue why it would be beneficial if this tool became common practice in the field, describe a conceptual walkthrough to the process of containerizing a model, and reflect on near future directions of containerization workflows.

To evaluate the concern over the reproducibility of computational science, we reviewed 2367 journal articles on agent-based models published between 1990 and 2014 and documented the public availability of source code. The percentage of publications that make the model code available is about 10%. The percentages are similar for publications that are reportedly dependent on public funding. There are big differences among journals in the public availability of model code and software used. This suggests that the varying social norms and practical convenience around sharing code may explain some of the differences among different sectors of the scientific community.

We present a repository for disseminating the computational models associated with publications in the social and life sciences. The number of research projects using computational models has been steadily increasing but the resulting publications often lack model code and documentation which hinders replication, verification of results and accumulation of knowledge. We have developed an open repository, the CoMSES Net Computational Model Library, to address this problem. Submissions to the library can be original models accompanying publications or replications of previous studies. Researchers can request that their models undergo a certification process that verifies that the model code successfully compiles and runs and that it follows documentation best practices. Models that pass the certification process are assigned persistent URLs and identifiers. We present the basic components of our repository, discuss our initial experiences with the library, and elaborate on future steps in the development of this cyberinfrastructure.

Keywords: Model archive; Open source; Computational modeling; Documentation; Agent-based modeling

This paper reports the results of the inaugural modeling competition sponsored by the Network for Computational SocioEcological Sciences (CoMSES Network). Competition participants were provided with a dataset collected from human-subjects experiments and were asked to develop an agent-based model that replicated behavioral patterns reflected in the data with the goal of using the model to predict behavioral changes in a slightly modified experimental treatment. The data were collected in a resource foraging experiment in which human subjects moved avatars on a computer screen to harvest tokens in a common pool resource. In the original experiments, on which the competition participants based their models, the subjects possessed full information about the state of the resource and the actions of the other group members sharing the resource. The competition challenged participants to predict what would happen if the experimental subjects had limited vision. Using only the data from the original experiment, participants had to design a model that would predict the behavioral changes that would be observed in the new experiment treatment. We compared the models on their assumptions about speed, direction, and harvesting decisions agents make. All the submitted models underestimated the amount of resources harvested. The best performing model was the simplest model submitted and had the best fit with the original dataset provided.

Keywords: Pattern-Oriented Modeling, Competition, Calibration, Empirical Data, Behavioral Experiments

Agent-based modelling has become an increasingly important tool for scholars studying social and social-ecological systems, but there are no community standards on describing, implementing, testing and teaching these tools. This paper reports on the establishment of the Open Agent-Based Modelling Consortium, www.openabm.org, a community effort to foster the agent-based modelling development, communication, and dissemination for research, practice and education.

Keywords: Replication, Documentation Protocol, Software Development, Standardization, Test Beds, Education, Primitives