What’s new in JAGS 4.0.0 part 1/4: Reproducibility and testing

As promised, this is the first of a series of posts explaining some of the changes in JAGS 4.0.0 compared to previous versions. This post deals with two important changes that are may not be obvious

Reproducibility

Previous versions of JAGS had problems with reproducibility. Specifically, JAGS did not produce exactly the same output when given the same seeds for the random number generators in each chain. This problem is now fixed in JAGS 4.0.0. It required a major overhaul to fix, but the effort was worthwhile. It is important for scientific research to be reproducible and it is a requirement in commercial environments where analyses need to be validated.

The underlying problem was that the very first versions of JAGS stored nodes in memory in an order that was, in principle, arbitrary, with the consequence that the sampling order of the nodes was also arbitrary. This was not obvious at first because I tested JAGS on small problems and used the standalone command line interface. Under these conditions the internal ordering of nodes was the same from one run to another. But when people started using JAGS with larger problems and the R interface was developed, the reproducibility problem emerged. It first came to my attention 3 years ago.

All analyses in JAGS 4.0.0 should be reproducible, and if they are not then please let me know. Please note that I can only guarantee reprodicibility of analyses using exactly the same JAGS version, i.e. when the next release comes out it may produce different results from JAGS 4.0.0, although it should still be internally consistent. One isse to watch out for if you care about reproducibility is that some highly optimized math libraries (such as Intel MKL) sacrifice bitwise reproducibility for speed.This behaviour can be modified but is much slower, losing the advantage of a high-performance library.

Testing

JAGS is becoming quite popular. Version 3.4.0 was downloaded nearly 50,000 times from the Sourceforge web site. With this popularity comes the responsibility to ensure that it is working correctly.  So JAGS 4.0.0 introduces unit tests to ensure that the functions and distributions in JAGS are working as documented.

If you are using a binary version of JAGS this will not affect you (You can assume the person who built the binary package ran the tests for you). If you are installing JAGS from source then you can run the tests from the build directory with

make check

The underlying test framework id CppUnit, so you need to have this installed to run the tests. This choice may have been a mistake because it turns out that CppUnit is no longer actively developed.  So I may change to an alternative testing framework in the future, but my immediate priority is to improve coverage of the tests during the 4.x.y release series.

Advertisements

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s