ILAMB

ILAMB v2.6 Release

18 May 2021 - Nathan Collier

It has been a while since our last release, but ILAMB continues to evolve. Many of the changes are ‘under the hood’ or bugfixes that are not readily seen. In the following, we present a few key changes and draw attention in particular to those that will change scores. We also have worked to make ILAMB ready to integrate with tools being developed as part of the Coordinated Model Evaluation Capabilities CMEC.

CMEC

  • Added CMEC-compliant JSON output to the standard outputs
  • Added an alternative landing page for ILAMB results which uses the LMT Unified Dashboard
  • Added support files for using cmec-driver as an alternative run environment

Quality of Life

  • Top page overhaul moving to a single result panel with a colorblind friendly palette
  • Shifted score colormaps to be more qualitative and colorblind friendly
  • ILAMB now has continuous integration testing using Azure Pipelines on each commit or pull request
  • ModelResults can be passed a list of paths to search for results, objects are cached as pickle files
  • Plotting limits are now based on the middle 98% across all models to help reduce the effect of a single model with extreme values washing out all the map plots
  • The configure file used to generate a run is now copied into the output directory as ilamb.cfg
  • ILAMB logfiles will now provide an estimate for peak memory usage in each confrontation which can be used in debugging and when running on large clusters with limited memory

Scoring

  • For scoring coupled models, we find that scoring the RMSE of the annual cycle is more reasonable. While the default is still set to score the full time series, this may be changed at runtime with --rmse_score_basis {series|cycle}
  • We have found that when comparing a set of models which contain a multimodel mean, the mean model’s interannual variability is typically lower which serendipitously better matches that of our reference data products. This makes the multimodel mean look even better relative to individual models but not for good reasons. We have disabled the interannual variability in our scoring.
  • We have updated a number of reference datasets to their most current version as well as many new datasets and comparions, run ilamb-fetch to update
  • Support for using observational uncertainty in scoring, currently disabled