2016 International Land Model
Benchmarking (ILAMB) Workshop

The International Land Model Benchmarking (ILAMB) Project:
Comprehensive evaluation of the carbon cycle, hydrology, and terrestrial ecosystem processes in Earth system models

Forrest M. Hoffman, William J. Riley, James T. Randerson, Gretchen Keppel-Aleks, and David M. Lawrence

Original Prospectus: December 3, 2015
Updated: April 19, 2016

Workshop Title:

2016 International Land Model Benchmarking (ILAMB) Workshop

Workshop Location:

DoubleTree by Hilton Hotel Washington DC

1515 Rhode Island Avenue NW

Washington, DC 20005-5595, USA

+1-202-232-7000

Workshop Dates:

May 16–18, 2016

Workshop Rationale:

As Earth system models (ESMs) become increasingly complex, there is a growing need for comprehensive and multi-faceted evaluation of model predictions. To advance understanding of biogeochemical processes and their interactions with hydrology and climate under conditions of increasing atmospheric carbon dioxide (CO₂), new methods are needed that use observations to constrain model predictions, inform model development, and identify needed measurements and field experiments. Better representations of biogeochemistry–climate feedbacks and ecosystem processes in ESMs are essential for reducing uncertainties associated with projections of climate change during the remainder of the 21st century. In an effort sponsored by the U.S. Department of Energy (DOE), a diverse team of National Laboratory and university researchers is developing new diagnostic approaches for evaluating ESM hydrological and biogeochemical process representations. This research effort supports the International Land Model Benchmarking (ILAMB) project (http://www.ilamb.org/) by creating an open source benchmarking system that leverages the growing collection of laboratory, field, and remote sensing data. This benchmarking system performs comparisons of model results with best-available observational data products, focusing on atmospheric CO₂, surface fluxes, hydrology, soil carbon and nutrient biogeochemistry, ecosystem processes and states, and vegetation dynamics. Next generation benchmarking priorities will focus on design of new perturbation experiments (e.g., atmospheric CO₂ enrichment, water exclusion, nutrient addition, soil/plant warming) and resulting model evaluation metrics, new metrics from extreme events (e.g., drought, floods), and process-specific experiments (e.g., litterbags, ¹⁴C tracers). This benchmarking system is expected to become an integral part of model verification for future rapid model development cycles. Moreover, it will contribute model analysis and evaluation capabilities to phase 6 of the Coupled Model Intercomparison Project (CMIP) and future model and model–data intercomparison experiments.

The complexity of today’s process-rich ESMs poses a verification challenge to developers, implementing new parameterizations or tuning process representations. Model developers and software engineers require a systematic means for evaluating changes in model results to assure that their developments improve the fidelity of the target process representations while not adversely affecting results in other parts of the model. In addition to objectively assessing the performance of ESMs and identifying model weaknesses, the ILAMB benchmarking system can also provide a framework for verifying and tuning model developments, supporting a more rapid development cycle and providing continuous and documented evaluation of model skill. DOE’s Accelerated Climate Modeling for Energy (ACME) project has adopted ILAMB for this purpose and is implementing new metrics for more advanced features currently under development and testing.

CMIP provides essential information about future climate scenarios and ESM behavior that is used by the Intergovernmental Panel on Climate Change (IPCC) for periodic assessment and more broadly by policy makers and resource managers for the design of effective climate mitigation and adaptation strategies. Planning for CMIP6 is well underway, and the sophistication of model representations of biosphere and carbon cycle processes will far exceed levels achieved in past efforts. The complexity and volume of archived simulation output will be unparalleled, creating new opportunities for increasing our knowledge of biosphere–climate interactions, yet creating important challenges with respect to effectively harvesting this information to reduce uncertainties and improve our understanding of fundamental Earth system processes. Coordination among several MIPs is needed, as well as enhanced information flow among the modeling, evaluation, and observational communities.

We propose to continue and expand DOE’s role in coordination of CMIP activities and to improve upon DOE’s capabilities to assess the fidelity of model development in the ACME project by supporting comprehensive terrestrial model evaluation and benchmarking through the ILAMB project. Specifically, we propose to convene a three-day workshop to be held in downtown Washington, DC, USA, during May 16–18, 2016. Following more than five years after the first ILAMB Workshop in the U.S.—which was co-sponsored by DOE, NASA, and the International Geosphere–Biosphere Programme’s (IGBP’s) Analysis, Integration and Modeling of the Earth System (AIMES) project in January 2010—this second U.S. workshop is designed to accomplish the following objectives:

1. To highlight new techniques for model evaluation that can reduce uncertainties with respect to biosphere processes and biogeochemical feedbacks with the climate system;
2. To enable coordination among the Coupled Climate–Carbon Cycle Model Intercomparison Project (C4MIP); the Land Surface, Snow, and Soil Moisture Model Intercomparison Project (LS3MIP); and the Land‐Use Model Intercomparison Project (LUMIP) activities, particularly with respect to synergies that may exist for model evaluation and analysis;
3. To increase awareness of new data streams that will be available for model verification and benchmarking from remote sensing, in situ measurements, and synthesis activities;
4. To increase the use and sharing of information and community tools for model evaluation and benchmarking, including the ILAMB software package;
5. To design new metrics and evaluation approaches for integration into the next generation ILAMB system; and
6. To create new metrics that integrate across carbon, surface energy, hydrology, and land use disciplines.

Workshop Outcomes:

To meet these objectives for DOE’s climate research mission and to provide needed international climate science leadership, the workshop will invite participation from approximately 40 leading Earth system modelers, remote sensing experts, ecosystem ecologists, and data producers from around the world, in addition to team members of the Biogeochemistry–Climate Feedbacks Scientific Focus Area (SFA) and Program Managers from DOE and other federal agencies. A list of candidate participants from 11 countries is provided below. The outcome of the workshop will include a formal written report summarizing invited presentations, breakout group findings, and discussions regarding model evaluation strategies, gaps, and synergies. We also plan to have demonstrations on how to install and run version 1.0 of the ILAMB software system. In particular, the workshop report is expected address the topics following the draft outline shown here.

1. Model evaluation and benchmarking concepts and principles
2. Benchmarking tools
1. PALS / PLUMBER
2. ESMValTool
3. NASA LIS Evaluation
4. ILAMB
5. Existing model evaluation capabilities in use at modeling centers
6. Synergies between different benchmarking activities
3. Existing and new metrics for carbon, water, energy, and ecosystem processes
1. Ecosystem processes and states
2. Hydrology
3. Atmospheric CO₂
4. Soil carbon and nutrient biogeochemistry
5. Surface fluxes (energy and carbon)
6. Vegetation dynamics
4. Model Intercomparison Project (MIP) benchmarking needs and evaluation priorities
1. CMIP6 historical and DECK
2. C⁴MIP
3. LS3MIP
4. LUMIP
5. TRENDY
6. MsTMIP
7. PLUME-MIP
5. Next generation benchmarking challenges and priorities
Process-specific experiments
1. Metrics from extreme events
2. Design of new perturbation experiments
3. High latitude processes
4. Tropical processes
5. Global remote sensing
6. Fluxnet and other surface hydrology and ecosystem networks
6. Model benchmarking gaps and synergies
1. Integration with uncertainty quantification frameworks
2. Computational requirements for post-processing and workflow
3. Frameworks, Open Model Benchmarking Architecture (OpenMBA)
4. Integration with archival and distribution cyberinfrastructure
5. Evaluating new process representations and rapid model development
7. Conclusions and next steps (5 and 10 y long term goals)
8. Appendix A. ILAMB Tutorial Materials and Data

The workshop will also serve as a venue for communicating recent scientific results in applying the “emergent constraints” approach for limiting the range of model predictions and reducing uncertainty, as well as new methodologies for generating and applying metrics in the assessment of model fidelity. Opportunities for information exchange will be through invited plenary presentations held over the course of the workshop, focused breakout group discussions, informal breakaway group meetings on the evening of the second day, and a poster session to be conducted on the first evening of the workshop in conjunction with a hosted dinner, consisting of food items suitable for eating while standing. Moreover, the workshop will provide the opportunity for key international participants to discuss the organization and proposal of special issues of scientific journals addressing new model analyses, evaluation and benchmarking, and verification and validation for new process representations and parameter optimization.

We expect the ILAMB benchmarking software development, supported primarily by DOE, to be of keen interest to workshop participants. Access to these software tools and related data sets will be provided at the workshop, and time will be set aside for a tutorial or training session focused on installing, using, and extending these tools for model development and verification, model validation exercises, and model–data intercomparison studies. This training session will be led by the software package developers from the BGC Feedbacks SFA to assure that participants receive hands-on assistance and to enable direct feedback from users on bugs, analytical issues, and desired features.

A sample workshop agenda is contained below.