Cummings, Jonathan W. , Hague, Merran J. , Patterson, David A. and Peterman, Randall M.2011. The impact of different performance measures on model selection for Fraser River Sockeye Salmon. North American Journal of Fisheries Management 31:323 — 334.
Abstract
Uncertainties prevalent in fisheries systems result in deviations between management targets and observed outcomes. As an example of attempting to deal with such uncertainty, fishery managers of sockeye salmon Oncorhynchus nerka from the Fraser River, British Columbia, use environmentally based management adjustment (MA) models to forecast indices of in-river loss of adults as they migrate upstream to spawn. Losses forecasted by MA models are directly incorporated into estimates of total allowable catch, resulting in harvest reductions that aim to increase the probability of achieving spawning escapement targets. However, the relative forecasting success of different MA models has not been rigorously assessed. Therefore, we used a suite of forecasting and hindcasting metrics to rank the performance of numerous MA models. We found that the rank of each model varied across sockeye salmon stock aggregates (i.e., run timing groups) and depended on the performance measures chosen for evaluation. Although model selection in fisheries research is often determined solely by model-fitting criteria, such as R2 and Akaike's information criterion (corrected for small-sample bias), in our case the models with the largest mean R2 value, the smallest mean corrected Akaike's information criterion, or both often ranked poorly for measures of model forecast performance (i.e., mean raw error, mean absolute error, and root mean square error). Although no single model performed best across all run timing groups, failure to apply an MA produced the worst outcome (for 3 of the 4 run timing groups) or second-worst outcome (for the fourth group). We provide a framework for model selection based on the relative importance of different model selection criteria and their associated performance measures. We urge scientists and managers to work closely together to develop appropriate metrics for assessing model performance and for objectively selecting forecast models that will best meet management objectives.