Brier score decompositions, including those attributed to Murphy and to Yates, provide popular metrics for estimating forecast performance attributes like calibration and discrimination. However, the decompositions are generally limited to situations where forecasters make successive forecast judgments against the same class of substantive event (e.g., rain vs. no rain). They do not readily translate to common situations where: forecasts are weighted unequally; forecasts can be made against a range of heterogeneous topics and events over varying time horizons; forecasts can be updated over time until an event occurs or an event deadline is reached; or outcome alternatives can vary in number and nature (e.g., ordered vs. unordered outcomes) across forecast questions. In this paper, we propose extensions of the Murphy and Yates decompositions to address these features. The extensions involve new analytic expressions for the decompositions of weighted Brier scores, along with proposed resampling methods. We use data from a recent forecasting tournament to illustrate the methods.

The accuracy of human forecasters is often reduced because of incompleteinformation and cognitive biases that affect the judges. One approach to improvethe accuracy of the forecasts is to recalibrate them by means of non-lineartransformations that are sensitive to the direction and the magnitude of thebiases. Previous work on recalibration has focused on binary forecasts. Wepropose an extension of this approach by developing an algorithm that uses asingle free parameter to recalibrate complete subjective probabilitydistributions. We illustrate the approach with data from the quarterly Survey ofProfessional Forecasters (SPF) conducted by the European Central Bank (ECB),document the potential benefits of this approach, and show how it can be used inpractical applications.