Hostname: page-component-78c5997874-t5tsf Total loading time: 0 Render date: 2024-11-15T05:20:46.533Z Has data issue: false hasContentIssue false
  • English
  • Français

Rate the Raters: A Note on Wine Judge Consistency

Published online by Cambridge University Press:  05 January 2021

Jeffrey Bodington Open the ORCID record for Jeffrey Bodington [Opens in a new window]
Jeffrey Bodington*
Affiliation:
Bodington & Company, 50 California St, San Francisco, CA, 94111; e-mail [email protected].
Get access Rights & Permissions [Opens in a new window]

Abstract

Much literature shows that the ratings assigned by wine judges are uncertain, some authors have proposed that judges be tested, and a few wine competitions do test judges. However, no literature or competition has yet proposed a test or rating for judges based on realistic competition conditions. This article uses coefficients of multiple correlation to rate each of 54 judges who assigned ratings to 2,811 wines entered in a commercial competition. Results show that there is a strong and positive correlation between the ratings assigned by most judges to most wines. However, those correlations also show that the ratings assigned by approximately 10% of judges are indistinguishable from random assignments. Using correlations to rate the raters, a program is underway to monitor those judges and variations in competition protocol that may affect their ratings. (JEL Classifications: A10, C00, C10, C12, D12)

Keywords

judgeratingsstatisticswine
Type
Articles
Information
Journal of Wine Economics , Volume 15 , Issue 4 , November 2020 , pp. 363 - 369
Copyright
Copyright © American Association of Wine Economists, 2020

Access options

Get access to the full version of this content by using one of the access options below. (Log in options will check for institutional or personal access. Content may require purchase if you do not have access.)

Footnotes

The author thanks an anonymous reviewer for perceptive and constructive comments.

References

Abdi, H. (2007). Multiple correlation coefficient. In Salkind, N. J. (ed.), Encyclopedia of Measurement and Statistics, 648–651. Thousand Oaks, CA: Sage Publications.Google Scholar
Ashton, R. H. (2012). Reliability and consensus of experienced wine judges: Expertise within and between? Journal of Wine Economics, 7(1), 70–87.CrossRefGoogle Scholar
Ashton, R. H. (2013). Is there consensus among wine quality ratings of prominent critics? An empirical analysis of red Bordeaux, 2004–2010. Journal of Wine Economics, 8(2), 225–234.CrossRefGoogle Scholar
Beverage Trade Network (2019). World's Top Wine Competitions. Beverage Trade Network, https://beveragetradenetwork.com/en/worlds-top-wine-competition-716.htm (accessed October 31, 2019).Google Scholar
Bodington, J. C. (2012). 804 Tastes: Evidence on randomness, preferences and value from blind tastings. Journal of Wine Economics, 7(2), 181–191.CrossRefGoogle Scholar
Bodington, J. C. (2017). The distribution of ratings assigned to blind replicates. Journal of Wine Economics, 12(4), 363–369.CrossRefGoogle Scholar
Brooks, L. M. (2018). Best practices for creating sensory trials and evaluations. Wine Business Monthly, May 2018.Google Scholar
Frick, B. (2020). The legacy of gurus: The impact of Armin Diel and Joel Payne on winery ratings in Germany. Journal of Wine Economics, 15(4), 370–377.CrossRefGoogle Scholar
Hodgson, R. T. (2008a). An examination of judge reliability at a major U.S. wine competition. Journal of Wine Economics, 3(2), 105–113.CrossRefGoogle Scholar
Hodgson, R. T. (2008b). On rating wines with unequal judges. Journal of Wine Economics, 3(2), 226–227.CrossRefGoogle Scholar
Hodgson, R. T., and Cao, J. (2014). Criteria for accrediting expert wine judges. Journal of Wine Economics, 9(1), 62–74.CrossRefGoogle Scholar
Jensen, A. R. (2006). Clocking the Mind: Mental Chronometry and Individual Differences. Amsterdam, The Netherlands: Elsevier.Google Scholar
Luxen, M. F. (2018). Consensus between ratings of red Bordeaux wines by prominent critics and correlation with prices 2004–2010 and 2011–2016: Ashton revisited and expanded. Journal of Wine Economics, 13(1), 83–91.CrossRefGoogle Scholar
OIV (2015). Review document on sensory analysis of wine. International Organisation of Wine and Vine, April 2015. Available at http://www.oiv.int/public/medias/3307/review-on-sensory-analysis-of-wine.pdf.Google Scholar