Hostname: page-component-745bb68f8f-g4j75 Total loading time: 0 Render date: 2025-01-12T18:37:18.311Z Has data issue: false hasContentIssue false
  • English
  • Français

Who watches the watchmen? Evaluating evaluations of El Sistema

Published online by Cambridge University Press:  13 June 2018

Geoffrey Baker ,
Anna Bull  and
Mark Taylor
Geoffrey Baker
Affiliation:
Music Department, Royal Holloway, University of London, Egham, Surrey, TW20 [email protected]
Anna Bull
Affiliation:
School of Social, Historical and Literary Studies, University of Portsmouth, Burnaby Road, Portsmouth PO1 [email protected]
Mark Taylor
Affiliation:
Sheffield Methods Institute, 219 Portobello, Sheffield, S1 4DP, [email protected]
Get access Rights & Permissions [Opens in a new window]

Abstract

Within the growing field of publications on El Sistema and Sistema-inspired programmes around the world, a marked divide can be observed between the findings of critical academic studies and commissioned evaluations. Using evaluations of El Sistema in Venezuela and Aotearoa New Zealand as our principal case studies, we argue that this gulf can be explained at least partly by methodological problems in the way that some evaluations are carried out. We conclude that many Sistema evaluations display an alignment with advocacy rather than explorative research, and that the foundation for El Sistema's claims of social transformation is thus weak.

Type
Research Article
Information
British Journal of Music Education , Volume 35 , Issue 3 , November 2018 , pp. 255 - 269
Copyright
Copyright © Cambridge University Press 2018 

Access options

Get access to the full version of this content by using one of the access options below. (Log in options will check for institutional or personal access. Content may require purchase if you do not have access.)

References

ALEMAN, X. et al. (2016) The effects of musical training on child development: A randomized trial of El Sistema in Venezuela. Prevention Science.Google Scholar
BAKER, G. (2014) El Sistema: Orchestrating Venezuela's Youth. New York: Oxford University Press.Google Scholar
BAKER, G. (2016a) Editorial Introduction: El Sistema in critical perspective. Action, Criticism & Theory for Music Education, 15 (1), 10–32.Google Scholar
BAKER, G. (2016b) Citizens or subjects? El Sistema in critical perspective. In Elliott, D., Bowman, W., & Silverman, M. (eds.), Artistic Citizenship: Artistry, Social Responsibility, and Ethical Praxis (pp. 313–38). New York: Oxford University Press.Google Scholar
BAKER, G. (2016c) Antes de pasar página: Conectando los mundos paralelos de El Sistema y la investigación crítica. Revista Internacional de Educación Musical, 4, 51–60.Google Scholar
BAKER, G. (2017) Big noise in Raploch? Scottish Review, 21 June. http://www.scottishreview.net/GeoffBaker285a.html.Google Scholar
BAKER, G. & FREGA, A. L. (2018) ‘Producing musicians like sausages’: New perspectives on the history and historiography of Venezuela’s El Sistema. Music Education Research. DOI: 10.1080/14613808.2018.1433151Google Scholar
BELFIORE, E. (2002) Art as a means of alleviating social exclusion: Does it really work? A critique of instrumental cultural policies and social impact studies in the UK. International Journal of Cultural Policy, 8 (1), 91–106.Google Scholar
BELFIORE, E. (2009) On bullshit in cultural policy practice and research: Notes from the British case. International Journal of Cultural Policy, 15 (3), 343–359.Google Scholar
BELFIORE, E. (2016) Cultural policy research in the real world: Curating ‘impact’, facilitating ‘enlightenment’. Cultural Trends, 25 (3), 205–216.Google Scholar
BELFIORE, E. & BENNETT, O. (2010) Beyond the ‘toolkit approach’: Arts impact evaluation research and the realities of cultural policy-making. Journal for Cultural Research, 14 (2), 121–142.Google Scholar
BORCHERT, G. (2012) Sistema Scotland: A critical inquiry into the implementation of the El Sistema model in Raploch. MMus, University of Glasgow.Google Scholar
BORZACCHINI, C. (2010) Venezuela en el Cielo de los Escenarios. Caracas: Fundación Bancaribe.Google Scholar
BRUNS, S. B. & IOANNIDIS, J. P. A. (2016) p-Curve and p-hacking in observational research. PLoS ONE, 11 (2), e0149144.Google Scholar
BULL, A. (2016) El Sistema as a bourgeois social project: Class, gender, and Victorian values. Action, Criticism, and Theory for Music Education, 15 (1), 120–53.Google Scholar
CID, A. (2014) Giving a second chance: An after-school programme in a shanty town interacted with parent type: Lessons from a randomized trial. Educational Research and Evaluation, 20 (5), 348–365.Google Scholar
CUESTA, J. (2011) Music to my ears: The (many) socioeconomic benefits of music training programs. Applied Economics Letter, 18 (10), 915–18.Google Scholar
DE ANGELIS, C. et al. (2004) Clinical trial registration: A statement from the International Committee of Medical Journal Editors. The New England Journal of Medicine, 351, 1250–1251.Google Scholar
DELATTRE, S. & ROQUAIN, E. (2015) New procedures controlling the false discovery proportion via Romano–Wolf's heuristic. The Annals of Statistics, 43, 1141–1177.Google Scholar
DOBSON, N. (2016) Hatching plans: Pedagogy and discourse within an El Sistema-inspired program. Action, Criticism, and Theory for Music Education, 15 (1), 89–119.Google Scholar
ELPUS, K. (2015) Music teacher licensure candidates in the United States: A demographic profile and analysis of licensure examination scores. Journal of Research in Music Education, 63 (3), 314– 335.Google Scholar
FINK, R. (2016) Resurrection symphony: El Sistema as ideology in Venezuela and Los Angeles. Action, Criticism, and Theory for Music Education, 15 (1), 33–57.Google Scholar
GEN. (2011). Evaluation of Big Noise, Sistema Scotland. Edinburgh: Scottish Government Social Research.Google Scholar
GERRING, J. (2006) Case Study Research: Principles and Practices. Cambridge: Cambridge University Press.Google Scholar
GLASGOW CENTRE FOR POPULATION HEALTH. (2015). Evaluating Sistema Scotland – initial findings report. Glasgow: GCPH.Google Scholar
HEAD, M. L., HOLMAN, L., LANFEAR, R. & JENNIONS, M. D. (2015) The extent and consequences of p-hacking in science. PLoS Biol, 13 (3), e1002106.Google Scholar
HOLLINGER, D. (2006) Instrument of Social Reform: A Case Study of the Venezuelan System of Youth Orchestras. DMA diss., Arizona State University.Google Scholar
HOLOCHWOST, S., WOLF, D. P. & BOSE, J. H. (2017) Building Strengths, Buffering Risk: Evaluating the Effects of El Sistema-Inspired Music Programs in the United States. http://wolfbrown.com/images/books_reports/Building_Strengths_Buffering_Risk.pdf.Google Scholar
JOHANSON, K. & GLOW, H. (2015) A virtuous circle: The positive evaluation phenomenon in arts audience research. Participations, 12 (1), 254–270.Google Scholar
LEES, L. & MELHUISH, C. (2015) Arts-led regeneration in the UK: The rhetoric and the evidence on urban social inclusion. European Urban and Regional Studies, 22 (3), 242–260.Google Scholar
LOGAN, O. (2015a) Doing well in the eyes of capital: Cultural transformation from Venezuela to Scotland. In McNeish, J.–A., Borchgrevink, A. and Logan, O. (eds.), Contested Powers: The Politics of Energy and Development (pp. 216–253). London: Zed Books.Google Scholar
LOGAN, O. (2015b) Hand in Glove: El Sistema and Neoliberal Research. https://www.researchgate.net/publication/287202150.Google Scholar
LOGAN, O. (2016) Lifting the veil: A realist critique of Sistema's upwardly mobile path. Action, Criticism, and Theory for Music Education, 15 (1), 58–88.Google Scholar
McKEGG, K., CROCKET, A., GOODWIN, D., & SAUNI, P. (2015) Sistema Aotearoa: Outcomes Evaluation Report. Hamilton: The Knowledge Institute Limited (Kinnect Group).Google Scholar
McPHAIL, G., RATA, E., & SITEINE, A. (2018) The changing nature of music education. In McPhail, G., Thorpe, V. & S. Wise (eds.), Educational Change and the Secondary School Music Curriculum in Aotearoa New Zealand (pp. 74–91). London: Routledge.Google Scholar
MERLI, P. (2002) Evaluating the social impact of participation in arts activities: A critical review of François Matarasso's Use or Ornament? International Journal of Cultural Policy, 8 (1), 107–118.Google Scholar
NEW ZEALAND LAW COMMISSION. (2001) Māori Custom and Values in New Zealand Law, Study Paper 9, Wellington, N.Z: The Law Commission.Google Scholar
PEDROZA, L. (2015) Of orchestras, mythos, and the idealization of symphonic practice: The Orquesta Sinfónica de Venezuela in the (collateral) history of El Sistema. Latin American Music Review, 36 (1), 68–93.Google Scholar
PEREZ, E. & ROJAS, Y. (2013) ¿Por Qué Quiero Que Mi Hijo Sea Músico? Expectativas de las Madres, Cuyos Hijos Están en la OSIC. Diss, Universidad Católica Andrés Bello. http://biblioteca2.ucab.edu.ve/anexos/biblioteca/marc/texto/AAS7348.pdf.Google Scholar
RIMMER, M., STREET, J. & PHILLIPS, T. (2014) Understanding the Cultural Value of In Harmony-Sistema England. Arts & Humanities Research Council/University of East Anglia.Google Scholar
ROSABAL–COTO, G. (2016) Costa Rica's SINEM: A perspective from postcolonial institutional ethnography. Action, Criticism, and Theory for Music Education, 15 (1), 154–87.Google Scholar
SCRUGGS, T. M. (2015) ‘The Sistema,’ the Euroclassical Tradition, and Education as a Transformative Agent to Supercede Class Status. Paper presented at the annual meeting of the Society for Ethnomusicology, Austin, Texas, December 3–6.Google Scholar
SELWOOD, S. (2002) Measuring culture. Spiked Online. http://www.spiked-online.com/newsite/article/6851#.WOd4UGTyugQGoogle Scholar
SISTEMA NACIONAL DE ORQUESTAS JUVENILES E INFANTILES. (2011) Evaluación de Impactos. http://idbdocs.iadb.org/wsdocs/getdocument.aspx?docnum=36583351Google Scholar
STATISTICS NEW ZEALAND. (2016) Life Expectancy. http://www.stats.govt.nz/browse_for_stats/snapshots-of-nz/nz-social-indicators/Home/Health/life-expectancy.aspx (accessed 4.28.17)Google Scholar
STUFFLEBEAM, D. L. & CORYN, C. L. S. (2014) Evaluation Theory, Models, and Applications (2nd edition). San Francisco: Jossey–Bass.Google Scholar
THRUPP, M. (2014) At the eye of the storm: Researching schools and their communities enacting National Standards. New Zealand Journal of Educational Studies, 49 (1), 6–20.Google Scholar
THRUPP, M. & EASTER, A. (2012) Research, Analysis and Insight into National Standards (RAINS) Project: First Report Researching Schools' Enactments of New Zealand's National Standards Policy. http://www.nzei.org.nz/site/nzeite/files/reports/RAINS-Final-2012-03-01.pdf.Google Scholar
THRUPP, M. & WHITE, M. (2013) Research, Analysis and Insight into National Standards (RAINS) Project. Final report: National Standards and the Damage Done. Wellington, New Zealand: The New Zealand Educational Institute Te Riu Roa (NZEI). http://www.education2014.org.nz/.Google Scholar
TRINIK, R. (2014) Window with a view: reflections on Sistema Aotearoa. E-Journal of Studies in Music Edu-cation, 10 (1), 12–21. http://www.arts.canterbury.ac.nz/music/merc/downloads/ejournal_v10no1.pdf.Google Scholar
TUHIWAI SMITH, L. (1999) Decolonizing Methodologies: Research and Indigenous Peoples. London: Zed Books.Google Scholar
TUNSTALL, T. (2012) Changing Lives: Gustavo Dudamel, El Sistema, and the Transformative Power of Music. New York: W. W. Norton.Google Scholar
TUNSTALL, T. & BOOTH, E. (2016) Playing for Their Lives: The Global El Sistema Movement for Social Change Through Music. New York: W. W. Norton.Google Scholar
WANG, R. et al. (2007) Statistics in medicine — Reporting of subgroup analyses in clinical trials. The New England Journal of Medicine, 357, 2189–2194.Google Scholar
WOODSIDE, A. (2010) Case Study Research: Theory, Methods and Practice. Bingley: Emerald Group Publishing.Google Scholar
YIN, R. Y. (2013) Case Study Research: Design and Methods (5th edn). Los Angeles: SAGE Publications.Google Scholar