Performance assessment plays a crucial role in engineering education. Yet most instructor assessment focuses on student outcomes to analyse achievements. Although there is extensive research analysing student productions, however, few studies have explored assessment from instructor perspectives, especially when reporting their assessment practice. This study examines instructors' assessment of student performance through the lens of course review reports (CRRs). The CRRs were collected from 5 core undergraduate courses submitted for annual review and were related to the mappings of the measurable outcomes to performance indicators, assessment methods, and level of engagement. Regardless of the variability in reporting the student design experience, instructors' assessment and potential gaps, as well as strong existing correlations between some indicators and associated assessment methods, the study showed that the CRR may be a powerful and complementary approach to investigate the complexity of multidisciplinary design and design assessment.

In recent decades, IGOs, NGOs, private firms and even states have begun to regularly package and distribute information on the relative performance of states. From the World Bank's Ease of Doing Business Index to the Financial Action Task Force blacklist, global performance indicators (GPIs) are increasingly deployed to influence governance globally. We argue that GPIs derive influence from their ability to frame issues, extend the authority of the creator, and — most importantly — to invoke recurrent comparison that stimulates governments' concerns for their own and their country's reputation. Their public and ongoing ratings and rankings of states are particularly adept at capturing attention not only at elite policy levels but also among other domestic and transnational actors. GPIs thus raise new questions for research on politics and governance globally. What are the social and political effects of this form of information on discourse, policies and behavior? What types of actors can effectively wield GPIs and on what types of issues? In this symposium introduction, we define GPIs, describe their rise, and theorize and discuss these questions in light of the findings of the symposium contributions.

Educators in bioethics have struggled to find valid and reliable assessments that transcend the “reproduction of knowledge” to target more important skill sets. This manuscript reports on the process of developing and grading a minimal-competence comprehensive examination in a bioethics master’s degree program. We describe educational theory and practice for the creation and deployment of scoring rubrics for high-stakes performance assessments that reduce scoring inconsistencies. The rubric development process can also benefit the program by building consensus among stakeholders regarding program goals and student outcomes.

We describe the Structure of the Observed Learning Outcome taxonomy as a mechanism for rubric design and provide an example of how we applied that taxonomy to define pass/fail cut scores. Details about domains of assessment and writing descriptors of performance are also presented. Despite the laborious work required to create a scoring rubric, we found the effort to be worthwhile for our program.

In this paper we present a probabilistic fast model for performance assessment of geothermal doublets for direct heat applications. It is a simple yet versatile and multipurpose tool. It can be well applied in better understanding the sensitivity of performance to key subsurface parameters and depth trends therein, and for assessing the probability of success for geothermal projects under technical and financial constraints.

The underlying algorithms deliver a sensible accuracy given the uncertainties associated with geothermal projects at exploration state. A public release of the software, available under the name of DoubletCalc, is easy to handle and requires a limited set of input parameters. Thanks to an open source code, DoubletCalc can be implemented in other software applications and extended as it has been implemented for the integration into the national geothermal information system in the Netherlands (ThermoGIS, 2011).

Apart from its application for site assessments, the tool can be integrated into automated workflows processing faster representations of key aquifer properties and capable to produce indicative maps for predicted doublet power, economic feasibility and prediction of cumulative amount of heat that can be recovered. These capabilities are specifically important for decision support for policymakers while assessing the effects of particular insurance schemes and funding mechanisms.

DoubletCalc cannot and is not intended to substitute geologic exploration approaches. As exploration measures, such as seismic surveys are cost intensive, DoubletCalc can be used to focus geothermal exploration on areas and sites where an enhanced probability of success can be expected.

In this paper we report on one component of a three-year study into the use of digital technologies for summative performance assessment in senior secondary courses in Western Australia. One of the courses was Italian Studies, which had an oral communication outcome externally assessed with an oral performance for which students travelled to a central location and undertook an interview with two assessors. Apart from the logistical difficulties for both students and the organising body, this method did not leave an enduring record of the process, and raised questions about the reliability of the assessment. Over the three years of this study, we tried several approaches to using digital technology to assess oral performance, including a portfolio of sub-tasks leading up to a video-recorded oral presentation, a computer-based exam, a video recorded interview, and an online exam that included oral audio-recordings. For each of the years online marking tools supported two methods of drawing inferences about student performance from the representations: the more traditional analytical method and the comparative pairs method. Rasch analysis of the results of the two methods showed that both were at an acceptable level of reliability. Overall, students and teachers reported that they liked using audiovisual recordings and online performance tasks for revision but not for summative assessment. The study also demonstrated that the scores from externally marked computer-based oral tasks carried out in class time correlated highly with the scores from traditional face-to-face recorded interviews. Therefore, online assessment of oral performance appears to be an equally effective way to facilitate assessment when compared with traditional methods and offers other affordances, such as convenience and access from a variety of locations, as well as providing an enduring record of student performance.

It is universally agreed that efforts to improve quality benefit from the analysis of outcomes. Yet, it is challenging to compare results across institutions because factors other than performance also impact outcomes. Two factors that complicate the analysis of outcomes after congenital cardiac surgery are case-mix and random statistical variation. Case-mix refers to differences in the mix of patients and their risk-factors at different institutions that may cause some centres to have more frequent complications and lower survival regardless of their true performance. Random statistical variation refers to fluctuations in outcomes that occur at random and follow the laws of probability. A variety of statistical methods exist to address these issues and make provider comparisons more fair. We explain a few common approaches including stratification, regression analysis, and confidence intervals. Concepts are illustrated using artificial data from two hypothetical hospitals, as well as real data from a multi-institution registry.

This paper examines problems of setting targets for health care performance in which the centre sets a uniform set of targets and levels of performance. The case study examined by the paper is from the system of performance assessment of ‘star ratings’ that was introduced from 2001 by the Department of Health to give each organisation in the NHS in England a single summary score from zero rating to three stars. Star ratings were directed at holding trusts' chief executives to account for the local delivery of national priorities through a process of ‘naming and shaming’, in which Chief Executives of zero-rated organisations were at risk of losing their jobs. This paper outlines the underlying model and its three underlying assumptions: the centre can determine a scoring system to prioritise what matters; failures of performance that are not reflected in the scoring system do not matter; and the advantages of a system of scoring on accountable targets outweigh the disadvantages of various types of gaming responses. This paper examines the application of the model of star ratings to a case study of Primary Care Trusts (PCTs), which are complex organisations with three diverse sets of responsibilities: delivering primary care, commissioning secondary care, and supervising public health. It outlines an alternative model and the issues this raises for governance of health care performance.