Validation of the Adult Decision-Making Competence in Slovak students

Abstract

A study using a high school and college sample (age 18–26) was conducted to validate the Slovak version of the Adult Decision-Making Competence. The results were similar to findings reported by Bruine de Bruin, Parker, and Fischhoff (2007) on the adult population in America. The internal consistency of component subscales and whole measure was confirmed as well as the factor structure. Gender differences in two of the six subscales were found. The results highlight the usefulness of A-DMC in assessing decision-making competence in Slovak language, but non-student samples are needed to enhance the generalisability of findings.

1 Introduction

Decisions in general are affected by three sets of factors—decision features, situational factors and individual differences (Einhorn, 1970; Hunt et al., 1989). The normative approach in decision making tries to identify the best principles of making decisions taking into consideration basic rules, mainly statistical and logical ones, and to assess decisions according to them. People need a suite of generally applicable decision-making skills such as extracting relevant information, applying general values in specific settings, and integrating these pieces with a coherent decision rule (Parker & Fischoff, 2005). A variety of general skills was identified. Stanovich and West (1998, 2000, 2008) showed correlations among different reasoning and decision-making skills.

The view of decision-making competence (henceforth: DMC) is very heterogeneous, with different DMC components identified: abilities to understand, appreciate, reason, express a choice (Grisso & Appelbaum, 1998); abilities to structure a decision problem, understand relevant information, integrate information and reason about it, appreciate the personal significance of information and the limits of one’s decision skills (Finucane & Lees, 2005); and belief assessment, value assessment, integration, and metacognition (Parker & Fischhoff, 2005). The ambiguity of DMC components is also manifest in the ways they are assessed. DMC has often been studied using self-report data from respondents or judgments from experts (Kim et al., 2001; Kitamura & Kitamura, 2000), but self-reports may not be valid indicators of behavior (Barker et al., 2002). Assessing performance and comparing with norms seems to work better. Appelt et al. (2011) state three general measures of DMC focusing on three age periods. Youth Decision-Making Competence (Y-DMC, Parker & Fischhoff, 2005) and the adapted adult version—Adult Decision-Making Competence (A-DMC, Bruine de Bruin et al., 2007) are quite similar in structure (six and seven components, respectively) whereas Older Adult Decision-Making Competence (OADMC, Finucane & Gullion, 2010) focuses on quite a narrow age range and assesses DMC in health, nutrition and finance domains. The other two measures in this category—Decision Outcome Inventory (Bruine de Bruin et al., 2007) and Problem Solving Inventory (Heppner & Petersen, 1982) are more specific and do not provide a complex picture of DMC. (The Decision Outcome Inventory is focused more on outcomes of decision-making than on DMC.) From widely used methods only A-DMC, Y-DMC and OADMC are performance-based measures of DMC.

1.1 Adult Decision-Making Competence

Bruine de Bruin et al. (2007, see also Parker & Fischoff, 2005 for a study of 18–19 years old subjects) developed a battery of decision tasks measuring decision-making competence divided into four abilities. Bruine de Bruin et al. (2007) report appropriate psychometric characteristics (Cronbach’s alphas, test-retest reliability, factor structure, correlations with fluid cognitive ability and real decision making outcomes) of the Adult Decision-Making Competence (henceforth: A-DMC). A-DMC was classified as decision-making measure in a classification of Appelt et al. (2011). A-DMC tries to capture four fundamental decision-making skills in six types of tasks. Decision-making skills include belief assessment (judging the likelihood of outcomes—one of the prominent areas in judgment), value assessment (how we can evaluate outcomes of a behaviour), integration (combining beliefs and values as a crucial step in matching person and environment) and metacognition (knowing the extent of one’s abilities as a skill to evaluate not only decision tasks, but also our potential to cope with them). Performance in these skills can be evaluated as accuracy (relative to external criterion) or as consistency (related judgments or choices). Bruine de Bruin et al., (2007) used seven components to identify decision-making skills, but one of them (Path Independence) was later eliminated because of low factor loadings and correlations with other subscales, and only six tasks are used (e.g., Del Missier et al., 2010). A description of decision-making skills and scoring follows.

Belief assessment involves judging the probabilities of events. Probability judgments are one of most analyzed topics in judgment and decision making (Kahneman & Tversky, 1972, Gigerenzer et al., 1991), often with emphasis on perception of risk (Slovic, 1987; Sjőberg, 2000). Belief assessment is operationalized in two tasks. The first of them, Consistency in Risk Perception, consists of 20 events, and subjects have to judge the probability of a given event (e.g., car accident) in a specified time period (one year, five years) on a linear scale from 0% (no chance) to 100% (certainty). Twenty pairs of events were identified and compared: a) probability in one and five years (10 pairs), b) probability of subset and superset events (6 pairs—e.g. to die in a terrorist attack and to die from any cause), and c) probability of complementary events (4 pairs—e.g. to get or not to get into a car accident). The resulting score is the percentage of correct item pairs. The second task tapping belief assessment is Recognizing Social Norms. Subjects answer 16 questions about whether they think it is sometimes OK to engage in different kinds of negative behavior (e.g., drink and drive, smoke cigarettes) and their answers serve to compute the actual proportion of people that would engage in this behavior. They also estimate how many out of 100 people their age behave like this. Spearman rank-order correlation is computed between the estimated and actual social norms.

Value assessment is also operationalized in two tasks. The first of them, Resistance to Framing, detects the vulnerability to be affected by the framing effect—the way a situation is described. Seven valence framing problems and seven attribute framing items are presented twice—as gains and as losses. The absolute differences between ratings for the loss and the gain versions of each item are subtracted from 5 in order to report higher values as better performance (smaller framing effect). The next task measuring insensitivity to irrelevant features is Resistance to Sunk Costs, containing 10 situations where prior investments were made (e.g., big dessert after a large meal when a person is full already). A choice between the sunk-cost option and normatively correct option is made on a six-point scale; performance is indicated as the average rating.

Combining beliefs and values is called integration. It is measured by one subscale in the present study—Applying Decision Rules. Subjects are asked for the best choice out of five DVD players for a hypothetical consumer with certain preferences regarding five characteristics (e.g. picture quality, brand reliability). Performance is indicated by the percentage of correct DVD players chosen.

Metacognition as the view of one’s competence is measured in the Over/underconfidence component involving 34 knowledge questions. Subjects indicate the correctness of each statement (true/false—e.g. alcohol causes dehydration) and their confidence in that answer. The resulting score is computed as one minus absolute difference between the mean confidence and percentage of correct knowledge answers.

1.2 Use of the A-DMC

In the first version of the DMC (as Youth Decision-Making Competence, Parker & Fischhoff, 2005) two of seven subscales (Resistance to Framing, Resistance to Sunk Costs) had low reliability in terms of Cronbach’s alpha. Correlations among the subscales were weak; a one-factor model was proposed as the best solution. The tasks in the adult version were partly changed—this change resulted in better psychometric characteristics of the A-DMC (Bruine de Bruin et al., 2007). A two factor model was proposed as the best solution with Resistance to Framing, Under/Overconfidence, Applying Decision Rules and Consistency in Risk Perception loaded onto Factor 1 and Recognizing Social Norms, Resistance to Sunk Costs and Path Independence loaded onto Factor 2. This factor structure does not correspond to any of the task characteristics (response mode, criterion, decision-making skill). The authors present correlations of A-DMC components with cognitive ability and experienced decision outcomes. The correlations between A-DMC components and decision-making styles were weak (from -0.28 to 0.29). In a subsequent analysis, Bruine de Bruin et al. (2010) reported a weak relationship of component scores with age controlling for demographic variables (gender, education, SES). While performance in Resistance to Framing and Applying Decision Rules decreased with age, Under/Overconfidence and Resistance to Sunk Costs improved and Consistency in Risk Perception and Recognizing Social Norms did not correlate with age. Carnevale, Inbar, and Lerner (2011) reported a positive correlation of two subscales (Resistance to Framing, Resistance to Sunk Costs) with need for cognition, but no relationship with the other two used components—Under/Overconfidence and Consistency in Risk Perception. In a subsequent comparison, leaders performed better in all components except for Under/Overconfidence. There was a tendency to maximize correlates negatively with the overall A-DMC score (Bruine de Bruin et al., 2007; Parker et al., 2007).

Besides the USA, the whole measure or some components of the A-DMC has been used in only two European countries—Italy (Del Missier et al., 2010; Del Missier et al., 2012) and Sweden (Marklund, 2008; Mäntylä et al., 2012), where it was also standardized. Del Missier et al., (2010) found a different contribution of executive functions (updating, shifting, and inhibition) on performance in Applying Decision Rules (mainly inhibition) and Consistency in Risk Perception (mainly shifting). These results were supplemented in a consequent study (Del Missier et al., 2012), where the monitoring/inhibition dimension of the executive functions was significantly related to the performance in Resistance to Framing and Applying Decision Rules and shifting component to Consistency in Risk Perception. Mäntylä et al. (2012) found no difference between adults with ADHD and the control group in Under/Overconfidence and the control group was significantly better in Applying Decision Rules.

The main aim of the present study is to examine the cross-cultural validity of A-DMC, namely characteristics of its Slovak version. In spite of good psychometric properties of the measure reported in its first use (Bruine de Bruin et al., 2007), it has been used in only the three previously mentioned countries (the USA, Italy, Sweden), and no European version provides complete psychometric characteristics. Here, I present descriptive statistics and Cronbach’s alpha, as well as exploratory and confirmatory factor analysis, correlations among component scores, and gender differences. Data of the original study by Bruine de Bruin et al. (2007) are provided for comparison, but no inferential comparisons are made, because of age and cultural differences.

2 Method

2.1 Subjects and procedure

The subjects were 508 high school and university students (318 females, 62.6%), 54.9% in the high-school sample and 68.8% in the college sample), from Eastern Slovakia (Central Europe). The average age of the sample was 20.71 years (SD = 2.38, range = 18 to 26)—a lower level of original study. As 18 old subjects are considered adult in Slovakia, no parental consent was needed. The complete English version of A-DMC was translated into Slovak by a professional translator and translated back by another one to ensure the fidelity of the tasks. The items from six components were in the original order—if a component consists of two parts, they are separated by items from another component. Data were collected in years 2011 and 2012 in a series of studies investigating a relation of the A-DMC with other cognitive and social characteristics. Each subject also provided demographic information. As well as the six component scores, the overall score was computed as the unweighted average of standardized z scores.

2.2 Results

Table 1 shows descriptive statistics for the six A-DMC component measures—higher scores in each component represent better performance. The observed range covers the majority of the potential range and covers most the original study. The biggest differences in mean score are in Applying Decision Rules (15% of the potential range), Consistency in Risk Perception (9% of the potential range) and Recognizing Social Norms (8% of the potential range). The statistical significance of the mean differences was not investigated, mainly because age differences were confounded with possible cultural differences.

Table 1: Descriptive statistics of nonstandardized A-DMC components. The first row for each component is from the present study; the second row is from the original A-DMC study by Bruine de Bruin et al. (2007).

Cronbach’s alpha was used as a measure of internal consistency. It is above .70 in four of six component measures and in the range .50–.60 in two component measures. The Slovak version yields better internal consistency in four of six component measures. Reliability in Recognizing Social Norms is assessed in two ways. Besides the joint measure, two independent Cronbach’s alphas were computed: for personal social norms (α =.69) and predicted peer social norms (α = .87). They were slightly lower in comparison with the original A-DMC study (α s = .75 and .93).

Table 2 presents bivariate correlations between the A-DMC component scores as well as correlations of component scores with the aggregate score—unweighted average of z scores of component scores. All of the correlations between component scores are positive and (with two exceptions) statistically significant (M = .26, Mdn = .24). The strongest correlations are found in Applying Decision Rules, Consistency in Risk Perception and Resistance to Framing. In spite of narrow age range, correlations with age were computed. Scores in five of six of A-DMC components as well as overall score increased with age.

Table 2: Pearson correlations among A-DMC components and aggregate z-score. The first row for each measure is from the present study; the second row is from the original A-DMC study by Bruine de Bruin et al. (2007).

^* p < .05

^** p < .01

^*** p < .001

RtF—Resistance to Framing, RSN—Recognizing Social Norms, UOC—Under/Overconfidence, ADR—Applying Decision Rules, CiRP—Consistency in Risk Perception, RtSC—Resistance to Sunk Costs. OS—Overall score (not available for original study).

Exploratory factor analyses on z scores was used to assess the inner structure of A-DMC because of expected cultural differences (Table 3). A one-factor model was obtained with the principal factors method explaining 40.3% of variance (compared with 30.1 % in Bruine de Bruin et al., 2007). Factor loadings are at least .537 except for Under/Overconfidence, which suggests the existence of an underlying construct of decision-making ability. The principal factors method with oblimin rotation allowing nonorthogonal factors was used to find a solution with more factors. Exploratory factor analysis assuming found only one factor fulfilling of eigenvalues ≥ 1 criterion; scree plot also identified only one factor. The two factors are correlated (r = .116, p = .009).

Table 3: Loadings for the one- and two-factor A-DMC models. The first row for each component is from the present study; the second row is from the original A-DMC study by Bruine de Bruin et al. (2007).

With the exception of Under/Overconfidence, all factors have loadings at least .563 on the first factor. This factor solution does not correspond to that reported by Bruine de Bruin et al. (2007) and also not to task characteristics like response mode (judgment or choice), criterion (consistency and accuracy) and general decision-making skill.

In order to confirm the factor structure found in the study by Bruine de Bruin et al. (2007), a confirmatory factor analysis was performed on six component scores (Factor 1: Resistance to Framing, Under/Overconfidence, Applying Decision Rules, Consistency in Risk Perception; Factor 2: Recognizing Social Norms, Resistance to Sunk Costs). As presented in Table 4, a number of indexes were used to determine the goodness of fit. The factor structure was similar to hypothesized (χ²(8)=12.64, p = 0.125). The comparative fix index (CFI), non-normed fit index (NNFI) and normed fit index (NFI) were above 0.95, which indicates a good fit of data. In addition, the root mean square error of approximation (RMSEA—lower than 0.08) confirms adequate fit to the data.

Table 4: Fit Indices for the two-factor structure of A-DMC.

Gender differences were found using a MANOVA (V = 0.06, F(6,501) = 5.18, p < 0.001). As shown in Table 5, separate univariate ANOVAs revealed that males have a higher score in Applying Decision Rules and Resistance to Sunk Costs. No statistically significant gender differences were found in the other four components or the overall score.

Table 5: Means (and standard deviations) of A-DMC components and overall score by gender.

3 Discussion

The main aim of the present study was to validate a measure for assessing decision-making competence—A-DMC. The results in general replicate characteristics of A-DMC in a different culture. The descriptive statistics are very close to data by Bruine de Bruin et al. (2007), and so are the correlations among component scores and measure’s factor structure. Small differences can be explained by the age structure of the sample (age 18–26 (M=20.7) versus 18–88 (M=47.7)), but the influence of a different culture is also very likely. The proportion of females was higher in original study (73.8%) than in the present research (62.6%).

Gender comparison in our results found minor significant differences in component scores. The apparent better performance in four of the six components in the Slovak version compared to the original study can be ascribed to the student population and relationship of performance scores to education.

The A-DMC provides a broad view on decision-making skills, including assessing belief, assessing values, their integration and metacognition. In spite of this variety in skills and tasks, correlations and factor analysis show that general decision-making competence can be in the background (a positive manifold). Each correlation except for two among component scores was positive and statistically significant. One of the main factors in the background can be general cognitive ability, which had a positive relationship with all of the used A-DMC components (Bruine de Bruin et al., 2007). Medium or weaker correlations indicate usefulness of each component tapping different aspects of judgment and decision making. Bruine de Bruin et al. (2007) name consistency in judgments and choice over time and resistance to biases arising from heuristics as possible skills. Correlations among A-DMC components further support the proposal that performance on conventional behavioral decision-making tasks reflects a positive manifold rather than random performance errors (Stanovich & West, 2000).

The A-DMC, using real-life situations, tries to be a valid predictor of real behavior. Although the paper-pencil form can raise questions about the validity of obtained scores in relation to real behavior, previous research indicates positive correlations of five of the six used components (except of Resistance to Framing) with the occurrence of negative decision outcomes (Bruine de Bruin et al., 2007). Using the whole measure or certain components seems appropriate particularly in the selection process of managers and other positions with decision making as a core responsibility. The variety of tasks allows capturing competencies specific for different occupations demanding assessing ourselves (Uner/Over-confidence) or groups (Recognizing Social Norms), release from task description (Resistance to Framing) and previous investments (Resistance to Sunk Costs) or following general rules of judgment (Consistency in Risk Perception) and choice (Applying Decision Rules).

Applications the A-DMC in the previously mentioned domains, and the generalisability of results from present and previous studies are partially limited by length of the A-DMC. Administering the whole measure lasts approximately one hour and raises concerns about attention, concentration and the ability to resist fatigue. These factors can affect components or their parts located closer to the end of the scale. However, this limitation is disconfirmed by studies using only some of the available tasks and reporting very similar results (Del Missier et al., 2010; Carnevale et al., 2011), so it seems reasonable to treat all component scores as reliable and not affected by task order. While this limitation seems not to be relevant, it could be useful to develop a shorter version using all six subscales. Validation of such a scale could lead to more extensive use in psychological research and in applied fields of psychology.

Another possible limitation of the present study, one that is frequent in academic research, is the use of students as research sample. Firstly, this practice reduces the age range. Previous research (Bruine de Bruin et al., 2010) indicates a mixed pattern of changes in decision-making competence related to age. When controlling demographic variables and fluid cognitive capacity, age was either in a positive or neutral relationship with A-DMC component scores. Furthermore, a student sample (mainly college students) is relatively consistent in education level, but variability of the A-DMC components is similar across high-school and college sample. Bruine de Bruin et al. (2007) report positive correlations between five of the six used tasks (all except for Resistance to Sunk Costs) and education. Further studies with the Slovak version should incorporate other age and education groups to verify obtained results. Relations with cognitive ability or real-life decision outcomes should also be investigated.

In spite of the possible limitations of the measure and of the present study, A-DMC seems to be appropriate to assess basic decision-making competence. As well as for research purposes, its use is also possible in work psychology (selection, promotion), clinical psychology (higher scores are correlated with fewer life negative outcomes) or in the prevention of excessive risk taking.