A new tool for assessing the cultural adaptation of cognitive tests: demonstrating the utility of the Manchester Translation Evaluation Checklist (MTEC) through the Mini-Mental State Examination Urdu

Abstract

Background

Low- and middle-income countries contribute to the majority of dementia and mild cognitive impairment cases worldwide, yet cognitive tests for diagnosis are designed for Western cultures. Language and cultural discrepancies mean that translated tests are not always reliable or valid. We propose a model for culturally adapting cognitive tests, one step of which is to assess the quality of any translation and cultural adaptation undertaken. We developed the Manchester Translation Evaluation Checklist (MTEC) to act as a tool for quality assessment and demonstrated its use by assessing a popular cognitive test that had been adapted.

Aims

Assess quality of the translation and cultural adaptation of the Urdu Mini-Mental State Examination developed for a Pakistani population.

Method

Two raters completed the MTEC for the Mini-Mental State Examination (MMSE) Urdu and compared feedback. All authors were fluent in English and Urdu and familiar with Pakistani culture.

Results

Raters had 78.5% agreement across the MTEC. The MMSE Urdu was appropriately translated and retained grammar and verb tense, but three questions had spelling errors. Across 20 MMSE questions, 5 required further cultural adaptation because the questions were not understandable in daily use, comfortable to answer, relevant to the language and culture, and relevant to original concepts.

Conclusions

The MTEC highlighted errors in the MMSE Urdu and demonstrated how this tool can be used to improve it. Future studies could employ the MTEC to improve existing translated measures of health assessment, particularly cognitive tests, and act as a quality check when developing new adaptations of tests and before psychometric validation.

There is an increased international demand for quick and accurate cognitive tests to facilitate the neuropsychiatric assessment process as instances of cognitive decline grow.¹ Mild cognitive impairment (MCI) is present in 42% of the global population over 60 years of age² and there are approximately 55 million people living with dementia globally, expected to rise to 139 million by 2050.³

The majority of cases are seen in low- and middle-income countries (LMICs), with dementia in these regions accounting for two-thirds of total global cases.^3,4 These high rates of dementia and MCI found primarily in LMICs may be attributed to growing numbers of elderly people in these countries.⁵ Although the number of older people is estimated to increase over the next 15 years, by 71% in Latin America and the Caribbean, 66% in Asia and 64% in Africa, there is only a 41% expected increase in North America and 23% in Europe.⁵

Furthermore, LMICs such as India, China and countries in Latin America show higher rates of risk factors for dementia and MCI.⁴ These include a higher prevalence of depression, diabetes, hypertension and obesity,^4,6–8 as well as living circumstances such as reduced education in early life, mid-life hearing loss, physical inactivity, smoking and social isolation.⁴ Research in high-income countries has determined that risk factors account for 35% of dementia cases but it is predicted that this number is actually higher when accounting for LMICs.⁵

Many countries in North America and Europe have also seen both increased economic and forced migration after the 1950s and 1970s, allowing them to form significant ethnic minority populations from LMIC backgrounds.^9,10 Whereas in the past ethnic minorities were more likely to return to their country of origin as they grew older, owing to the growing disparity in human development between LMICs and high-income countries, many now choose to permanently settle in their old age in the country they have migrated to.¹¹ Therefore, we now also see increasingly growing ageing ethnic minority populations in Western countries.¹²

These older people from ethnic minorities remain at risk for developing dementia and MCI owing to their retained susceptibility for risk factors found in LMICs that they share a heritage with.^12,13 Furthermore, ethnic minorities in Western countries are more likely to come from a low socioeconomic background, which has been associated with developing dementia and MCI.^14,15

Language and cultural shortcomings of current cognitive tests

Cognitive tests, which are incorporated in both the screening and diagnosis of dementia and MCI, allow for early detection and accuracy in neuropsychiatric assessment.¹⁶ However, the majority of cognitive tests available at present are designed for English speakers in the context of Western cultures such as those in Europe and North America.^1,17,18 They were standardised on male, White out-patients, not accounting for the populations of non-Western countries, including LMICs.^17,18

Owing to the content of cognitive tests, both language and culture affects an individual's perception of test questions and responses to them. When these English-language cognitive tests are administered to non-English speakers and to those from Western cultural backgrounds, such as Europeans and North Americans, we see reduced sensitivity and specificity and higher rates of false-positive and false-negative scores for the non-native English speakers.^19–22 Thus, there is a gap in the tests available to assess the cognitive health of culturally diverse populations, as well as populations in countries in the global East.

One proposed solution is to develop and pilot new cognitive tests but this is a time-consuming endeavour and not always feasible.²³ An alternative is to adjust cut-off scores of existing cognitive tests for different language-speaking groups and cultural populations but this is criticised for reducing sensitivity, specificity and likelihood ratios.^24,25

A third solution in practice has been to translate existing cognitive tests into target languages, with the use of the back-translation method.²⁶ This involves translating a test directly into a target language, translating it back into the original language and comparing the old and new versions to see if there are any discrepancies.²⁶ However, despite the rate at which back-translation is used, it still results in cognitive tests that produce false-positive and false-negative scores.²⁷ This is because the process of changing a cognitive test from one language to another and then back-translating can result in words being mistranslated, questions developing different meanings and loss of the concepts being assessed.^26–28

Furthermore, translation does not account for cross-cultural influence on the ability to answer questions beyond fluency in a target language.^1,27,28 This is commonly seen in cognitive tests as they utilise questions requiring familiarity with the Western alphabet, names, idioms or history to assess cognitive domains such as language and memory.^21,27,28 This cultural bias in test questions compromises the reliability of these cognitive tests:^29,30 the same person may produce different scores owing to misunderstandings propagated by mismatched language and culture.²⁷ The validity of these cognitive tests is also questionable: we are uncertain whether the results are due to the individual's state of health or due to errors in translation or differences in culture.²⁷ Therefore, existing cognitive tests require not just translation but adaptation for different cultures.^29–31

A model for the translation and cultural adaptation of cognitive tests

Flaherty et al²⁸ proposed that the first step in translating and culturally adapting any health measure is the selection process: literature is searched to see which tests may be suitable for a target population.^28,32 A list of potential tests is drawn up and one test is hand-selected based on whether it has been used in cultural groups similar to the target population, whether it has been translated for several languages before and whether it has been psychometrically validated for diverse groups.^28,32 There are also considerations beyond language and culture, such as the frequency of a test's use across published literature, the format it is delivered in and the cost of use due to copyright. Once a test has been selected according to this guidance it is determined whether it is available for the target population's language and cultural context. Owing to the need for cross-cultural health measures, if the selected test is not available in both the target language and cultural context then it requires translation and cultural adaptation. There may be instances where an adapted version of a test matches the target language but not the cultural context, or vice versa. In that instance further translating or culturally adapting that version of the test is appropriate as an attempt at adaptation has already been started.

As regards cognitive testing, our team demonstrated that, for every cognitive test, incorporation of literature on that test and feedback from adaptors of that test allows the development of a set of guidelines on culturally adapting each of that test's questions.³³ These methods were illustrated through forming cultural adaptation guidelines for the Addenbrooke's Cognitive Examination Version III³³ and we later replicated this for the Montreal Cognitive Assessment.³⁴

Following this methodology, a cognitive test can be translated and culturally adapted for any language and cultural context using its guidelines.^33,34 Such newly adapted tests can then be culturally validated, a process in which they are administered to healthy controls within the target population and a cognitive interviewing approach is undertaken.³⁵ This determines whether the questions in the newly adapted version are both understandable and acceptable in those without cognitive impairment and ascertains that there is no bias in the language and cultural context that may cause poor performance.³⁵ Finally, the new version of the cognitive test can undergo a psychometric validation in people with dementia or MCI within the target population.

We propose a translation and cultural adaptation model consisting of the above steps, which also incorporates not just selecting and culturally adapting an appropriate test, such as a cognitive test, but also assessing the quality of the adaptation (Fig. 1 outlines the model in the context of cognitive testing).

Fig. 1 Proposed model for the translation and cultural adaptation of health measures in the context of cognitive testing.

When selecting a cognitive test for a target population, finding one that has made accommodations for cultural context alone is not enough; quality assurance should be undertaken to determine whether the translation and cultural adaptation is of good quality. Similarly, if a suitable cognitive test is not available and one has to be translated and culturally adapted, a final quality assurance check can identify whether the translation or cultural adaptation needs to be changed or improved further. We therefore developed the Manchester Translation Evaluation Checklist (MTEC),³² a new tool for assessing the quality of the translation and cultural adaptation of questions in a health measure, to address this.

Aims

In this paper we formally introduce the MTEC and demonstrate its utility as a quality assurance tool for the translation and cultural adaptation of health measures, particularly cognitive tests, by assessing an existing translation of a widely implemented cognitive test, the Mini-Mental State Examination (MMSE). The MMSE is a cognitive test that was developed to screen for both dementia and MCI.³⁶ It consists of 13 overall questions, some with further subquestions, giving a total of 19 questions. It primarily assesses the cognitive domain of language, but also assesses orientation, registration, attention, memory and visuospatial abilities.³⁷

The MMSE is one of the most commonly used cognitive tests globally. However, when translated and psychometrically validated in several non-English speaking countries, it demonstrated varied sensitivity and specificity owing to cultural biases^38–40 and differing education levels.⁴¹ This is attributed to the heavy reliance on language-based tasks, which often require cultural adaptation beyond literal translation.³³ Based on our proposed model, use of a quality assurance tool prior to psychometric validation may have highlighted where further translation and cultural adaptation was needed.

As it has universal application in the screening of dementia and MCI and many existing translations,⁴² the MMSE was an appropriate cognitive test for this demonstration. Given our own language and cultural background (we are all fluent in English and Urdu and are familiar with Pakistani culture), we utilised the MTEC to assess the quality of the translation and cultural adaptation of an Urdu version of the MMSE that had been translated for a Pakistani population.⁴²

Method

Setting and ethics approval

This research took place at the Centre for Primary Care and Health Services Research at the University of Manchester, UK. This research did not require review or ethics approval.

Materials

Manchester Translation Evaluation Checklist

The MTEC was developed by one of the authors (W.W.) at the University of Manchester to assess the quality of the translation and cultural adaptation of diagnostic health measures, particularly cognitive tests.³² Within the context of the MTEC, translation refers to whether the questions of a test were changed in terms of language, taking into account spelling, grammar, verb tense and retention of the original meaning of the question. Cultural adaptation refers to modifying or changing the question in terms of understanding, appropriateness and relevance to the cultural setting of the person being assessed, without changing the underlying concepts the question is assessing.

The MTEC was designed to account for Flaherty et al's five dimensions of cross-cultural equivalence, which reduce cultural bias in research instruments.²⁸ These dimensions are technical, semantic, content, conceptual, and criterion equivalence, and should be retained by instruments, including cognitive tests, when they are translated and culturally adapted for different language and cultural backgrounds.²⁸ When referring to cognitive tests, the dimensions would be defined as follows.³⁴

1. (a) Technical equivalence: the test is administered in a manner that is culturally appropriate for the person being assessed.

2. (b) Semantic equivalence: after translation, the individual test questions retain the same meaning as their original versions.

3. (c) Content equivalence: after accounting for culture, the individual test questions are relevant to the culture of the person being assessed.

4. (d) Conceptual equivalence: after accounting for culture, the individual test questions still assess the same cognitive domains they were meant to at the original level of difficulty.

5. (e) Criterion equivalence: after accounting for culture, the overall test is still assessing for dementia or MCI.

The MTEC assesses whether a test, including cognitive tests, meets these equivalences, through a seven-query checklist (the MTEC queries are listed in the Appendix below and the full MTEC questionnaire with guidance on administration is given in Supplementary Material 1, available at https://doi.org/10.1192/bjo.2022.620). First, an original test question is input, translated into the target language and then back-translated. Following this, the seven queries are addressed to gather feedback on the quality of the test question's translation and cultural adaptation. Queries 1, 2 and 3, which are answered ‘yes’ or ‘no’, relate to the translation and address technical and semantic equivalence. Queries 4, 5, 6 and 7 are answered on a three-point Likert scale, with possible answers being ‘not acceptable’, ‘just acceptable’ and ‘acceptable’. They relate to the cultural adaptation and address content, criterion and conceptual equivalence. The amalgamation of this feedback shows any errors in translation or cultural adaptation that exist in the test.

MMSE Urdu

The MMSE Urdu had been translated using the back-translation method by researchers fluent in both English and Urdu for a Pakistani population.⁴² It was administered to 400 participants with varied literacy: 100 with dementia and 300 healthy controls.⁴² That study, which was the first validation study of the MMSE Urdu in Pakistan, found the MMSE Urdu to be a reliable and valid cognitive test in the screening of dementia in a Pakistani Urdu-speaking population. We contacted the corresponding author of the MMSE Urdu to acquire their version of the test, the original English version they translated and culturally adapted, and the publication on their MMSE Urdu's psychometric validation.⁴² Table 1 gives a comparison between the back-translated questions in the MMSE Urdu and the original English versions they based them on (see Supplementary Material 2 for the MMSE Urdu).

Table 1 The Mini-Mental State Examination (MMSE) and back-translated MMSE Urdu questions⁴²

Procedure

Two authors (N.M. and M.W.W.), a psychology postgraduate researcher and a medical student respectively, read through the MMSE Urdu. They then filled out an MTEC for each question, answering the queries using the guidance provided. They also qualitatively reflected on why they chose their answer for each query. The authors used a talk-aloud approach to compare this MTEC feedback and reasoning.⁴³ When differences arose they were reviewed by a third author (W.W.), who is an experienced psychiatrist, until consensus was reached.

Results

Initially, across the 20 MMSE Urdu questions with 7 MTEC queries each, N.M. and M.W.W. agreed on MTEC scoring 110 out of 140 times, resulting in a 78.5% agreement. After discussion with W.W. a consensus was achieved on all queries for all questions. The final MTEC results are shown in Table 2.

Table 2 The Manchester Translation Evaluation Checklist (MTEC) output for each question of the Mini-Mental State Examination (MMSE) Urdu

Quality of translation

All MMSE Urdu questions had been appropriately translated (Query 1) and their tense was retained (Query 2). MTEC Query 3 showed that in questions 3, 4 and 6, the Urdu word ‘sahee’, meaning ‘correct’, had been incorrectly spelled.

Quality of cultural adaptation

Of the 20 MMSE Urdu questions, the MTEC found no issues in the cultural adaptation of 15 questions. Questions 1, 3–9, 11–12, 14–15 and 18–20 were found to be acceptable across all MTEC queries in the following ways:

1. (a) the wording of the questions was understandable and applicable for daily use in the Urdu-speaking Pakistani population (Query 4)

2. (b) Urdu-speaking Pakistanis being assessed would be comfortable responding to these questions (Query 5)

3. (c) the questions related to concepts relevant to the Urdu language and Pakistani culture (Query 6)

4. (d) the back-translation of the questions related to the same concepts as the original version of the questions (Query 7).

For the five MMSE Urdu questions that scored ‘just acceptable’ or ‘not acceptable’ on MTEC queries, the following reflective reasoning was demonstrated by the authors (N.M. and M.W.W.).

Question 2: What is the season?

This question received ‘not acceptable’ on Query 7 as the back-translation did not relate to the same concept as the original. The Urdu word for ‘season’ – ‘mausum’ – can also be directly translated as ‘weather’, depending on how the question is phrased. The translation in the MMSE Urdu, when back-translated, became ‘What weather is this?’. This Urdu translation would need rephrasing to ensure that the question conveys ‘season’, as opposed to ‘weather’.

Question 10: Where are we (city)?

This question received ‘not acceptable’ on Query 7 as it is not in the original version of the test, despite fulfilling other requirements of the MTEC.

Question 13: Spell ‘WORLD’ backwards (alternative to Serial 7s)

This question received ‘just acceptable’ on Queries 4, 5 and 6 as it asks for the English word ‘world’ to be spelled backwards. Since the majority of the Pakistani population are not fluent in English and there is a significantly low literacy rate, people may not find it easy to understand or comfortable to answer, and the concept does not relate to the Urdu language or Pakistani culture. Typically, this should result in a ‘not acceptable’ rating, however, in the MMSE Urdu another option is given if individuals are unable to spell ‘world’ backwards for reasons other than cognitive impairment. They could state the order of the days of the week in reverse and this addresses the issues mentioned above. Therefore, a combination of both these tasks results in a ‘just acceptable’ rating. The question received ‘not acceptable’ on Query 7 as the portion of it that asks to state the order of the days of the week in reverse is not in the original, despite fulfilling other requirements of the MTEC.

Question 16: Repeat the following – ‘no ifs, ands or buts’

This question received ‘not acceptable’ on Queries 4, 6 and 7 as the phrase ‘No ifs, ands or buts’ is not a saying in Urdu and it was presented as the literal translation of ‘ifs and or but no is’ in Urdu, which has no meaning. It would not be understandable and it is not relevant in Urdu or to Pakistani culture. As the purpose of this question is to see whether the individual will repeat a well-known phrase, the back-translation of this question shows that this concept is not retained. Therefore, ratings of ‘not acceptable’ were given. However, to address these issues in the MMSE Urdu an alternative was given that allows the administrator to state any phrase which could then be repeated by the participant. This received a rating of ‘just acceptable’ for Query 5, as repeating any phrase could be comfortable for an individual as long as it was in their target language. But as it is an addition made to the MMSE Urdu and not justified within the publication, it cannot be related to any original concept.

Question 17: Follow a three-stage command: ‘take a paper in your hand, fold it in half and put it on the floor’

This question received ‘just acceptable’ on Query 7 as the back-translation of the Urdu version asks the individual to use their right hand specifically, despite this not being in the original version of the test.

Discussion

There is currently an international need for cognitive tests designed to assess for neuropsychiatric conditions such as dementia and MCI in individuals in LMICs or in ethnic minority populations in high-income countries.^17,18 Such cognitive tests must account for the language, cultural background and specific understanding of individuals being assessed, to avoid bias in performance.^19–22

We proposed a model for how existing cognitive tests could be translated and culturally adapted for a target population and a crucial step in that model is assessing the quality of the cultural adaptation being produced (Fig. 1). This quality assurance should act as a check to ensure that a test not only accounts for the language and culture of the person being assessed, but also maintains the integrity of the concepts being assessed. The MTEC does this by highlighting any errors in existing translations or cultural adaptations of a cognitive test, and allows for further improvements. We demonstrated this by using the MTEC to assess the quality of the translation and cultural adaptation of the MMSE Urdu.

As regards translation, the MMSE Urdu contained spelling errors but retained verb tense and had been appropriately translated overall. From the perspective of cultural adaptation, 5 out of 20 questions demonstrated several problems. These questions pertained to the cognitive domains of orientation and language, which are heavily reliant on cultural influence.^33,34 As the MMSE Urdu was developed through a back-translation method⁴² the development process would not have accounted for cultural bias and performance beyond fluency in Urdu.^26–28 This may explain why raters marked these questions as ‘just acceptable’ or ‘not acceptable’ on the MTEC.

Question 2, which assesses orientation, does not account for the word for ‘season’ also meaning ‘weather’ in Urdu. By avoiding a literal verbatim translation and engaging in nuanced cultural adaptation, rephrasing the question to ‘Which of the four seasons is it?’ would ensure that ‘season’ could not be confused for ‘weather’ in the Urdu question, while still retaining the original concept of the question.

Similarly, Question 13 for language (Question 12 in English), which asks Urdu speakers to spell the English word ‘WORLD’ backwards, demonstrates both a language and cultural mismatch, with Query 3 ‘just acceptable’ and Query 1 ‘not acceptable’. It attempts to mitigate this by offering a culturally appropriate alternative – stating the days of the week in reverse order. However, the task itself is already an alternative for the Serial 7s task, which individuals being assessed may not have the literacy requirement to complete.⁴² Instead of offering two alternatives to the Serial 7s task, a culturally appropriate MMSE Urdu might remove the ‘WORLD’ task and retain stating the days of week backwards as the only alternative.

Question 16 for language (Question 15 in English) had the most problems in terms of cultural adaptation, with Query 3 ‘not acceptable’ and Query 1 ‘just acceptable’. It is another example of a literal verbatim translation, where back-translation from one language to another has removed meaning from the phrase completely. As idioms and phrases are not necessarily universal it would have been more appropriate to have selected one within the cultural context of Pakistan, as opposed to random selection. To rectify this there is an option allowing the administrator to choose one themselves. However, the length of the phrase when spoken, the frequency of its use in the cultural setting, and the number of words and syllables within it all have psychometric significance in the validity of cognitive tests when assessing language ability. Therefore, idioms and phrases should not be selected at random by each assessor.

In Question 17 for language (Question 16 in English) the individual is asked to take a paper in their right hand as opposed to any hand. This may be accounting for cultural context because in Pakistan there is a preference for using the right hand, which is often encouraged and associated with cleanliness and purity. But no justification for this change has been provided in the publication detailing the development of the MMSE Urdu. Additionally, this does not mitigate the issues that may occur for those who are left handed and may instinctively use their dominant hand. By prioritising the right hand, the question's conceptual equivalence may be compromised.

Overall, we found that despite being psychometrically validated and currently in use, the MMSE Urdu still possesses errors in translation and potential for cultural bias for a wider Pakistani population. Therefore, this cognitive test would have benefited from a quality check with the MTEC beforehand to detect and amend these problems.

The MTEC allows for reflection on where a health measure, such as a cognitive test, currently has gaps in terms of its language and culture when it applies to a specific group, even if that test is currently in circulation. We have purposely ensured that the MTEC does not provide a quantitative score for judging whether the translation or cultural adaptation is deemed to be understandable and appropriate; there is no cut-off score at which a test is or is not suitable. This is because even a single error in translation or cultural adaptation can result in an overall compromised test. We can see from our demonstration with the MMSE Urdu that even one of the errors could potentially affect an individual's performance. Instead, the MTEC presents feedback and qualitative reflection that allow raters, whether they be reviewers such as in this instance or adaptors in the future, to consider their responses and utilise them to improve the quality. This ensures more standardisation across translated and culturally adapted versions of the same test.

Limitations and future research

We acknowledge that raters must be familiar with the target language and culture of the desired population to be able to use the MTEC. However, this is likely to be expected of any research team when translating a test and, in fact, we would advocate this to ensure a more robust cultural adaptation. We have shown that the MTEC is suitable and easy to use for those who may not have clinical experience or extensive knowledge of psychometrics, and this expands its potential beyond research team members for use by non-experts in target populations. In studies where cultural adaptation of a test is undertaken as part of the methods, the MTEC could be administered by non-experts as a form of patient and public involvement, and in the role of co-producers.

We acknowledge that in this particular illustration we are limited by only having two raters provide feedback on the MMSE Urdu, despite high agreement. However, this still gave enough feedback to highlight key problems and allowed for a demonstration of the MTEC. In future we recommend a more diverse range of raters, which would result in more extensive and nuanced feedback.

Although future studies may wish to identify errors in existing cultural adaptations of cognitive tests as we have done, the MTEC can also be used by research teams in the development of new translations and cultural versions of tests. We propose repeatedly using the MTEC and making revisions in accordance with its feedback at various stages of scale development; this can be done by both researchers and co-producers. This would facilitate the making of tests for populations in LMICs and for ethnic minority groups in high-income countries.

As detailed in our model, the MTEC can also be used as a precursor to psychometric validation studies of cognitive tests in diverse populations to eliminate any remaining cultural bias. As the design of the MTEC has been theoretically influenced by Flaherty et al's principles of translating and culturally adapting health measures,²⁸ future studies should also use the MTEC to assess the quality of other culturally adapted health measures beyond cognitive tests.