Measurement of Disaster Literacy in Turkish Society: Disaster Literacy Scale (DLS) Design and Development Process

Abstract

Objective:

This study aimed to design and develop a self-report Disaster Literacy Scale (DLS) tool that could evaluate the knowledge and skills of an individual specific to Turkish society.

Method:

Item development, expert opinions, language control, pilot study and field testing processes were monitored in the measurement tool based on a conceptual model and recognition.

Results:

23 items were taken out since their common variance values (>0,508, >0.500, >0.500, >0.400, respectively) and factor load relationship (>0.46, >0.50, >0.50, >0.50 and >0.55, respectively) in the mitigation, preparedness, response and recovery phases of the Exploratory Factor Analysis were insufficient. The Cronbach Alpha value of the remaining 61 items in the Disaster Literary Scale is 0.954 and between 0.83-0.88 in lower dimensions. DLS scoring was standardized between 0-50 points.

Conclusions:

The objectives, scopes, limitations and steps of the design and development process of the Disaster Literacy Scale were given in detail and made understandable for other societies. The Disaster Literacy Scale was developed as a self-report scale that could evaluate the knowledge and skills of Turkish society in disasters. The Disaster Literacy Scale is, therefore, expected to be accepted in more countries to improve the understanding of disaster literacy in different societies.

The importance of health literacy was demonstrated in the World Health Organization and American Medical Association report in 1990 with the determination of the health consequences of health literacy. ¹ It is seen that a generally accepted definition of health literacy has emerged with increasing health literacy studies over time: Health literacy is linked to literacy and entails people’s knowledge, motivation and competences to access, understand, appraise, and apply health information in order to make judgments and take decisions in everyday life concerning healthcare, disease prevention and health promotion to maintain or improve quality of life during the life course. ^2,3 Many health problems are involved in the scope and boundaries of the definition. For example, the effects of individuals’ limited/insufficient health literacy on health emerge as inadequate health protection and development practices, ⁴ self-care problems, ⁵ medication errors, hospitalization rate, and increased health expenses. ⁶ In a similar way, inadequate disaster literacy can have similar negative consequences by causing problems in people’s disaster preparedness. Health literacy is of great importance in determining health risks and consequences. However, the role of disaster literacy in affecting health and safety is not yet known. ^7–9 However, it can be said that, when an individual is exposed to health risks and caught unprepared for disasters, that person can be negatively affected in a physical, spiritual, and social way. The concepts of health and disaster literacy are only a part of the holistic view required to define a healthy individual. Therefore, concepts can be seen as interrelated structures that support each other in the process of designing a healthy person.

Disaster literacy is a term used to describe the capacity of individuals in modern society to meet complex demands on threats and hazards. As a result of disaster education activities, ¹⁰ disaster literacy includes cognitive and social skills that determine the ability and motivation of accessing, understanding, and using information to protect and promote an individual’s well-being. ¹¹ The concept has become increasingly important in disaster sciences in recent years ¹² because it analyzes the relationship between factors, which cause disasters, and their potential consequences. ¹³ Although the demand for tools to measure both empirical studies and disaster literacy is increasing, there is a lack of supply for a tool that measures the concept ¹³ in terms of disaster literacy.

This study aimed to measure the knowledge and skills of Turkish society in disasters through the evaluation of the Disaster Literacy Scale (DLS). In this respect, the development process in the field of disaster literacy can be achieved with qualified studies in the field of health literacy. In this context, in the development process of the scale, the European Health Literacy Project, ² which Çalışkan and Üner ¹³ previously referenced, and the health literacy definition, conceptual model, matrix, and formula, which were developed by Sorensen et al., ³ were used. The DLS captures the basic phases of disaster literacy as in the outlines of the definition and conceptual model proposed by Caliskan and Üner. ¹³ In this context, the study explains the design and development process of the DLS. The study also provides a detailed output of the structured and systematic approach followed in item development, expert consulting, language control, and field testing applications for the measurement of disaster literacy within the scope of the specified definition and conceptual framework model in Turkish society.

Methods

Conceptual Validity

The designing process of the DLS is based on the concept and definition of disaster literacy ¹³ derived from a systematic compilation of different definitions and models related to disaster literacy by Caliskan and Üner. ¹³ The model is based on disaster phases and a disaster literacy that combines literacy qualifications:

Disaster literacy is individuals’ capacity to access, understand, appraise, and apply disaster information to make informed decisions and to follow instructions in everyday life concerning mitigating/prevention, preparing, responding, and recovering/rehabilitation from a disaster in order to maintain or improve quality of life during the life course. ¹³

Based on this recognition, Caliskan and Üner ¹³ developed a logical conceptual framework model that outlined the main dimensions of disaster literacy and showed individual, social and communal, socioeconomic, cultural, and environmental factors that could affect these, as stated in the literature. This model emphasizes that the process of processing disaster information is a lifelong transforming process from the individual to the social level. The core of the model depicts a 16-cell matrix that outlines the processes of access, comprehension, evaluation, and implementation for the 4 phases of disaster and for the 4 areas of the disaster information processing (Table 1) ¹³ :

• Reducing or preventing the danger as part of the mitigation phase

• Reducing the negative outcomes of a disaster as part of the preparedness phase

• Reducing damage and loss as part of the response phase

• Restructuring actions in family, social, professional, and political dimensions as part of the recovery phase

Table 1. DLS Disaster Literacy Matrix

Çalışkan C, Üner S. Disaster literacy and public health: a systematic review and integration of definitions and models. Disaster Med Public Health Prep. 2021:15(4);518-527.

Scale Development

In the methodologically epidemiological study, a systematic and structured development process based on a conceptual model was followed:

Item development

The items that measured each of the 16 cells of the matrix were developed for measuring the capacity of individuals to read, understand, and use this information by following the instructions that must be followed for the 4 phases of disasters. The item pool was developed from the instructions that individuals should follow against disasters and from the terms on the basis of these instructions. The distribution of the terms taken as a basis during the item development process by disaster phases is given below. Some terms are in more than 1 phase ^14–16 :

• Terms of the Mitigation/Prevention phase: Risk reduction, risk outcome reduction, risk avoidance, risk acceptance, risk transfer, resilient structure, building codes and regulatory measures, structural design, displacement, community shelters, barrier diversion or retention systems, detection systems, environmental control, physical change, education, behavior design, non-structural physical designs, financial applications, social bonding, sharing and dissemination.

• Terms of the Preparedness phase: Planning, exercise, education (public), equipment, media usage, change of behavior, public messages, awareness, warning.

• Terms of the Response phase: Warning and evacuation, risk avoidance, last minute preparation, first aid, warning systems, use of resources, search and rescue, moral value, coordination.

• Terms of the Recovery/Rehabilitation phase: Planning, public information, coordination, health and social services, natural and cultural resources, insurance, money, materials and economic recovery, housing, infrastructure systems.

Expert consulting

Thirteen experts having studied in the field of health literacy and disaster were consulted for the requirement, clarity, and specificity assessments of the draft items developed. The items that did not comply with the conceptual model and justification of the survey or had no direct or indirect relevance to 16 subgroups were eliminated. Only the items that were indirectly associated with the objectives of the survey were combined with the other items. Within the scope of the expert reviews regarding 95 draft items developed, 11 items were removed due to similarity and difficulty with the other items.

Language control

Prior to the pilot study of the draft items, the compliance of the items with the rules of meaning and grammar was evaluated and regulated by a linguist. In the pilot study, feedback was received from the participants about the design, clarity, and content of the survey.

Draft survey

The draft scale consists of 84 DLS items. The answering system of the draft DLS items was prepared in the 5-point Likert type from “very difficult” to “very easy.”

Field test

The study, which was approved by Hacettepe University’s Non-Interventional Clinical Research Ethics Board, included a group of people between ages 18–60 years and lived in urban areas. The number of participants was equally distributed according to the subgroups of gender (female, male), age (18-34, 35-44, 45-60), education (literate, elementary and secondary schools, high school, university), and self-reported economic situation (bad, moderate, good), and consequently reached 864 people. The quota sampling method was used when the participants were included in the study. In order to conduct the test-retest method in the study, the same survey was reapplied between 2 and 3 weeks for 168 people who agreed to participate in the second study. By means of 6 pollsters trained for field testing, the participants were given surveys and they completed them under observation.

Analysis

The SPSS 19.0 statistical software package (IBM; Armonk, New York, USA) was used for the analysis. The analysis related to the study was collected under 3 main titles, described in the following texts.

Validity analyses

Content Validity Ratio (CVR), Content Validity Index (CVI), a normality test, and Exploratory Factor Analysis (EFA) were conducted within the scope of the validity analyses of the draft DLS items. According to the Lawshe technique and feedback from 13 experts, the items were improved and the CVR and CVI analyses were carried out. In deciding whether the items should remain in the draft survey, the CVR criterion was calculated as ≥ 0.54 and the CVI criterion was chosen as > 0.67, while these were calculated separately in each dimension. ^17–19 The skewness and kurtosis values of each item were considered to be normally distributed because they were in the range of ±2.0. ²⁰

Since the study consisted of different components, an EFA was performed to uncover the structures of the variables whose structure was not fully known but was present. ²¹ Four separate EFA studies were carried out in the phases of Mitigation, Preparedness, Response, and Recovery. In order to achieve better results in these 4 phases and to adapt to the conceptual framework of the DLS (see Table 1), the factors above 0.46 in Mitigation, 0.50 in Preparedness and Recovery, and 0.55 in Response were used in load values. The difference between the 2 items was at least 0.15 ²¹ so that the factor loads of these were not counted as overlapping. The Principal Components Analysis was used to reveal disaster-related factors. At the end of the analysis, the value of 1 or higher was taken for the determination of the number of factors that emerged. However, after the first EFA in the Recovery phase, the number of factors was entered as 4 in the “Factor Analysis: Extraction” section, and the distribution of the existing items was reexamined. It was, then, observed that the items were loaded in their factor groups. For the determination of the factors, the value for Mitigation, Preparedness, and Response was accepted as 1. ²¹ In the Recovery phase, this was overlooked because the number of factors was entered manually. The oblique rotation method was preferred in the study as the concepts between the phases of disaster are transitional. For the suitability of the samples for factor analysis, the values were taken as > 0.5 for Kaiser-Meyer-Olkin (KMO), P < 0.05 for Bartlett Test, and > 0.5 for Anti-Image Correlation Matrix. ²¹ The communality value, which refers to the variance that a variable shares with other variables in the analysis, was taken as follows: > 0.508 in Mitigation, > 0.500 in Preparedness and Recovery, and > 0.400 in Response. ²¹ The items whose factor load value was below the determined value under relevant phases were excluded from the analysis, and the analyses were repeated.

Reliability analyses

Within the context of the scale’s reliability analysis, Test Re-test (P < 0.05), Intraclass Correlation Coefficient (P < 0.05), Item Analysis, Cronbach’s alpha, Split Half Reliability (Spearman-Brown), Collectible (Tukey’s Additivity Test) (P < 0.05), Response Bias (Hotelling’s T-squared) (P < 0.05), and Floor and Ceiling Effect (< 20%) tests were used. ^18,21–23 For the item analysis, Pearson’s correlation coefficient was taken as ≥ 0.25, ¹⁸ Cronbach’s alpha value was taken as > 0.80, and the total correlation coefficient of the corrected items was taken as > 0.30. ²¹ Reliability calculations were made for the general and disaster phases of the scale.

Scoring

After taking the Z-score values of the items left in the item pool, the cut-off points of the DLS, which took its final version, were determined as SS -1 and below, SS -1 and 0, SS 0 and +1, and SS +1 and above.

In terms of ease of calculation, the total score was standardized with the help of the following formula, ² which would take a value between 0 and 50:

$$\rm{Formula = Index = (arithmetic average-1) x (50/4)}$$

• Index = Index original to the calculated person

• Arithmetic average = Average responses to each item

• 1 = Lowest possible value of the average (causes the lowest index to be 0)

• 4 = Average range

• 50 = Highest selected value for the new criteria

• 0 shows the lowest DLS and 50 shows the highest DLS in the scale.

Results

The process leading to the final version of the DLS is given below in 4 steps, and the final version of the DLS is given in Appendix 1.

Field Results

Due to the design of the study, the number of the subgroups of gender, age, education, and income status was evenly distributed. The average age (SD) of the participants was 39.4 (10.9), the median was 40.0, the youngest age was 18, and the oldest age was 60; 71.2% of the participants were married, 55.9% were actively working, and 37.7% (n = 326) of those who participated in the survey had experienced at least 1 disaster.

Validity Results

The skewness and kurtosis values of each item in the data set were examined with the normality test. Since the skewness and kurtosis values of the items were ±2.0, it was considered to be normal.

The Test Re-test was found to be significant in the Pearson’s correlation coefficient calculation, which was conducted to determine whether there was a relationship between the first and second responses that the participants (n = 168) gave to all scale items (r = 0.79, P < 0.01). The Cronbach’s alpha value in the Intraclass correlation coefficient calculation, another analysis carried out for the same purpose, was high and had a significant level (Cronbach’s α = 0.983, P < 0.001). In the item analysis, the total score correlation coefficients of 84 draft items were found positively significant between r = 0.41 and r = 0.59 (P < 0.001). The item and total correlation coefficients were > 0.25 and significant, and the EFA was consequently started.

The KMO value in the EFA of the Mitigation phase of the draft items was determined as > 0.6 and therefore sufficient, and the Bartlett test was found to be significant (P < 0.01). As stated in the method, since the communality of the items was > 0.508 and the load values of the factors (r < 0.500) were inadequate, the analyses were repeated 4 times and 4 items were subsequently removed. When the table of the rotated components of the last analysis was examined, a 4-factor structure was reached (Table 2) and the total explained variance eigenvalue was found to be greater than 1. The total variance of the 4 factors was 60.17%.

Table 2. Distribution of the EFA factor load values of 61 items in the scale

The KMO value in the EFA of the Preparedness phase of the draft items was > 0.6 and sufficient, whereas the Bartlett test was found to be significant (P < 0.01). As stated in the method, since the communality of the items was > 0.500 and the load values of the factors (r < 0.500) were inadequate, the analyses were repeated 4 times and 7 items were removed. When the table of the rotated components of the last analysis was examined, a 4-factor structure was reached (see Table 2) and the total explained variance eigenvalue was found to be greater than 1. The total variance of the 4 factors was 63.12%.

The KMO value was > 0.6 and sufficient in the EFA of the Response phase of the draft items, and the Bartlett test was found to be significant (P < 0.01). As stated in the method, since the communality of the items was > 0.400 and the load values of the factors (r < 0.550) were inadequate, the analyses were repeated 5 times and 8 items were removed. When the table of the rotated components of the last analysis was examined, a 4-factor structure was reached (see Table 2) and the total explained variance eigenvalue was found to be greater than 1. The total variance of the 4 factors was 61.72%.

The KMO value was > 0.6 and sufficient in the EFA of the Recovery phase of the draft items, and the Bartlett test was found to be significant (P < 0.01). As stated in the method, the communality value of the items was > 0.500 and the load values of the factors were r > 0.500. Since the communality values of the 4 items were insufficient, they were removed and the analyses were repeated. The communality values were suitable for the new analysis. However, the total variance emerged as a 3-factor structure. However, a 4-factor structure was obtained in the Mitigation, Preparedness, and Response phases. At the same time, when preparing the items in the first place, the items were prepared according to the sentence/verb structure specific to the groups within the literacy itself (16 matrix structures in Table 1). When these were considered, it was observed that 3 items were loaded on factors other than their sentence/verb structures. For these reasons, after this stage, the number of factors was entered as 4 to the “Factor Analysis: Extraction” section and the distribution of the existing items was re-examined. When the table of the rotated components of the final analysis was examined, it was determined that the items were dispersed appropriately. When the table of the rotated components of the last analysis was examined, a 4-factor structure was reached (see Table 2), and the total explained variance eigenvalue was found to be greater than 1. The total variance of these 4 factors was 66.01%. As a result of the validity analyses, 23 items out of 84 items were removed and the reliability analyses were started.

Reliability Findings

The Spearman-Brown test, which was conducted to avoid some of the drawbacks that would be caused by using the same test twice on 1 group in order to determine the reliability of a scale in written tests, was high with 0.975. This test was conducted through the first 84 items to determine whether the responses to the items were reliable before the EFA. However, here, it was given under the reliability section.

Whether the participants’ responses to the draft items were equal was evaluated with the Hotelling T ² test. As a result of the test, the Hotelling’s T-square value was found as T ² = 1770.196, P < 0.001 and there was no response bias.

The base and ceiling impact value percentages of the draft items were calculated below 20% in all phases, and the responses to the items were found to be homogeneous.

The Cronbach’s alpha value of the draft items is 0.954, which is quite high. The Mitigation Cronbach’s alpha value is 0.874, whereas the Preparedness Cronbach’s alpha value is 0.860, the Response Cronbach’s alpha value is 0.831, and the Recovery Cronbach’s alpha value is 0.883. Since the adjusted correlation numbers of the items were > 0.30, it was decided that the remaining 61 items should remain after the EFA (Table 3).

Table 3. Reliability values of the draft items

The DLS items were found to have collectible properties based on Tukey’s collectible test results (P < 0.001).

DLS Cut-Off Points

The DLS is a scale of self-report developed to assess disaster literacy in people ages 18–60 years. The conceptual framework includes 4 disaster-related phases (Mitigation, Preparedness, Response, and Recovery) and learning processes on disaster-related decision-making and implementation (access, comprehension, decision-making, and implementation). The conceptual framework of the 61 items in the DLS, each of which was based on reconciliation-based item selection process and analysis, consists of 4 phases and 16 subgroups. The number of questions in each group varies between 2 and 6 items. The full expression of each item is given in Appendix 1. Each item is rated and evaluated as 1 point (“Very difficult”), 2 points (“Difficult”), 3 points (“Undecided”), 4 points (“Easy”), and 5 points (“Very easy”). There are no reverse items in the scale. The total score from the scale is between 0 and 50.

With the help of the formula given in the method, the cut-off points of the scale were determined with the SS values of the mean score. Table 4 provides average item cut-off points by average and their equivalents in the 50-point system.

Table 4. Cut-off points of the DLS scores and their equivalents in the 50-point system

Discussion

Studies in the field of disaster literacy have just started to increase, ²⁴ and with the increase of knowledge and experience to be learned from this field, people’s vulnerabilities will decrease and the field will become a popular study topic. With this increased interest, the demand for tools that measure disaster literacy will increase. ¹³ The DLS is not specific to a particular group but aims to measure the disaster literacy of the general population. However, it can be said that there is a technical focus because disaster literacy is a specific form of health literacy and is therefore relevant to a specific target group. In addition, as it captures a broad public health perspective, it resolves the processes of disaster information-acquiring, understanding, evaluating, and practicing in the Mitigation, Preparedness, Response, and Recovery phases to alleviate the vulnerability of people. In other words, as shown by Çalışkan and Üner, ¹³ the DLS investigates the factors and outcomes that prevent a person from accessing disaster information in their daily life so that it would help reduce risk of vulnerability while moving between home, school, work and social life, and promoting good behaviors. ¹³ As shown in the integrated disaster literacy model, disaster literacy can be considered a formal, informal, and disaster training trilogy; therefore, disaster literacy can be seen as a whole of a range of improvable qualifications.

The DLS is based on a definition and conceptual model that Çalışkan and Üner ¹³ compiled and derived from various disaster literacy studies ^11,25–30 by examining important health literacy studies. ^2,3 The DLS has both a specific form of health literacy and a technical focus as it relates to a particular area of interest. Therefore, the DLS basically references the public health approach as it promotes the protection and improvement of the health of populations such as health literacy. In this direction, the opinions of many experts, who have a professional education background and experience in public health and disaster science, were consulted during the development of the measurement tool. Considering the feedback from experts, related analyses, and the description of Caliskan and Üner ¹³ regarding disaster literacy, a match regarding the purpose of the study was made by sticking to the original ideas presented in the conceptual model and matrix.

The definition, conceptual model, matrix, and formula of the DLS were inspired by significant health literacy studies. ^2,3 However, 1 aspect that distinguishes the DLS from health literacy studies is related to the item production process that creates the DLS. The terms of modern disaster management are used in the item production process, which is presented in detail in the method section. This means that each item in the DLS matches 1 or more of the disaster terms.

The first item pool was derived from 46 different terms related to disasters. As a result of the expert evaluation and the validity and reliability analyses of the item pool, 34 items were dropped from 95 items, and 61 items were reached. When the terms contained in the remaining items were examined, it was seen that no terms were left out, and that each term included 1 item at least once. For this reason, it can be said that the scale developed covers all the dimensions of the disaster comprehensively. However, when the number of items that are currently long is added to the questions to determine the factors affecting disaster literacy, the number of items will increase more. Therefore, the response percentage of the respondents may decrease as the response time will be longer. In future studies, studies can be conducted to reduce the number of items by preserving the structure expressing the integrity of the items.

The DLS scale was derived from 46 terms that were transitive in the stages of disaster management related to the concept of literacy. ^14–16 The terms in the disaster stages and the areas of literacy competence were covered. Thus, the emergency response options of the individual specific to each phase were listed before, during, or after disasters. In addition, since the DLS did not have a similar work in the development process, the DLS was developed using a systematic way detailed in the method.

A comprehensive disaster literacy index was created using scores from 61 items that measured disaster literacy. The index was adapted from a formula used in the European health literacy project to ensure computational ease and to compare this scale with the item pool created as a result of validity-reliability studies in other languages. ² The formula, as in health literacy studies, ³¹ is seen as an equation that makes comparisons easier by standardizing DLS practices between 0 and 50 points. In this standardization, the overall metrics of the DLS, which include those who answered the scale, were found by determining the cut-off points as inadequate, limited, adequate, and perfect. In addition to global metrics, scores can be resolved separately, specific to phases and groups. Thus, specific studies can be carried out within the framework of the practice of disaster training modules, which will be developed in accordance with the general and lower phases of the scale, and within the framework of the individual’s access, understanding, interpreting, and implementing information related to disasters.

There are some restrictions on the design and development of the DLS. The DLS items can have a sorting effect because they are given within the usual course of their phase and disaster information access processes. In addition, since the DLS items carry a positive sensual burden, it may indirectly affect the participant. Therefore, it is assumed that those who agreed to participate in the study filled out the survey correctly. A standard test, which can be compared to the recently developed survey, does not exist.

Conclusion

This study explained the design and development process of the Disaster Literacy Scale in Turkish society. The objectives, scopes, and limitations were defined and described by giving a detailed explanation regarding the steps in the design and development process of the DLS for other communities that could use the tool. The DLS development process resulted in 61 core items that eliminated the difficulties reported in the Mitigation, Preparedness, Response, and Recovery phases of disaster as well as in information about disaster information, understanding, evaluating, and practicing processes. The DLS has positive items and was prepared with the 5-point Likert answering technique from very difficult to very easy. The DLS scores were divided into metrics in itself with the help of a formula that determined the disaster literacy level of an individual. Each metric, phase, and field-specific training module can be developed to support the continuation of an individual’s good condition. In addition, countries and societies are affected at different levels from disasters that occur in this day. It is important to predict the consequences of a disaster in determining these differences in order to be able to determine the response efforts and to analyze the preparedness levels of countries and societies. For this reason, the DLS with conceptual-based and versatile features can be used as an indicator of development by official institutions and organizations at the international level after countries make the necessary adaptation studies to their communities.