The development of a measure of orthographic knowledge in the Arabic language: A psychometric evaluation

Abstract

Although Arabic is an official language in 27 countries, standardized measures to assess Arabic literacy are scarce. The purpose of this research was to examine the item functioning of an assessment of Arabic orthographic knowledge. Sixty novel items were piloted with 201 third grade Arabic-speaking students. Participants were asked to identify the correctly spelled word from a pair of two words. Although the assessment was designed to be unidimensional, competing models were tested to determine whether item performance was attributable to multidimensionality. No multidimensional structure fit the data significantly better than the unidimensional model. The 60 original items were evaluated through item fit statistics and comparing performance against theoretical expectations. Twenty-eight items were identified as functioning poorly and were iteratively removed from the scale, resulting in a 32-item set. A value of 0.987 was obtained for McDonald’s coefficient ω from this final scale. Participants’ scores on the measure correlated with an external word reading accuracy measure at 0.79 (p < .001), suggesting that the tool may measure skills important to word reading in Arabic. The task is simple to score and can discriminate among children with below-average orthographic knowledge. This work provides a foundation to develop Arabic literacy assessments.

Orthographic processing has been defined as the ability to acquire, store, and use orthographic representations (Apel, 2011; Stanovich & West, 1989). Theories of reading suggest that this ability to develop and use acquired information about orthography (i.e., any system of symbols or marks that represent spoken language in print; Chomsky, 1970) may be essential to efficient, fluent reading development (Ehri, 1995, 2005; Ehri & McCormick, 1998; Share, 1995; Venezky, 1999). Awareness of orthographic regularities is believed to help students learn the correct spelling of morphologically complex words that vary in phonological form (e.g., “muscle” and “muscular”) in a way that enables effective decoding (Apel, 2011; Chomsky, 1970; Graham, Harris, & Chorzempa, 2002). Children who experience reading difficulty tend to exhibit deficits on assessments of orthographic processing (Bergmann & Wimmer, 2008; Rothe, Corell, Ise, & Schulte-Körne, 2015).

Much of the current understanding of orthographic processing has been based on results from studies conducted in English or other alphabetic European orthographies (Share, 2008), though there has been a substantial increase in studies examining nonalphabetic orthographies (e.g., radical awareness as an important orthographic processing unit in Chinese reading development; Ho, Ng, & Ng, 2003; Kuo et al., 2015). Findings from one language may not generalize to languages with different orthographic systems, making it essential therefore to expand research across all orthographic systems (Daniels & Share, 2018; Share & Daniels, 2015).

Arabic, a language whose orthography differs substantially from European orthographies, is spoken by 420 million people worldwide (Gordon, 2005), while Arabic orthography is used by millions of non-Arabic speakers (e.g., Pashto, Persian, & Urdu; Mirdehghan, 2010). Furthermore, Arabic is one of the fastest-growing languages in North America, reflecting increasing trends of linguistic diversity across the globe (Brown, 2016). The number of individuals who speak Arabic in the United States was estimated to be approximately 1 million in 2013 and continues to increase (US Census Bureau, 2015). However, there are currently few examples of psychometrically-sound, standardized measures for assessing Arabic literacy development (Tibi, Tock, & Kirby, 2019). Given the need for validated measures to conduct research to examine connections across languages and orthographies, the development and psychometric evaluation of such measures is essential.

The purpose of the present paper is to begin addressing this gap in the literature through the psychometric evaluation of a tool designed to assess one aspect of orthographic processing in Arabic: orthographic knowledge at the lexical level. The results of this work will contribute to ongoing investigation of orthographic knowledge and, more broadly, reading skill in Arabic. Findings may also add to the larger understanding of cross-linguistic differences in orthographic knowledge and provide a foundation for ongoing evaluation of the role of orthographic processing in literacy development. In the following literature review, we define orthographic knowledge in relation to orthographic processing and reading development, discuss measurement of orthographic knowledge at the sublexical and lexical levels, and describe key features of Arabic orthography essential to consider in assessment development.

Defining Orthographic Knowledge

Although orthographic processing is considered to play an integral role in reading and spelling (e.g., Apel, Wolter, & Masterson, 2006), the precise directionality of this relation remains in question (Conrad & Deacon, 2016; Deacon, Pasquarella, Marinjus, Tims, & Castles, 2019; Querido, Fernandes, Verhaeghe, & Marques, 2020; Roman, Kirby, Parilla, Wade-Woolley, & Deacon, 2009). Some studies have suggested that orthographic processing uniquely contributes to word reading (Cunningham, Perry, & Stanovich, 2001; Pasquarella, Deacon, Chen, Commissaire, & Au-Yeung, 2014; Ricketts, Dorothy, Bishop, Pimperton, & Nation, 2011). Other studies have found no contribution of orthographic processing to gains in word reading (Georgiou, Parrila, & Papadopoulos, 2008), a pattern where word reading gains contribute longitudinally to lexical and sublexical orthographic knowledge (Conrad & Deacon, 2016; Deacon, Benere, & Castles, 2012), or even a bidirectional relation where both skills contribute to each other over time (Querido et al., 2020). These discrepancies may stem from the possibility that the construct of orthographic processing is multidimensional, composed of two distinct underlying aspects: orthographic knowledge and orthographic learning (Deacon et al., 2019). Orthographic learning refers to the dynamic ability to develop orthographic representations (Deacon et al., 2012), whereas orthographic knowledge has been defined as the crystalized store of these recognized representations or patterns (Conrad & Deacon, 2016; Deacon et al., 2019).

Crystalized orthographic knowledge centers on individuals’ established awareness of letter patterns and word-specific orthographic representations stored in memory (Apel, 2011; Conrad, 2008; Deacon et al., 2019; Ehri, 2005, 2014, 2017). Chambre, Ehri, and Ness (2017) noted that orthographic knowledge involves understanding of phoneme–grapheme correspondence, awareness of morpho-graphemic patterns, and familiarity with general conventional spelling rules. This knowledge can exist either at an implicit level or an explicit level (Ouellette & Sénéchal, 2008a, 2008b; Pacton, Perruchet, Fayol, & Cleeremans, 2001; Wolter & Apel, 2010). Sometimes children learn the orthographic regularities in their lexicon implicitly through exposure to orthographic patterns that recur in different words (Cassar & Treiman, 1997; Ehri, 2017; Treiman & Kessler, 2006; Tunmer & Nicholson, 2011). In contrast, these regularities can also be directly taught and/or recognized through metacognition, such that individuals make conscious decisions about the acceptability of the spelling or letter sequences in one’s language (Apel, 2011; Kaefer, 2016).

Orthographic knowledge is integral to many major models of written language acquisition, including the later stages of Ehri’s five-phase model of reading acquisition (Ehri, 1995, 2005; Ehri & McCormick, 1998), Share’s influential self-teaching hypothesis (Shahar-Yames & Share, 2008; Share, 1995; Share, 1999; Tucker, Castles, Laroche, & Deacon, 2016), the Lexical Quality Hypothesis (Perfetti, 2007; Perfetti & Hart, 2001, 2002), and the connectionist model (Seidenberg, 2005; Seidenberg & McClelland, 1989). More recently, Ehri (2005, 2014, 2017) emphasized that it is through orthographic mapping that connections between the spellings, pronunciations, and meanings of specific words are formed and stored in memory. These connections improve qualitatively as children develop and learn to read and spell.

Measurement of Orthographic Knowledge

In broad terms, orthographic knowledge can be measured at the lexical or sublexical level(s). At the lexical level, the pseudohomophone orthographic choice task put forth by Olson, Forsberg, Wise, and Rack (1994) is a classic task used to measure orthographic knowledge (e.g., Deacon, Cossissaire, Chen, & Pasquarella, 2013). This task requires participants to choose the correct spelling of a word from a phonologically matched pair of orthographic representations, with one being correctly spelled and the other a pseudohomophone (e.g., hurt vs. hert). Conversely, tasks designed to measure orthographic knowledge at the sublexical level are based on pseudowords. An example of a typical sublexical assessment of orthographic knowledge in English is the letter sequence judgment task. The task requires participants to indicate whether a given letter sequence would be acceptable (e.g., baff vs. bbaf; Cassar & Treiman, 1997; vaid or vayd; Commissaire, Pasquarella, Chen, & Deacon, 2014).

An advantage of measuring orthographic knowledge at the sublexical level is that sublexical assessments focus on general orthographic knowledge about letter sequences and rules in reading and spelling, rather than orthographic knowledge of specific words (Burt, 2006; Castles & Nation, 2006; Nation, Angells, & Castles, 2007). Any task based on specific words is susceptible to bias based on individuals’ familiarity with the included words. Several studies have even shown that individuals’ performance on sublexical orthographic knowledge tasks is distinct but related to performance on lexical tasks (Deacon et al., 2012; Pasquerella et al., 2014). It is likely that lexical and sublexical tasks for assessing orthographic knowledge measure differing underlying abilities.

In the context of Arabic language, the relation between orthographic knowledge at the lexical and sublexical levels is relatively unclear. As discussed in the next section, the rules governing Arabic orthography are intertwined with morphology and semantics. Consequently, evaluation of orthographic knowledge at the sublexical level in Arabic would require the specification of the morphological, semantic, and orthotactic context for any given item. Given the preliminary nature of this work and the metacognitive demands of such a task, our focus in the present paper is on examining orthographic knowledge at the lexical level in Arabic.

Arabic Orthography

Arabic is a Semitic language with a primarily consonantal orthography (Daniels, 1992). It has 28 letters, two of which can function as either long vowels or consonants. The three Arabic long vowels /a: /, /u: /, and /i:/ also have three corresponding short vowels. Several of the key features of the Arabic writing system are described briefly in the following paragraphs. These include the use of fully vowelized versus partly vowelized writing, homography, ligaturing, diglossia, dense morphology, occasions of phoneme–grapheme mismatch, and context-dependent orthotactic rules (Abu-Rabia, 1996, 1999; Tibi & Kirby, 2018, 2019). Each of these characteristics were directly considered in the development of the current task.

The Arabic long vowels are always explicitly represented in the writing system, whereas the three short vowels (/a/, /u/, /i/) may or may not be represented by diacritics placed on top or underneath the consonants (e.g., -, -, -). Depending on the medium, text may be fully or partly vowelized. Generally, texts written for beginning readers or religious contexts are fully vowelized and include orthographic representations of short vowels using diacritics (Abu-Rabia, 1996, 1999), but texts for adults generally do not include the short vowels, hence partly vowelized. The full vowelization of Arabic script allows for a higher degree of grapheme–phoneme correspondence. Despite the high grapheme–phoneme correspondence possible in fully vowelized texts, Arabic orthography has other features that make it more complex than alphabetic orthographies that are transparent (Share & Daniels, 2015; Tibi & Kirby, 2018).

Because words can be written without short vowels, Arabic orthography is also characterized by homography. This refers to words that are spelled in the same way, but have different pronunciations. For example, when the three consonant-word /ktb/ appears without short vowels, there are multiple possibilities for reading it: it can be read as /kataba/ “to write,” /kutub/ “books,” or /kutiba/ “was written.” In a partly vowelized text, the reader must infer the correct meaning and pronunciation from the semantic context and the regularity in the morphological information provided by the root and the word pattern (Boudelaa & Marslen-Wilson, 2001; Tibi, Edwards, Schatschneider, & Kirby, 2020; Tibi & Kirby, 2019; Tibi et al., 2019). This ambiguity can be eliminated by the representation of short vowels in the text, because the short vowels explicitly represent additional phonological information, hence restricting the possible pronunciations of the homographic words.

Arabic letters can generally be divided into connecting (i.e., ligatured) letters versus nonconnecting letters. Ligaturing between some consecutive letters is an obligatory feature. Therefore, a word containing such letters must be written in a connected “cursive” way; without this ligaturing, its spelling is considered incorrect. In addition, when ligaturing takes place, the connected graphemes change their shape depending on their word position (initial, medial, or final; for review, see Tibi et al., 2020). For example, the letter “h” in Arabic is written in four different ways depending on its position in the word and whether it is connected to a letter or not. The connectivity (or allography) between letters, in addition to the optional diacritics (short vowels), results in multiple letter sequences being permissible at the sublexical level in Arabic (Tibi & Kirby, 2018).

Diglossia is another linguistic phenomenon noted in the Arabic writing system. Diglossia is defined as a difference between the standard literary form and the spoken form of a language (Ferguson, 1959). For example, the literary form is used in writing and some formal speech situations, whereas the spoken form is used in daily conversations. Diglossia in Arabic has been observed to impact literacy acquisition (Abu-Rabia, 2000; Tibi, Joshi, & McLeod, 2013; Tibi & Kirby, 2018, 2019). Diglossia and its effect on reading and writing has been investigated by Saiegh-Haddad (2004), who reported that the linguistic distance between the standard form and the spoken Arabic exists in all aspects of the language, including phonology, morphology, and semantics. For example, the phoneme /q/ in standard Arabic can be produced as //, or /g/, or /k/ in spoken Arabic depending on the spoken dialect. Eviatar and Ibrahim (2000) argued that Palestinian kindergarten and Grade 1 children living in Israel function like bilinguals because the standard literary form is different enough from their spoken form. For a detailed account on diglossia in Arabic, see Maamouri (2009). For a counterargument on diglossia, see Boudelaa and Marslen-Wilson (2013), who presented experimental evidence showing that the standard and the spoken varieties of Arabic are cognitively processed in similar ways despite their structural, functional, and distributional differences.

A description of Arabic orthography would be incomplete without a mention of morphology. Arabic is characterized by its dense morphology, which involves both linear and nonlinear processes (Boudelaa & Marslen-Wilson, 2001; Tibi & Kirby, 2017, 2019). Unlike Indo-European languages, Arabic words are derived by means of combining two abstract units (consonantal root and the mostly vocalic word pattern) in a nonlinear way. This mechanism allows for the derivation of words from the same consonantal root, hence allowing for orthographic transparency, shared semantics, and morphological complexity. The shared orthographic and semantic qualities between the root and its derived words contribute to reading accuracy (Tibi, 2016; Tibi & Kirby, 2017, 2019 Tibi et al., 2019, 2020), fluency, and comprehension (Layes, Lalonde, & Rebaï, 2015; Tibi & Kirby, 2019; Tibi et al., 2019). We know from research in English that redundancy of letters and letter patterns allow for well-defined orthographic representations which in turn have been shown to be crucial for reading proficiency (Chomsky, 1970; Ehri, 1992; Perfetti, 1992; Rahbari & Sénéchal, 2010; Share, 1999). Arabic also allows for linear morphology when prefixes and suffixes are added linearly to words. The prefixes and suffixes provide information about the grammatical functions of a word such as gender, number (singular, dual, and plural), person, and time. For example, the inflected word /mudarrisata:n/ means “two female teachers.” In this word, the “t” is a morpheme denoting the female gender, and the /a:n/ morpheme represents the “dual” marker; both suffixes are added to the noun /mudarris/ “teacher.” It should be noted here that in the case of the masculine plural suffix /u:/, it is represented orthographically with two letters “”, yet it is pronounced only with one sound /u:/.

There are other situations in which orthographic representations do not match their corresponding pronunciations in Arabic. For example, diacritics are used instead of a letter to represent a phoneme in certain grammatical contexts. Specifically, for an indefinite noun in Arabic, double diacritics are added to the vowelized word to represent the /n/ phoneme although the letter “n” is not written (e.g., “an apple” is written as “”). Mismatches between the orthography and phonology occur in several other contexts in Arabic. Consequently, individuals developing literacy in Arabic must know the rules corresponding to each orthographic context.

Additional examples of phoneme–grapheme mismatch include the use of the definite article /al/ before a group of letters known as “sun letters” (see Saiegh-Haddad & Henkin-Roitfarb, 2014). Sun letters dictate that the preceding /l/ phoneme is not pronounced, despite it being represented orthographically. Saady, Ibrahim, and Eviatar (2015) reported that the rate of missing a target letter was higher in the sun letters context among skilled readers of Arabic. There are also orthotactic rules that place constraints on the sequencing of letters in certain contexts (e.g., the closed /t/ cannot occur in the initial position nor can it be connected to a letter after it). Moreover, the feature of allography, which was described previously, also influences orthotactic rules in Arabic. This is because letters change their shapes depending on the position of occurrence in a word. Using a letter shape in the wrong position (e.g., medial position instead of final) is considered incorrect spelling (e.g., the letter “s” in the final position is written as “” as in the word “teacher” [], but will be deemed inaccurate if written in the middle position as an allograph “”).

The orthographic features of Arabic discussed thus far are components of a complex and multifaceted writing system, which is orthographically deep despite its generally transparent phoneme–grapheme correspondence (Share & Daniels, 2015; Tibi & Kirby, 2018). To decode and spell, speakers of Arabic must draw on their knowledge of the conventional rules present in the written language, phoneme–grapheme correspondence, and the morphological information inherent in its lexical structure. To develop an assessment of children’s orthographic knowledge in Arabic, each of these aspects of the system are important to consider. Further, the underlying dimensionality (or potential divisions of underlying knowledge) of children’s orthographic knowledge in Arabic is relatively unknown. Consequently, evaluation of a wide array of orthographic rules pertaining to morphology, diglossia, ligaturing, and orthotactic rules is essential to the development of a valid assessment tool.

The Current Study

The measure developed and evaluated for the current study was an orthographic choice task (Tibi, 2016). Each item consisted of a pair of words: one correctly spelled target and one incorrectly spelled foil. Foils were in some cases pseudohomophones, and in other cases were pronounced differently from the target. This task is similar to previously established tasks intended to assess orthographic knowledge at the lexical level in English and French (Deacon et al., 2013).

The purpose of the current study was to develop a measure consisting of a set of items with strong relations to the latent underlying construct of lexical orthographic knowledge. Correlations between each item and the total test score were examined, followed by confirmatory factor analysis to determine the dimensionality of the measure. Given the nature of the task, and the variations in types of foils used, it was important to test whether there were any sets of items that clustered together. Such clustering would be indicative of additional dimensions beyond the main dimension of orthographic knowledge. After assessing dimensionality, the measure was further refined by removing poorly fitting items, resulting in a set of well-functioning items, which were retained in the final version of the measure. Finally, children’s total scores on the refined version of the measure were correlated with an external measure of word reading accuracy in Arabic to provide preliminary information about the tool’s construct and potential discriminant validity.

Method

Measure construction

Orthographic choice task

The primary measure of interest within the present paper was an Arabic Orthographic Choice task (based on Deacon et al., 2013) developed by the first author, which included of set of 60 novel items. The items were constructed to evaluate children’s knowledge of the conventional rules of Arabic spelling by targeting the specific characteristics of Arabic orthography noted previously (Tibi & Kirby, 2018). For each item, students were presented with two words written in Arabic: a real word and an incorrectly spelled foil. Children were instructed to choose the word that was spelled correctly.

Of the 60 items, 45 included target–foil pairs that were phonological matches, or homophones. In these cases, the foil can be referred to as a pseudohomophone. The remaining 15 items included target–foil pairs that were not phonological matches when pronounced aloud. The target words were selected carefully to be representative of multiple characteristics of Arabic orthography and appropriate for the age group of interest. The item set was constructed by considering the influences of vowelization, instances of allography, ligaturing, diglossia, the connection between morphology, semantics, and orthography, occasions of phoneme–grapheme mismatch, and context-dependent orthotactic rules.

Because the task was intended for use with children at the elementary school age, all target words were fully vowelized. As discussed previously, Arabic can be written with or without the short vowels (Abu-Rabia & Siegel, 2002). Children typically begin learning to read in Arabic using the transparent orthography, but are expected to develop sufficient skills to read using the more opaque orthography as they get older (Abu-Rabia, 2001; Taha, 2016). Only fully vowelized words were represented in the items to allow for specific examination of children’s ability to decode fully vowelized Arabic writing. Full vowelization also eliminated the potential for homography among correctly spelled items.

To account for the presence of ligaturing and diglossia in Arabic writing, several items were constructed to assess children’s knowledge of these features. To evaluate children’s awareness of ligaturing, foils were constructed with the incorrect letter form based on ligaturing and the letter’s position within the word (e.g., “earthquake” /zil/-/za:l/ “” instead of /zilzal/ “”). To evaluate their awareness of diglossia, foils were constructed to represent a dialectal difference (e.g., “student” /tilmi:z/ “”instead of /tilmi:ð/ “”). It should be noted that diglossia and coarticulation, in some instances, provide opportunities for phonemes to be replaced with phonological neighbors (Asaad & Eviatar, 2013; Tibi et al., 2020) as in the phonemes /ð/ and /z/, or /d/ and //.

Next, items were designed to probe children’s understanding of the interconnectedness between morphology, semantics, and orthography in Arabic (Tibi, 2016; Tibi & Kirby, 2019; Tibi et al., 2019, 2020). Different parts-of-speech such as adjectives, nouns, pronouns, and verbs were all included as target words. These four commonly occurring word classes require different vowel patterns. For a child to identify the correct spellings of words from each word class, a thorough familiarity with the orthographic patterns is critical.

The targets also included words that were relatively simple morphologically and those that were more complex. As noted previously, most Arabic content words are composed of at least two bound morphemes. These include the root and the word pattern. The root provides the general semantic field of the word and the word pattern indicates the grammatical category of the word. Both elements are required for the reader to derive a meaningful word, and distinguish the meaning and function of the word. Moreover, when a prefix and/or suffix is added to these elements forming a simple or complex word, the word is considered morphologically complex (for review on morphological complexity, see Tibi et al., 2020). Words without prefixes, suffixes, and infixes are considered morphologically simple (Boudelaa & Marslen-Wilson, 2005; Tibi & Kirby, 2017). Both morphologically simple and complex words were included in the target word set.

Items were also constructed with consideration of the situations in which graphemes do not map precisely onto specific phonemes in Arabic. In some Arabic words, graphemes represent multiple phonemes or phonemes influenced by coarticulation. To design items to gauge children’s knowledge of these orthographic transparency exceptions, some words with imperfect phoneme–grapheme correspondence (i.e., opaque orthography) were included as targets. The foils were written as would be expected based only on the pronunciation of the target word (e.g., “a Dirham” /dhamεn/ was spelled with the “n” letter at the end instead of double diacritic strokes “tanween”). For a child to distinguish correctly between these target–foil pairs, he or she would need to be aware of the specific contexts in which phonemes and graphemes do not map onto each other perfectly.

Finally, several target–foil pairs were created to evaluate participants’ knowledge of orthotactic rules, which can change based on the position of letters within words (e.g., closed /ta/ in the initial position). Similar to the previous example, words whose orthography illustrate these context-dependent orthotactic rules were selected as targets. Foils were written to match the target words’ phonological pronunciation, but also to include a violation of the context-dependent orthotactic rules.

Word reading accuracy

As part of an external measure designed to assess word reading accuracy, participating children individually completed a set of 87 vowelized word reading items. The words included in the item set were taken from Tibi and Kirby (2019). Words were presented in order of increasing difficulty (Tibi, et al., 2020). Participants were instructed to read each word out loud with its complete vowelization. Children received 1 point for reading the word accurately and zero points for an incorrect response. This tool was included to provide an external comparison for the orthographic choice task. Word reading accuracy has been shown to be positively associated with measures of orthographic knowledge (Apel et al., 2006; Berninger, Abbott, Vermeulen, & Fulton, 2006; Cunningham et al., 2001; Deacon et al., 2013; Roman et al., 2009). Cronbach’s α for the word reading measure was 0.98.

Participants

The study sample included 201 Arabic-speaking third-grade students attending public schools in the United Arab Emirates. The participants were recruited from elementary public schools in Dubai that were randomly selected from a list of elementary public schools provided by the Ministry of Education in Dubai. With approval from the Research Ethics Board, potential participants were then randomly sampled from a list of students enrolled at those schools provided by the United Arab Emirates Ministry of Education and recruited to participate. The children came from diverse geographical locations and completed testing during the first term of the 2014 school year (mean age = 8.08 years, SD = 0.45). All the participants were native speakers of Arabic and attended schools in which the language of instruction is Arabic. The sample was 50% female (n = 99).

The first author, who is a native speaker of Arabic, assessed the participants. The children viewed the orthographic choice items in a paper-based format and completed three practice items before beginning the assessment to ensure their understanding of the task. On the practice items, the evaluator provided corrective feedback as needed. Children were shown the two words presented for each item and instructed to circle the word that was spelled correctly. This forced-choice format yielded items that were scored as either correct or incorrect by the test administrator. Children received 1 point for a correct answer and zero points for an incorrect answer. The participants’ total scores were then obtained by summing the points earned across the full measure.

Data analysis

Several steps were taken to ensure retention of the most valid and reliable items for the orthographic knowledge measure, providing the best balance of statistical fit and theoretical content validity. These included descriptive statistics, evaluating the dimensionality of the measure, measure refinement based on item loadings, and reliability analysis of the final item set. Each step included a review of the findings to assess their consistency with the rules of Arabic orthography. This was done to reduce the likelihood of spurious findings that could negatively impact the measure’s content validity.

To obtain a general overview of item functioning, item-level percent correct, its associated standard deviation, and corrected item-total correlations were examined. These descriptive statistics were inspected for evidence of problems in item functioning (e.g., all children responded to a single item correctly or incorrectly). Any item that (a) was answered correctly at a rate above 95% or below 5% and (b) had a negative item-total correlation was flagged. All flagged items were then reexamined by the first author for inconsistencies with their theoretical construction and intended difficulty. Any item with a mismatch between its observed percent correct and theoretical construction was removed from subsequent analysis.

Dimensionality

The underlying factor structure of the items was preliminarily evaluated using a confirmatory factor analysis approach with unweighted least squares means and variance (ULSMV) estimation in Mplus 7.4 (Múthen & Muthen, 1998–2015). ULSMV was selected over weighted least squares means and variances estimation because of the relatively small size of the sample, which can negatively bias the quality of the weighted least squares means and variances tests of model fit (Forero, Maydeu-Olivares, & Gallardo-Pujol, 2009). Although the measure was written to assess a single underlying construct, the creation of items with different types of targets and foils may have resulted in clustering of children’s responses by these characteristics. Unidimensionality could not be assumed without adequate vetting. Failure to account for any underlying similarities among subgroups of items within the full measure would violate the assumption of unidimensionality and consequently invalidate the use of the measure as a single scale (Yang & Kao, 2014). Therefore, five competing models, constructed based on the item characteristics, were compared against a unidimensional (one-factor) model of orthographic knowledge.

The base model, the one-factor model, served as the comparison for each competing model and is shown in Figure S.1 (Model A). All the competing models were tested against this model to assess change in model fit relative to the parsimony of the model. Because the unidimensional model was the simplest and most parsimonious model tested, any competing models would have to improve the fit significantly to suggest rejection of the unidimensional model. Criteria for model fit statistics were based on Kline’s (2005) for indicating a good fit. That is, a root mean square error of approximation (RMSEA) below 0.10, a comparative fit index (CFI) above 0.90, and Tucker–Lewis index (TLI) above 0.90. Model comparisons were conducted using the DIFFTEST option for chi-square difference testing in Mplus (Muthén & Muthén, 1998–2015).

One set of competing models was specified based on features of the target words and is shown in Figure S.2. First, a model allowing for clustering of children’s responses by the word class of the target item (adjective, noun, pronoun, or verb, as depicted in Model B) was compared against the unidimensional model. Of the 60 included items, 8 were classified as adjectives, 33 as nouns, 4 as pronouns, and 15 as verbs. Next, a model accounting for the morphological complexity of the target words was tested (simple vs. complex as defined by Boudelaa & Marslen-Wilson, 2005, and Tibi et al., 2020; depicted in Model C). A total of 46 items were identified as simple, whereas 14 were complex. A model based on the phoneme–grapheme correspondence of the target words was also tested. This model included classifications of the words as fully transparent (i.e., each phoneme corresponded with a single, regular grapheme within the target word) or more opaque (e.g., coarticulation, “sun letters,” or tanween). For a coarticulation example, the /d/ consonant is pronounced as an emphatic // because of the adjacent emphatic // “” letter as in the word “frog” (). In the examples of “sun letters,” the /l/ letter in the determiner “the” /al/ is not pronounced when one of the “sun letters” follows /al/. Another example of orthographic opacity is when two different graphemes (regular alef “”or “broken alef” “”) share the same sound /a:/. It is worth noting that in the case of the broken alef ”, “” another factor contributes to its opacity is its visual similarity with the letter “” (y) with only two dots underneath the letter “” (y) making both letters visual neighbors (Asaad & Eviatar, 2013). Slightly over half of the items were classified as transparent (n = 35). The remaining 25 items were classified as opaque (Model D).

Two additional models, specified based on the differences between the targets and foils, were compared against the unidimensional model and are illustrated in Figure S.3. The first model was a phonological match model, where items that included target–foil pairs with different pronunciation were loaded onto one factor and those that included target–foil pairs with the same pronunciation were loaded onto a second correlated factor (Model E). The second model was specified based on the general rule of Arabic orthography violated. Items including foils that violated rules of morphological orthography loaded onto one factor, those that violated rules of orthotactic loaded onto another factor, and those that violated rules of phoneme–grapheme correspondence were loaded onto a third factor (Model F). As an additional check of the dimensionality of the measure, the eigenvalues were plotted in a scree plot for the model with 59 items.

Measure refinement

Upon identification of the model best representing the underlying factor structure of the measure, items that did not appear to fit into that structure were systematically examined for consistency with their theoretical construction and removed if appropriate. We retained items that both fit well, defined having a standardized loading onto an identified factor of Arabic orthographic knowledge of 0.30 or higher, and that matched the theoretical framework of the scale. The overall measure and individual item functioning were reevaluated at each step of this process. All items identified as not fitting well through examination of factor loadings were reviewed and discussed carefully prior to removal from the scale. Chi-square difference tests were used to reassess the fit of the scale after each item or group of items was removed (through constraining item loadings on the underlying factor to zero). A nonsignificant chi-square test indicated that removing the item from the scale did not significantly worsen the fit of the overall model. Items were removed in order of their fit within the model, quantified by their factor loading. Groups of items that exhibited the same size loadings were removed simultaneously.

To obtain an estimate of the internal consistency reliability for the final set of items, McDonald’s coefficient ω (or ω total; McDonald, 1999) was computed using the item loadings obtained from the final factor analysis. Coefficient ω is considered a robust metric of internal consistency reliability (McNeish, 2017). Coefficient ω also was for item loadings onto the underlying factor to differ in magnitude. Given that the loadings are not expected to be consistent for all items, this metric is a good estimate of internal reliability. Coefficient ω may be interpreted similarly to coefficient α (McDonald, 1999), with values below 0.80 being questionable and those above 0.90 being preferred.

Upon identification of the final set of items, two models were examined to evaluate individual item functioning within the context of the final scale. A two-parameter model, which allows items to vary in both difficulty and discrimination, was compared against a one-parameter model, which allows items to vary in difficulty but holds discrimination constant. Both models were estimated using ULSMV estimation and compared using the DIFFTEST option in Mplus (Muthén & Muthén, 1998–2015).

External validity

The final set of orthographic knowledge items were then scored according to the final retained model. A Pearson’s r correlation coefficient was obtained between children’s scores on the refined orthographic knowledge task and those on the word reading accuracy measure. Although the external measure is empirically limited, this correlation may provide preliminary information about the relations among literacy constructs in Arabic. In addition, it serves as an indicator of whether or not the orthographic choice task is measuring its intended underlying construct of orthographic knowledge in Arabic.

Results

Descriptive findings

Initial examination of the descriptive statistics for the full 60 items (see Table 1) revealed that the percentage correct values for the items ranged from 31% correct to 99% correct, with an average of 67% (SD = 0.16). One item met criteria for exclusion based on the descriptive results. That item both (a) yielded a percent correct value above 95% and (b) had a negative item-total correlation. Upon theoretical examination of the item, it was determined to be functioning differently than originally anticipated (i.e., no reasonable theoretical rationale supporting the high percent accuracy obtained for the item) and was removed from subsequent analyses. No other items exhibited percentage correct values above 95% or below 5% within the participant sample.

Table 1. Descriptive statistics for all items

Dimensionality

Results from the confirmatory factor analysis revealed that the 59-item unidimensional model (see Model A shown in Figure S.1) did not meet criteria for good fit to the data, but may be considered to be in the acceptable range: χ² [1652] = 1725.083, RMSEA = 0.015, 90% confidence interval [CI] [0, 0.023], CFI = 0.875, TLI = 0.870). None of the competing models resulted in a significantly better fit to the data. Model fit statistics for each of the models tested are provided in Table 2, along with the results from the chi-square difference tests used to compare the models against the unidimensional model. In addition, the scree plot of eigenvalues supported a one-factor solution, with a clear elbow in the plot between one and two factors (see Figure 1).

Table 2. Fit indices for models of Arabic orthographic processing

Note: Δχ² is reported for the model comparisons against the one-factor model.

Figure 1. Scree plot showing eigenvalues for model with 59 items.

Measure refinement

Of the 59 items included in the dimensionality analysis, 27 items loaded onto the unidimensional orthographic knowledge factor at less than 0.30. Following the preestablished steps for item evaluation, these items were systematically reexamined and removed as determined to be appropriate. None of these item removals resulted in a significantly worse fit for the overall scale.

The final scale included 32 items loaded onto a single underlying factor, χ² (464) = 502.46, RMSEA = 0.020, 90% CI [0, 0.032], CFI = 0.933, TLI = 0.929. The final items included target words representing all four of the word classes originally targeted, targets with simple and targets with more complex morphology, and transparent and more opaque target words. Similarly, the foils were representative of those originally included: both pseudohomophones and non-pseudohomophones were included in the final set of items. The only orthographic feature originally targeted that was not represented in the final item set was the addition of a grapheme to a foil when conventions of Arabic dictate that it should not be written.

Constraining the item loadings (i.e., discrimination) to be equivalent did not result in a significantly worse fit to the data: Δχ² (31) = 37.57, p = .194. Consequently, this constraint was retained. Item results from are provided in Table 3. Item characteristic curves, which represent the likelihood of a correct response on a given item conditional on a child’s underlying ability, are shown in Figure 2. The test information function curve is provided in Figure 3. Both figures suggest that the developed scale functions optimally for children with lower levels of performance. The sum score on the refined tool was consequently obtained, as indicated by the one-parameter model, and correlated with the external measure of word reading accuracy. Results revealed a significant, positive correlation between children’s sum scores on the two measures, r = 0.793, p < .0001, 95% CI [0.734, 0.839].

Table 3. Item characteristics obtained from final one-parameter model estimated with unweighted least squares means and variance (ULSMV)

Note: Items are ordered from least difficult to most difficult. Item discriminations were fixed at 0.547 (SE = 0.032). Item difficulties can be interpreted as the (z-scored) underlying ability level needed for a child to have a probability of .50 of answering the item correctly. The underlying latent trait mean was set to zero, with a SD of 1.

Figure 2. Item characteristic curves for final set of items. Note. Each line represents a different item’s item characteristic curve, which is a function of children’s underlying ability (average ability = 0) and the probability of responding correctly to that item.

Figure 3. Test information function curve for final set of items.

McDonald’s coefficient ω total was computed as a measure of internal reliability for the subset of items retained in the final scale. The model met the assumption of unidimensionality required for this reliability estimate (McNeish, 2017). Given the obtained factor loadings and standard error, the estimate for coefficient ω was 0.987.

Discussion

The purpose of the present study was to examine the psychometrics of a tool designed to assess orthographic knowledge at the lexical level in Arabic. The tool was an orthographic choice task and included 60 items designed for monolingual Arabic-speaking children in Grade 3. The items were written to evaluate a range of skills based on key features of Arabic orthography (e.g., morphological, grammatical, orthographic, diglossic, and coarticulation features). Measure refinement yielded a set of 32 items, which emphasized a range of different features of Arabic orthography and loaded reliably onto a single underlying latent factor.

The final item set appeared to discriminate particularly well among children with lower levels of orthographic knowledge relative to the rest of the sample. This was evidenced by the item characteristic curves, test information curve, and results from the one-parameter model. Children with ability levels below average (θ below zero) varied sufficiently in their performance on the items to allow for more precise measurement of their ability. The measure discriminated less effectively among children with very low orthographic knowledge (i.e., 2 SD below average) and among children with higher ability levels (i.e., 0.75 or more SD above average).

Results suggest that orthographic knowledge at the lexical level may be a unidimensional construct in Arabic. The single-factor model was found to have the best fit, and to explain children’s performance on the task better than all the competing multidimensional models. The competing models tested a variety of other possible underlying factors representing orthographic features of Arabic, none of which were found to improve the model fit significantly. Although further work is needed to examine the dimensionality of Arabic orthographic knowledge, and orthographic processing generally, on a broader scale, these preliminary results suggest that lexical orthographic knowledge may be a unitary ability indicated by individuals’ ability to distinguish correct versus incorrect spelling in Arabic. The unidimensionality of our findings is in accordance with findings by Commissaire et al. (2014), who found a one factor model of orthographic knowledge to capture both lexical and sublexical tasks in both French and English, suggesting a common orthographic knowledge skill across languages and variables.

The significant correlation between the final item set and the external measure of word reading accuracy are consistent with prior work that has identified positive associations, and potential overlap, between orthographic knowledge and word reading measures (Cunningham et al., 2001; Deacon et al., 2013, 2019; Mimeau, Ricketts, & Deacon, 2018; Pasquarella et al., 2014; Tibi & Kirby, 2019). At a minimum, the observed relation suggests that this Arabic orthographic choice task measures a literacy-focused construct closely associated with Arabic word reading. The separability of, and potential directionality between, these constructs requires further examination (e.g., Deacon et al., 2019). At present, these findings suggest that the developed measure has utility in assessing a construct related to reading in Arabic.

Limitations

This study has several limitations that should be noted. Perhaps most importantly, there is some discord regarding the construct validity of the orthographic forced choice task at the lexical level. Some scholars suggest that this task could be measuring the same construct as word reading (Deacon et al., 2019). In other words, this measure may be susceptible to circularity of measurement (Burt, 2006; Castles & Nation, 2006; Deacon et al., 2012), a valid concern given that the external criterion measure included in the present study was a word reading task, and that a high correlation that was found between students’ scores on the orthographic choice and word reading accuracy tasks. Deacon and colleagues (2012, 2019; Mimeau et al., 2018) suggested that this measure could be tracking the outcome of reading rather than an ability that contributes to reading development. They further noted that the potentially interconnected nature of the measures would apply to both the lexical and sublexical levels (Deacon et al., 2012). Given the need for standardized measures and the evidence in support of the use of orthographic choice tasks (Conrad, 2008; Deacon et al., 2013; Olson et al., 1994), this research can serve as a basis for comparison for future work utilizing alternative tasks. As more Arabic-language assessments of orthographic knowledge and other related skills are developed, the validity of this measure can be assessed by examining its function relative to other validated tools. This will assist practitioners and researchers to better understand both the utility of the developed measure and similar measures.

It is also worth highlighting the fact that all the study participants spoke the same dialect of Arabic and were in third grade. Consequently, generalization of the findings should be made with caution. Results need to be replicated with additional samples of students from other countries and of different ages, in order for the possible influences of educational system, socioeconomic status, and spoken dialect to be understood.

Future directions

Researchers may consider adapting this task to pilot the assessment of orthographic knowledge at the sublexical level. This may be done by designing a wordlike task based on the orthographic and morphological patterns represented in this measure. Embedding these patterns in pseudowords would assist researchers in exploring the feasibility of assessing orthographic knowledge at the sublexical level in Arabic, similar to the studies by Cassar and Treiman (1997), Pacton et al. (2001), Commissaire et al. (2014), and Deacon et al. (2013). This could be particularly informative for assessing children’s tacit orthographic knowledge prior to formal reading instruction.

Work is also needed to understand how orthographic knowledge fits into the literacy development of Arabic-speaking individuals. There is some evidence that a child’s ability to complete an orthographic choice task may be the result of developing sufficient literacy skills, rather than being a unique ability fundamental to literacy acquisition (Deacon et al., 2012). To assess the role of orthographic knowledge in Arabic literacy development, additional standardized and validated measurement tools are needed. Specifically, scales including items with more open formats and reduced reliance on visual recognition of words (e.g., spelling tasks) may be valuable. With a more comprehensive battery of tools to assess reading and writing development in Arabic, researchers and practitioners may be better equipped to investigate and support the literacy of Arabic-speaking individuals.

Practical implications

Despite the limitations of present study, the tool examined has the potential to serve as a key resource for educators and researchers, particularly because of the current lack of standardized assessments of Arabic literacy development. This measure may provide insight into distinguishing strengths and weaknesses among Arabic-speaking children who are learning to read (Cassar & Treiman, 1997; Ehri, 2017; Pacton et al., 2001; Tibi & Kirby, 2019; Treiman & Kessler, 2006). Children’s performance on this measure may assist educators in baseline assessment of children’s orthographic knowledge and in monitoring progress in word reading over time. Although normative data is not yet available for this tool, educators may use it as a criterion-referenced measure. Because it covers a broad array of orthographic patterns and rules in Arabic, error analysis may provide practitioners with insight into individuals’ strengths and weaknesses (Tucker et al., 2016). As literacy is an important part of the responsibilities of speech–language pathologists (American Speech Language Hearing Association, 2001; Fumero & Tibi, 2020; Wofford & Tibi, 2018), this measure provides an important tool for clinical practitioners.

Of note, the results from this research indicate that the tool can be scored using a simple sum-score system. This is valuable because it can be scored simply by awarding a single point for each correct response and then summing those points upon completion of the task. The reliable nature of the final item set provides preliminary evidence for its use both in research and in educational practice.

A lack of valid assessment tools is detrimental to any researcher. However, developing standardized tools is an iterative process that must begin with some work focused on internal psychometrics and reliability. We consider the work completed in the current study as a step forward in refining and standardizing tools to assess literacy in Arabic. This is a priority for policy makers, researchers, and practitioners, particularly considering that standardized Arabic is the written language used in all the Arab countries.

Conclusion

The purpose of this study was to provide foundational information about the assessment of orthographic knowledge in Arabic. The findings indicate that this orthographic choice task provides reliable information about a single underlying ability for Arabic-speaking children in Grade 3. Further research is needed to evaluate the reliability and validity of this tool across different age groups, as well as its validity in relation to other reading skills such as reading fluency and reading comprehension. The current study is an early step on a long road toward understanding literacy development in Arabic.