Referential processing relies on similar cognitive functions across languages – in particular, working memory. However, this has only been investigated in spoken languages with highly similar referential systems. In contrast to spoken languages, American Sign Language (ASL) uses a spatial referential system. It is unknown whether the referential system of ASL (L1) impacts referential processing in English (L2). This cross-language impact may be of particular importance for deaf, bimodal bilinguals who sign in ASL and read in English. Self-paced reading times of pronouns in English texts were compared between ASL–English bimodal bilinguals and Chinese–English unimodal bilinguals. The results showed that L1 referential characteristics influenced pronoun reading time in L2. Furthermore, in contrast to Chinese–English bilinguals, ASL–English bilinguals’ referential processing during reading of English texts relied on vocabulary knowledge – not working memory. These findings emphasize the need to expand current theories of referential processing to include more diverse types of language transfer.