Contrastive pitch accents benefit native English speakers’ memory for discourse by enhancing a representation of a specific relevant contrast item (Fraundorf et al., 2010). This study examines whether and how second language (L2) listeners differ in how contrastive accents affect their encoding and representation of a discourse, as compared to native speakers. Using the same materials as Fraundorf et al. (2010), we found that low and mid proficiency L2 learners showed no memory benefit from contrastive accents. High proficiency L2 learners revealed some sensitivity to contrastive accents, but failed to fully integrate information conveyed by contrastive accents into their discourse representation. The results suggest that L2 listeners’ non-native performance in processing contrastive accents, observed in this and other prior studies, may be attributed at least in part to a difference in the depth of processing of the information conveyed by contrastive accents.