Different speakers sometimes convey similar meanings differently. This study examined whether listeners could learn to associate a specific linguistic ‘style’ with a certain speaker, with no apparent difference in meaning, and the role of unnatural linguistic choices (or unexpectedness) in such learning. We created an inter-speaker variation in ‘style’ using the weak adjective ordering preferences in Hebrew. Participants were exposed to two different speakers, each producing a different adjective order, consistently. We manipulated the combinations of order pairings, based on their naturalness (with two natural orders, a natural and an unnatural order, and two unnatural orders), and examined participants’ ability to associate a unique order with a specific speaker. In two experiments, using different statistical analyses, we show that listeners can learn speaker-specific language use when it is irrelevant for meaning inferences, when deviance from natural or expected language use is involved. We further discuss whether learning may be facilitated by differences in naturalness or structural form. Our findings suggest that listeners are sensitive to inter-speaker variability in ‘style’, mostly when this ‘style’ is unexpected. This is in line with the predictions of Surprisal theory, and may suggest that surprisal plays a major role in learning speaker-specific language use.