How does the general public perceive immigrants, whom do they think of when thinking about “immigrants,” and to what extent are these perceptions related to the actual composition of immigrant populations? We use three representative online surveys in the United States, South Africa, and Switzerland (total N = 2,778) to extend existing work on the perception of immigrants in terms of geographic coverage and individual characteristics. We also relate these responses to official statistics on immigration and integration patterns. In all three countries, there are significant discrepancies between perceptions of immigrants and their real proportion in the population and characteristics. Although we observe clear country differences, there is a striking pattern of people associating “immigrants” with “asylum seekers” in all three countries. We consider two possible explanations for the differences between perceptions and facts: the representativeness heuristic and the affect heuristic. In contrast to previous research, we find only partial support for the representativeness heuristic, whereas the results are consistent with the affect heuristic. We conclude that images of “immigrants” are largely shaped by pre-existing attitudes.