Analyses of skeletal phenotypic traits have the potential to yield information pertaining to social organization, such as kinship and residence patterns. In this research, we examine sex-specific phenotypic variability of craniofacial and mandibular metric traits in eight skeletal samples from four sites (San Gerónimo, Chiribaya Alta, Chiribaya Baja, and El Yaral) attributed to the Chiribaya polity (A.D. 772–1350) and a contemporary Ilo-Tumilaca/Cabuza group (El Algodonal) from southern Peru. Through this data set we investigate various aspects of social organization (e.g., postmarital residence patterns) within the Chiribaya polity. We also examine Chiribaya interactions with the contemporaneous, yet culturally distinct, Tumilaca cultural group. The pattern of between-group bias-corrected estimates of biological distances points toward a more distant biological relationship between the two cultures than has previously been reported. Among the Chiribaya groups considered, determinant ratio analyses indicate that males from two cemeteries (Chiribaya Alta Cemetery 4 and Chiribaya Alta Cemetery 7) were significantly more variable than females, suggesting that males were the more mobile sex. Several explanatory models are considered, including a matrilocal residence pattern and the in-migration of males. We also examine the unique nature of results from within Chiribaya Alta, which support the argument made by previous researchers that this site represents a regional political center.