Relic archeological settlement is used to indicate the development of agriculture. We extracted 8865 relic archeological settlements from the Atlas of Chinese Cultural Relics to analyze how the spatiotemporal distribution of archaeological settlements was influenced by temperature changes and social factors during the last millennium. During the Liao dynasty (AD 916–1125) and Jin dynasty (AD 1115–1234) in the Medieval Warm Period (MWP), a large number of settlements indicated the development of agriculture as far north as 47°N. The warm climate of the MWP provided sufficient heat resources to promote the implication of positive policies of the Liao and Jin dynasties to develop agriculture and settlements. By contrast, during the dynasties of Yuan (AD 1279–1368), Ming (AD 1368–1644), and Qing (AD 1644–1911) in the Little Ice Age (LIA), the number of settlements declined drastically, and the northern boundary of the settlement distribution retreated by 3–4 degrees of latitude to modern Liaoning Province. Although the southward retreat of the settlements and related agriculture occurred in the cold climate of the LIA, it could not be completely explained by the drop in temperature. Social factors including nomadic customs, ethnic policies, and postal road systems played more important roles to the northern boundaries of the settlement distributions during the Yuan, Ming, and Qing dynasties.