Although the correlation between cognition and physical function has been well studied in the general population, the genetic and environmental nature of the correlation has been rarely investigated. We conducted a classical twin analysis on cognitive and physical function, including forced expiratory volume in one second (FEV1), forced vital capacity (FVC), handgrip strength, five-times-sit-to-stand test (FTSST), near visual acuity, and number of teeth lost in 379 complete twin pairs. Bivariate twin models were fitted to estimate the genetic and environmental correlation between physical and cognitive function. Bivariate analysis showed mildly positively genetic correlations between cognition and FEV1, rG = 0.23 [95% CI: 0.03, 0.62], as well as FVC, rG = 0.35 [95% CI: 0.06, 1.00]. We found that FTSST and cognition presented very high common environmental correlation, rC = -1.00 [95% CI: -1.00, -0.57], and low but significant unique environmental correlation, rE = -0.11 [95% CI: -0.22, -0.01], all in the negative direction. Meanwhile, near visual acuity and cognition also showed unique environmental correlation, rE = 0.16 [95% CI: 0.03, 0.27]. We found no significantly genetic correlation for cognition with handgrip strength, FTSST, near visual acuity, and number of teeth lost. Cognitive function was genetically related to pulmonary function. The FTSST and cognition shared almost the same common environmental factors but only part of the unique environmental factors, both with negative correlation. In contrast, near visual acuity and cognition may positively share part of the unique environmental factors.