Multi-Trait Analysis of GWAS and Biological Insights Into Cognition: A Response to Hill (2018)

Abstract

Hill (Twin Research and Human Genetics, Vol. 21, 2018, 84–88) presented a critique of our recently published paper in Cell Reports entitled ‘Large-Scale Cognitive GWAS Meta-Analysis Reveals Tissue-Specific Neural Expression and Potential Nootropic Drug Targets’ (Lam et al., Cell Reports, Vol. 21, 2017, 2597–2613). Specifically, Hill offered several interrelated comments suggesting potential problems with our use of a new analytic method called Multi-Trait Analysis of GWAS (MTAG) (Turley et al., Nature Genetics, Vol. 50, 2018, 229–237). In this brief article, we respond to each of these concerns. Using empirical data, we conclude that our MTAG results do not suffer from ‘inflation in the FDR [false discovery rate]’, as suggested by Hill (Twin Research and Human Genetics, Vol. 21, 2018, 84–88), and are not ‘more relevant to the genetic contributions to education than they are to the genetic contributions to intelligence’.

Hill (2018) presents a critique of our recently published paper in Cell Reports entitled ‘Large-Scale Cognitive GWAS Meta-Analysis Reveals Tissue-Specific Neural Expression and Potential Nootropic Drug Targets’ (Lam et al., 2017). Specifically, Hill offers several interrelated comments suggesting potential problems with our use of a new analytic method called Multi-Trait Analysis of GWAS (MTAG) (Turley et al., 2018). Below, we respond to each of these concerns. For context, in our paper (Lam et al., 2017), MTAG was applied to two sets of genomewide association study (GWAS) results: One for cognitive ability (GWAS_COG), and the other for educational attainment (GWAS_EDU).

First, Hill (2018) suggests that our GWAS_COG and GWAS_EDU datasets ‘differ a great deal in power’ and that MTAG is not appropriate when the traits examined are ‘highly dissimilar’ with respect to power. As noted by the creators of MTAG (Turley et al., 2018), power for a GWAS can be quantified by the non-centrality parameter (NCP) of the mean χ² statistic, which scales with sample size and heritability. In our study, mean χ² was 1.245 for GWAS_COG and 1.638 for GWAS_EDU, and the ratio of the NCPs is equal to 0.384. Are these ‘highly dissimilar’ by the standards of MTAG?

The term ‘highly dissimilar’ is a subjective appraisal, while we seek to address the question quantitatively. Fortunately, the developers of MTAG provided simulations across a range of scenarios in order to establish benchmarks for when MTAG might be problematic; see Turley et al. (2018, Supplementary Figure 1.2.a). The scenarios tested included: (a) both GWAS have equally low power (mean χ² = 1.1 for each; NCP ratio = 1); (b) one low-powered GWAS (mean χ² = 1.1) and one moderate-powered GWAS (mean χ² = 1.4), resulting in an NCP ratio of 0.25; (c) one low-powered GWAS (mean χ² = 1.1) and one high-powered GWAS (mean χ² = 2.0), with NCP ratio = 0.1. Of these, only the last of these scenarios resulted in problematic inflation of the false discovery rate (FDR). By contrast, our NCP ratio was most comparable to (and somewhat more favorable than) scenario (b) above, which showed virtually no increase in FDR relative to the scenario in which the two GWAS were equally powered (scenario a).

Most importantly, Hill (2018) raises the specific concern that the difference in power between GWAS_COG and GWAS_EDU would lead to an inflated FDR, and notes that our paper (Lam et al., 2017) did not report the ‘max FDR’ calculation as made possible by the MTAG software. Here, we can report that the max FDR calculation for the MTAG results reported in Lam et al. (2017) was 0.0068. This is nearly an order of magnitude below the commonly accepted 0.05 standard for false discovery. Moreover, our max-FDR value is comparable to those reported by Turley et al. (2018) for their empirical MTAG study of depression, neuroticism, and subjective wellbeing. Placed in context of the specific results of Lam et al. (2017), in which 70 independent loci (with 82 independent lead SNPs) were identified by MTAG, a FDR of 0.0068 suggests that between 0 and 1 of these loci is false.

Hill (2018) points to the pattern of genetic correlations reported in our manuscript (Lam et al., 2017, Supplementary Table 14) as evidence that the polygenic signal derived from our MTAG ‘is indistinguishable from that of education’. This characterization is inaccurate for several reasons. First, and most simply, our Supplementary Table 14 demonstrates the set of genetic correlations for the MTAG data differ from those of educational attainment (from the Okbay et al., 2016 dataset) in many cases by an absolute value (for r _g) of 0.10 or greater. As an important example, the polygenic signal from MTAG demonstrated a strong correlation with childhood IQ (r _g = 0.86), which was virtually identical to that observed between childhood IQ and GWAS_COG (r _g = 0.87); by contrast, the correlation between childhood IQ and GWAS_EDU was notably smaller (r _g = 0.74). Additionally, there are several instances in which the MTAG correlations are not only more similar to those for GWAS_COG than for GWAS_EDU, but the absolute value of the MTAG correlations are marginally greater than those observed for either GWAS_COG or GWAS_EDU; for example, for infant head circumference, the genetic correlation (r _g) with MTAG results is 0.2962 (p = 1.88E-07), as compared to its correlation with either GWAS_COG (0.2705; p = 2E-04) or GWAS_EDU (0.2597; p = 2.71E-06).

While not mentioning these counterexamples to his argument, Hill (2018) focuses on the genetic correlations between our MTAG results and other educational variables, which are indeed large (r _g > 0.90). However, Hill (2018) elides the fact that the calculation method employed by LD score regression is known to sometimes produce values for r _g > 1, if the variables are so highly similar as to be self-same (Walters, 2016). Such values (between GWAS_EDU and other virtually identical educational variables) are accurately reported in our Supplementary Table 14, but are truncated to 1 in Hill's partial reproduction of our table (Hill, 2018, Table 1), thereby implying a greater similarity than may be actually present. Importantly, the genetic correlation between the MTAG results and its two constituent GWAS provide evidence contrary to Hill's claim that the ‘Intelligence-MTAG phenotype derived by Lam et al. (2017) is more similar to Education than it is to Cognitive ability’. Specifically, the correlation between MTAG and GWAS_COG (r _g = 0.96, SE = 0.0058) is significantly greater than that between MTAG and GWAS_EDU (r _g = 0.91, SE = 0.0052). Further, Hill (2018) calls attention to the intriguing pattern of results that obtains for schizophrenia and bipolar disorder, which we also noted with interest in our published paper (Lam et al., 2017). Rather than invalidating our approach, we believe that this finding highlights a biologically meaningful set of relationships that we have extensively analyzed in the context of a subsequent manuscript that is currently in preparation. Finally, it is important to note that the overall pattern of genetic correlations is highly similar between all three sets of measures, as is evident in Figure 5 of Lam et al. (2017).

While the foregoing paragraphs address each of the specific points raised by Hill (2018), we wish to note two additional facts reported in our paper (Lam et al., 2017) that weigh against Hill's conclusion. First, our leave-one-out analyses (Lam et al., 2017, Figure 3) demonstrate that prediction of held-out samples, phenotyped for cognitive ability, are better for MTAG than for either GWAS_COG or GWAS_EDU alone. This finding supports our interpretation that MTAG is boosting polygenic signal for cognition, and does not support the conclusion of Hill (2018) that the MTAG polygenic signal is ‘indistinguishable from that of education’. Second, as demonstrated in Figure 2 (and associated text) in Lam et al. (2017), the top results (genome-wide significant loci) emerging from MTAG show notable differences from those emerging from GWAS_EDU, but are almost a complete superset of those emerging from GWAS_COG. Furthermore, based on results we have already seen, we are confident that the novel loci we have identified will receive additional support in forthcoming, larger GWAS studies of cognitive ability that do not incorporate educational attainment data using MTAG.

For all of the empirical reasons cited above, we believe our MTAG results do not suffer from ‘inflation in the FDR [false discovery rate]’, as suggested by Hill (2018), and are not ‘more relevant to the genetic contributions to education than they are to the genetic contributions to intelligence’. We continue to be confident of the evidence reported in Cell Reports (Lam et al., 2017). As suggested above, further efforts underway by the COGENT consortium and other collaborators continue to build additional supporting and clarifying evidence with regard to these issues. Specifically, we have additional large-scale GWAS data, and we continue to further investigate the genetic architecture of cognition and potential downstream implications to neuropsychiatric disease and other health outcomes.