In this article, we highlight how simulation methods can be used to analyze power of economic experiments. We provide the powerBBK package programmed for experimental economists, that can be used to perform simulations in STATA. Power can be simulated using a single command line for various statistical tests (nonparametric and parametric), estimation methods (linear, binary, and censored regression models), treatment variables (binary, continuous, time-invariant or time varying), sample sizes, experimental periods, and other design features (within or between-subjects design). The package can be used to predict minimum sample sizes required to reach a user-specific level of power, to maximize power of a design given the researcher supplied a budget constraint, or to compute power to detect a user-specified treatment order effect in within-subjects designs. The package can also be used to compute the probability of sign errors—the probability of rejecting the null hypothesis in the wrong direction as well as the share of rejections pointing in the wrong direction. The powerBBK package is provided as an .ado file along with a help file, both of which can be downloaded here (http://www.bbktools.org).

Behavioral and psychological researchers have shown strong interests in investigating contextual effects (i.e., the influences of combinations of individual- and group-level predictors on individual-level outcomes). The present research provides generalized formulas for determining the sample size needed in investigating contextual effects according to the desired level of statistical power as well as width of confidence interval. These formulas are derived within a three-level random intercept model that includes one predictor/contextual variable at each level to simultaneously cover various kinds of contextual effects that researchers can show interest. The relative influences of indices included in the formulas on the standard errors of contextual effects estimates are investigated with the aim of further simplifying sample size determination procedures. In addition, simulation studies are performed to investigate finite sample behavior of calculated statistical power, showing that estimated sample sizes based on derived formulas can be both positively and negatively biased due to complex effects of unreliability of contextual variables, multicollinearity, and violation of assumption regarding the known variances. Thus, it is advisable to compare estimated sample sizes under various specifications of indices and to evaluate its potential bias, as illustrated in the example.

In this rejoinder, we examine some of the issues Peter Bentler, Eunseong Cho, and Jules Ellis raise. We suggest a methodological solid way to construct a test indicating that the importance of the particular reliability method used is minor, and we discuss future topics in reliability research.

Biodiversity monitoring programmes should be designed with sufficient statistical power to detect population change. Here we evaluated the statistical power of monitoring to detect declines in the occupancy of forest birds on Christmas Island, Australia. We fitted zero-inflated binomial models to 3 years of repeat detection data (2011, 2013 and 2015) to estimate single-visit detection probabilities for four species of concern: the Christmas Island imperial pigeon Ducula whartoni, Christmas Island white-eye Zosterops natalis, Christmas Island thrush Turdus poliocephalus erythropleurus and Christmas Island emerald dove Chalcophaps indica natalis. We combined detection probabilities with maps of occupancy to simulate data collected over the next 10 years for alternative monitoring designs and for different declines in occupancy (10–50%). Specifically, we explored how the number of sites (60, 128, 300, 500), the interval between surveys (1–5 years), the number of repeat visits (2–4 visits) and the location of sites influenced power. Power was high (> 80%) for the imperial pigeon, white-eye and thrush for most scenarios, except for when only 60 sites were surveyed or a 10% decline in occupancy was simulated over 10 years. For the emerald dove, which is the rarest of the four species and has a patchy distribution, power was low in almost all scenarios tested. Prioritizing monitoring towards core habitat for this species only slightly improved power to detect declines. Our study demonstrates how data collected during the early stages of monitoring can be analysed in simulation tools to fine-tune future survey design decisions.

The current study explored the impact of genetic relatedness differences (ΔH) and sample size on the performance of nonclassical ACE models, with a focus on same-sex and opposite-sex twin groups. The ACE model is a statistical model that posits that additive genetic factors (A), common environmental factors (C), and specific (or nonshared) environmental factors plus measurement error (E) account for individual differences in a phenotype. By extending Visscher’s (2004) least squares paradigm and conducting simulations, we illustrated how genetic relatedness of same-sex twins (HSS) influences the statistical power of additive genetic estimates (A), AIC-based model performance, and the frequency of negative estimates. We found that larger HSS and increased sample sizes were positively associated with increased power to detect additive genetic components and improved model performance, and reduction of negative estimates. We also found that the common solution of fixing the common environment correlation for sex-limited effects to .95 caused slightly worse model performance under most circumstances. Further, negative estimates were shown to be possible and were not always indicative of a failed model, but rather, they sometimes pointed to low power or model misspecification. Researchers using kin pairs with ΔH less than .5 should carefully consider performance implications and conduct comprehensive power analyses. Our findings provide valuable insights and practical guidelines for those working with nontwin kin pairs or situations where zygosity is unavailable, as well as areas for future research.

Despite the particular relevance of statistical power to animal welfare studies, we noticed an apparent lack of sufficient information reported in papers published in Animal Welfare to facilitate post hoc calculation of statistical power for use in meta-analyses. We therefore conducted a survey of all papers published in Animal Welfare in 2009 to assess compliance with relevant instructions to authors, the level of statistical detail reported and the interpretation of results regarded as statistically non-significant. In general, we found good levels of compliance with the instructions to authors except in relation to the level of detail reported for the results of each test. Although not requested in the instructions to authors, exact P-values were reported in just over half of the tests but effect size was not explicitly reported for any test, there was no reporting of a priori statistical analyses to determine sample size and there was no formal assessment of non-significant results in relation to type II errors. As a first stage to addressing this we recommend more reporting of a priori power analyses, more comprehensive reporting of the results of statistical analysis and the explicit consideration of possible statistical power issues when interpreting P-values. We also advocate the calculation of effect sizes and their confidence intervals and a greater emphasis on the interpretation of the biological significance of results rather than just their statistical significance. This will enhance the efforts that are currently being made to comply with the 3Rs, particularly the principle of reduction.

Dr Nick Martin has made enormous contributions to the field of behavior genetics over the past 50 years. Of his many seminal papers that have had a profound impact, we focus on his early work on the power of twin studies. He was among the first to recognize the importance of sample size calculation before conducting a study to ensure sufficient power to detect the effects of interest. The elegant approach he developed, based on the noncentral chi-squared distribution, has been adopted by subsequent researchers for other genetic study designs, and today remains a standard tool for power calculations in structural equation modeling and other areas of statistical analysis. The present brief article discusses the main aspects of his seminal paper, and how it led to subsequent developments, by him and others, as the field of behavior genetics evolved into the present era.

This article provides an accessible tutorial with concrete guidance for how to start improving research methods and practices in your lab. Following recent calls to improve research methods and practices within and beyond the borders of psychological science, resources have proliferated across book chapters, journal articles, and online media. Many researchers are interested in learning more about cutting-edge methods and practices but are unsure where to begin. In this tutorial, we describe specific tools that help researchers calibrate their confidence in a given set of findings. In Part I, we describe strategies for assessing the likely statistical power of a study, including when and how to conduct different types of power calculations, how to estimate effect sizes, and how to think about power for detecting interactions. In Part II, we provide strategies for assessing the likely type I error rate of a study, including distinguishing clearly between data-independent (“confirmatory”) and data-dependent (“exploratory”) analyses and thinking carefully about different forms and functions of preregistration.

Previous European guidance for environmental risk assessment of genetically modified plants emphasized the concepts of statistical power but provided no explicit requirements for the provision of statistical power analyses. Similarly, whilst the need for good experimental designs was stressed, no minimum guidelines were set for replication or sample sizes. Furthermore, although substantial equivalence was stressed as central to risk assessment, no means of quantification of this concept was given. This paper suggests several ways in which existing guidance might be revised to address these problems. One approach explored is the `bioequivalence' test, which has the advantage that the error of most concern to the consumer may be set relatively easily. Also, since the burden of proof is placed on the experimenter, the test promotes high-quality, well-replicated experiments with sufficient statistical power. Other recommendations cover the specification of effect sizes, the choice of appropriate comparators, the use of positive controls, meta-analyses, multivariate analysis and diversity indices. Specific guidance is suggested for experimental designs of field trials and their statistical analyses. A checklist for experimental design is proposed to accompany all environmental risk assessments.

The aim of the present study was to quantify the inter-day reliability of force platform data from horses at the trot. The subjects were ten horses judged to be sound on the basis of clinical evaluation and similarity of ground reaction force peaks of left and right limbs. The same handler trotted horses in hand for four consecutive days over a force platform at their natural and comfortable velocity between 2.70 and 3.60 m s−1. For each horse, velocity was controlled in a range of±0.15 m s−1. Five contacts per limb were recorded on each day. Custom-written code in MATLAB was used to smooth the data and time-normalize to 101 data points. The agreement boundary was used to determine the reliability of seven force variables. Stance duration, peak vertical force and vertical impulse were the most reliable variables (<10% variability in the mean), while peak braking and propulsive forces and impulses had larger variation across the three days (>20% variability in the mean). Variables with low variability may require fewer animals, trials and days to obtain accurate data while maintaining adequate statistical power. These results may act both as a reference when studying variability in lame horses and as a guide for planning future equine kinetic studies.