Do participation rates vary with participation payments in laboratory experiments?

Abstract

This paper reports a series of experiments designed to evaluate how the advertised participation payment impacts participation rates in laboratory experiments. Our initial goal was to generate variation in the participation rate as a means to control for selection bias when evaluating treatment effects in common laboratory experiments. Initially, we varied the advertised participation payment to 1734 people from $5 to $15 using standard email recruitment procedures, but found no statistical evidence this impacted the participation rate. A second study increased the advertised payment up to $100. Here, we find marginally significant statistical evidence that the advertised participation payment affects the participation rate when payments are large. To combat skepticism of our results, we also conducted a third study in which verbal offers were made. Here, we found no statistically significant increase in participation rates when the participation payment increased from $5 to $10. Finally, we conducted an experiment similar to the first one at a separate university. We found no statistically significant increase in participation rates when the participation payment increased from $7 to $15. The combined results from our four experiments suggest moderate variation in the advertised participation payment from standard levels has little impact on participation rates in typical laboratory experiments. Rather, generating useful variation in participation rates likely requires much larger participation payments and/or larger potential subject pools than are common in laboratory experiments.

1 Introduction

The generalizability of laboratory experiments is potentially limited because participation is voluntary, which could introduce selection bias. In principle, one can use variation in the participation payment to help control for selection bias if more people are willing to participate in an experiment when the advertised participation payment is increased,¹ Harrison et al. (2020, p. 567) do just this in a field experiment and call “for future experimental designs [to] exogenously vary show-up fees and evaluate the effects [of selection bias] on a case-by-case basis." As a practical matter, the impact of varying the participation payment depends on the responsiveness of the potential subjects.

While our original intention was to answer the call of Harrison et al. (2020), we observed little variation in participation rates when advertised participation payments varied between $5 to $15 using otherwise standard procedures for our laboratory. This paper documents our initial experiment and three related follow-up studies: one conducted with participation payments ranging from $5 to $100 with the offered amount in the subject line of the recruitment email; one with verbal offers being made to potential participants; and one conducted at a different laboratory using its standard recruitment procedures. Collectively our findings suggest small increases in the advertised participation payment are unlikely to have a substantial impact on the participation rate. One implication of this finding is that a researcher may need access to a very large number of potential participants and/or be able to offer substantially different participation payments in order to use variation in advertised payments as means to control for selection bias. As a practical matter, such efforts may be better suited to settings where these features are more typical, such as in large scale field experiments, rather than standard laboratory settings.

2 Initial study

2.1 Recruitment procedure

On the Sunday evening prior to a planned experiment, we sent recruitment emails to 1734 potential participants in the subject pool of the University of Alabama's TIDE Lab. The pool is primarily comprised of undergraduate students in the College of Business and includes nearly every upper-level student in the college. In the email, each of the potential participants was told the following information: (1) they were being invited to participate in a study where they would receive a specific amount of money for showing up on time (i.e., the advertised participation payment), and they could earn extra money based on their decision-making during the study; (2) the study would last about 30 min; (3) they could choose any one of the sessions scheduled every half an hour from 9:30 am to 4:00 pm on Thursday or Friday of that week; (4) they could sign up for the study until midnight Wednesday via a link in the email. The recruitment email sent to each potential participant was identical except for the advertised participation payment and used the lab's standard recruitment email template for paid studies.

The range of advertised participation payments varied from $5 (a participation payment commonly used in lab experiments in the U.S.) to $15. Based on our own intuition, we expected participation rates to vary from approximately 10% to approximately 25% as the advertised participation amount increased from $5 to $15. We randomly drew 100 different levels of participation payments (in dollars and cents) from a uniform distribution over this range.² The 1734 potential participants were then randomly assigned to those payment levels. In total, 125 individuals signed up for the study and 105 showed up and participated in the experiment. Thus, the registration and participation rates are 7.21% and 6.06%, respectively, while the participation rate conditional on registration is $84\%$ . It is not clear whether the registration and participation rate are respectively more or less equal to, lower than, or higher than the typical rates in the lab since statistics are not available for the overall registration and participation rate at TIDE Lab. The participation rate conditional on registration was in line with the typical rate at the lab.

2.2 Registration and participation decisions

Here we examine the impact of the advertised participation payment on the registration decision, the participation decision, and the participation decision conditional on registration. Figure 1 presents a scatter plot of the data. Given that our outcome variables are binary in nature (e.g., 1 if the person registers and 0 if not), our regression results are based on Probit and Complementary Log-Log models. For each model, we regress the outcome of interest on the advertised participation payment and rely on bootstrapped standard errors.³ The results can be found in Table 1 for all three outcomes of interest. As shown in columns 1 and 2 for each outcome variable, the estimated coefficient for the advertised participation payment is small and not statistically different from zero. Figure 1 also provides the predicted probability of participation given an advertised participation payment from the Probit regression as well as 95% bootstrapped confidence bounds for the estimated probabilities. To help interpret the Probit regression, we present Fig. 2 which shows the predicted difference in registration rates for different advertised participation payments.⁴ Each curve in this contour plot denotes pairs of advertised amounts for which the difference in the predicted registration rate equals a given amount. For example, the curve just below the 45 $_{}^{\circ }$ line (not plotted) identifies advertised amounts where the predicted change in the registration rate is 0.1 percentage points. Intuitively, points close to the 45 $_{}^{\circ }$ line should be associated with small changes in the prediction registration rate since the two advertised amounts are almost identical. At the other extreme, increasing the advertised amount from $5 to $15 is associated with about a 1.1 percentage point increase in the registration rate, as can be seen in Fig. 2 as (15, 5) lies to the lower right of the (unlabeled) 1.0 percentage point change contour line. Using a bootstrapping procedure we do not find that the predicted change in the registration rate is significantly different from 0 at the 95% confidence level for any pair of advertised amounts between $5 and $15.

Fig. 1 Scatter plot of advertised participation payment versus registration decision, participation decision and participation decision conditional on registration with fitted values from Probit regression and bootstrapped confidence bounds for the initial study

Fig. 2 Contour plot of the predicted change in registration rate between two advertised amounts for the initial study (lower offer vs higher offer)

Table 1 Effect of advertised participation payment on an individual's registration decision, participation decision and participation decision conditional on registration based (999 bootstrapped standard errors are in parentheses below each point estimate) for the initial study

(a) Dependent variable: registration decision
	Probit model	Complementary log–log model
	(1)	(2)
Advertised participation payment	0.009	0.018
	(0.015)	(0.030)
Constant	−1.559	−2.785
	(0.162)	(0.322)
Observations	1734	1734
Log likelihood	−448.932	−448.934
AIC	901.863	901.868
BIC	912.780	912.785

(b) Dependent variable: participation decision
	Probit model	Complementary log–log model
	(1)	(2)
Advertised participation payment	0.009	0.018
	(0.016)	(0.032)
Constant	−1.644	−2.962
	(0.173)	(0.356)
Observations	1734	1734
Log likelihood	−396.048	−396.052
AIC	796.097	796.103
BIC	807.013	807.020

(c) Dependent variable: participation decision (conditional on registration)
	Probit model	Complementary log–log model
	(1)	(2)
Advertised participation payment	−0.001	−0.001
	(0.050)	(0.041)
Constant	1.003	0.613
	(0.547)	(0.452)
Observations	125	125
Log likelihood	−54.959	−54.959
AIC	113.917	113.917
BIC	119.574	119.574

2.3 Email read rate

One limitation of this study is that we do not observe which potential participants actually read the recruitment email. Thus, the 7.21% registration rate we observe is a lower bound. To evaluate how the possibility of unread emails affects the impact of the advertised participation payment on the registration decision, we reconduct the above analysis after simulating all possible email read rates. Specifically, for each value of K from 1 (the minimum positive number of emails that may not have been read) to 1608 (the maximum number of emails that may not have been read such that registration rate is less than 100%), we went though the following process 1000 times: (1) randomly select K observations from the set of 1609 non-registrants to drop from the data set; (2) estimate the Probit model for the registration rate as a function of the advertised amount using the 1734-K retained observations. We only randomly selected non-registrants to drop because the 125 people who registered had to have read the email and the decision to read the email could not be conditioned on the advertised participation payment as that was contained in the body of the message. The upper panel of Fig. 3 shows the average of the coefficients relating registration rate to advertised payment for the 1000 Probit regressions for each value of K. The bottom panel of the figure shows, for each value of K, the average of the 1000 p-values associated with the one-sided alternative hypothesis that the estimated coefficient in a Probit regression is greater than zero. What is clear from the figure is that the results do not depend on whether only 125 people who registered read the recruitment email or if all 1734 people we contacted read the recruitment email—the advertised participation payment had little affect on participation.⁵

Fig. 3 Average estimated coefficient in Probit regression and average p-value of the coefficient from 1000 simulations for the given number of emails not read in the initial study

2.4 Impact of participation payment on treatment effects

We expected participation rates to vary from, say 10% to 25%, as the advertised amount increased from $5 to $15. As previously indicated, our intention was to use this variation as an exclusion restriction in a two-step Heckman (1979) selection correction approach where the first stage estimates the impact of the payment amount on registration and the second stage uses that analysis in estimating the treatment effect. Specifically, we planned to investigate how the stakes impacted risk taking, how a match impacted charitable giving, and how cognitive load impacted reasoning in a beauty contest. Because the advertised show-up payment did not significantly impact the participation decision, there is no variation in participation rates to use as a control for selection bias when measuring treatment effects from the in-lab experiment conducted on the 105 people who came to the lab. As such, we relegate the details of the in-lab experiment and analysis of the treatment effects to the Appendix.

3 Large show-up payments

We were surprised that offering $15 did not lead to substantially higher participation rates than offering $5. To determine what advertised amount would lead to a sizable increase in participation, we conducted an exploratory second study using a larger range of participation payments: $5 to $100.⁶ We invited 96 randomly selected individuals from TIDE Lab's subject pool excluding those who participated in the initial study. Each person was offered a unique integer dollar show-up payment, which was listed in the subject line of the individualized recruitment email (as opposed to the initial study that only revealed the payment amount in the body of the email as is standard in TIDE Lab).

The overall registration rate of 7.29% in this additional study was nearly identical to the 7.21% observed in the initial study. The participation rates were similar as well (7.29% vs 6.06% in the initial study).⁷ With our 96 observations, we conduct both Probit and Complementary Log–Log regressions to test whether increasing the participation payment increases the likelihood potential participants register for the lab experiment. The results, presented in Table 2, indicate the coefficient (in the Probit model) for the participation payment is relatively small and insignificant based upon the bootstrapped standard errors. If we use the asymptotic standard errors (0.012), the coefficient is marginally significant (p-value = 0.088). Figure 4 presents a scatter plot of the data along with the predicted probability of participation given an advertised participation payment based on the Probit regression as well as 95% bootstrapped confidence bounds for the estimated probabilities.

Table 2 Effect of advertised participation payment on an individual's registration/participation decision (999 bootstrapped standard errors are in parentheses below each point estimate) for the large show-up payment study (all those who registered participated)

Dependent variable: registration/participation decision
	Probit model	Complementary log–log model
	(1)	(2)
Advertised participation payment	0.013	0.024
	(0.019)	(0.021)
Constant	−2.250	−4.023
	(1.791)	(2.017)
Observations	96	96
Log Likelihood	−23.563	−23.670
AIC	51.126	51.340
BIC	56.255	56.524

Fig. 4 Scatter plot of advertised participation payment versus registration decision with fitted values from Probit regression and bootstrapped confidence bounds for the study with large show-up payments

To help interpret the Probit regression results, we present Fig. 5 which shows the predicted difference in registration rates for different advertised participation payments, similar to Fig. 2 for our initial experiment. Using our bootstrap procedure, we do not find that any of these changes are significantly different from 0 at the 95% confidence level using a one-tailed test, although there is marginally significant evidence at the 90% confidence level for large participation payment values. However, one should be cautious in drawing conclusions given the relatively small number of observations.

Fig. 5 Contour plot of the predicted change in registration rate between two advertised amounts for the study with large show-up payments (lower offer vs higher offer)

Two aspects of Fig. 5 are worth highlighting. First, the estimated increase in the participation rate when going from $5 to $15 is 0.5 percentage points, which is very similar to the predicted increase in the first study for that same change. Second, the predicted increase in the participation rate when going from $60 to $100 is about 10.5 percentage points. For comparison, in their Danish field experiment Harrison et al. (2020) report an increase in the participation rates of about 6 percentage points with approximately equivalent dollar amounts. Thus, our predicted increase in participation over this range of advertised payments is nominally (although not statistically) larger than what they observed. We also note that our registration rates for these amounts are lower than those reported by Harrison et al. (2020). At $60 our predicted participation rate is 7.3% while their registration rate was 18.1% for a comparable amount. At $100 our predicted registration rate is 17.9% while theirs was 24.1% for a comparable amount.

During the review process, we were met with some skepticism regarding certain aspects of our first two experiments. We therefore ran two additional experiments to determine if our results were a fluke. The first uses verbal participation offers and the second was conducted at another university. The next two sections briefly describe each experiment.

4 Verbal offer of show-up payments

We conducted another exploratory study in which people were verbally offered either $5 or $10 to participate in an experiment as they left a separate unrelated experiment conducted by another researcher. The goal of this study was to ensure everyone who was solicited to participate knew how much money was being offered. The verbal offer is conducted as follows: when collecting their payment for the unrelated study, in private, each person was informed that “There is another study starting now that will last approximately 15 min. You will be paid $[5 or 10] plus what you earn in the study. Would you like to participate?” The amount offered to each person was randomly predetermined. In total, we approached 62 people. Thirty of 32 people agreed to stay for $5 (93.75%) and 29 of 30 agreed to stay for $10 (96.67%). These rates do not differ statistically (p-value = 0.593); however, there is a ceiling effect given the high participation rate for the lower amount.⁸ $_{}^{,}$ ⁹ Those who stayed completed a real effort task as in Azar (2019) with a piece rate compensation scheme. Subjects could complete up to 20 tasks, with the per task compensation decreasing from $1.50 for the first task to $-\$1.00$ for the 20th task. The 16th task paid $0.05 and the ${17}^{\mathrm{th}}$ task paid $0.00. The modal response was to complete 16 tasks and 36 people stopped exerting effort at piece rates between $0.25 and $0.00, inclusive, despite the average earnings in the prior experiment being $30.39.¹⁰

5 Recruitment at another university

One may be concerned the results of our initial experiment are due to something peculiar with TIDE Lab's participant pool or recruitment process. After all, we ourselves originally anticipated the registration rate would increase from something like 10% to 25%. Therefore, we conducted an additional study at the Economic Science Institute at Chapman University using their standard recruitment procedures.¹¹ To be clear, this new study is not designed to be a formal replication of the initial study. To have 80% power when testing an alternative hypothesis that the registration rate would increase from 6% to 7% at the 95% confidence level would require over 15,000 subjects. The study at Chapman University is meant to determine if the initial study was a fluke and that increasing the participation payment would increase the participation rate along the lines of our original priors.¹²

An additional benefit of conducting the study at the Economic Science Institute is that approximately 600 people in their subject pool had previously completed a survey including demographic information such as sex assigned at birth, class standing, GPA, and major as well as assessments of personal characteristics including CRT score, competitiveness, risk tolerance, time preferences, and belief in others' trustworthiness. From the subset of people who had completed the survey, 200 people were randomly selected and evenly split into two groups. Both groups received a standard recruitment email from the lab inviting them to participate in a thirty-minute session. The only difference in the recruitment messages was that one group was offered a show-up payment of $7 (the standard amount for the lab) and the other group’ was offered a payment of $15. As in the first two studies, all recruitment emails were sent out at the same time and registration closed prior to the start of the first session. Table 3 compares the characteristics of the two groups receiving recruitment emails and shows that the groups are balanced.

Table 3 Comparison between the two groups with different advertised participation payments in terms of characteristics

	Samples		Test for equality
	Recruited at $7 (1)	Recruited at $15 (2)	p-values (3)
Sex assigned at birth (Female = 1, Male =0)	64.0% (0.48)	69.0% (0.46)	0.454
Major (Econ, ACCT, BA = 1, Else = 0)	38.0% (0.49)	38.0% (0.49)	1.000
GPA (Scale = 0 to 4)	3.64 (0.32)	3.68 (0.29)	0.389
Class Standing (fr. = 1, soph. = 2, jr. = 3, sr. =4)	1.91 (1.18)	2.11 (1.15)	0.227
CRT Score (Scale = 0 to 7)	3.38 (2.11)	3.30 (2.31)	0.799
Competitiveness (Scale = 0 to 10)	7.08 (2.00)	7.14 (1.85)	0.826
Willingness to take risks (Scale = 0 to 10)	6.08 (1.99)	5.99 (1.78)	0.736
Prefer enjoying oneself today (Scale = 1 to 5)	2.90 (0.97)	3.09 (0.98)	0.169
Belief in others' trustworthiness (Scale = 1 to 5)	2.56 (1.03)	2.76 (1.01)	0.166
Observations	100	100

Standard deviations are reported in parentheses. To test for equality in characteristics between the two groups recruited with different advertised participation payments, we use proportion tests for the first two variables (sex assigned at birth and major) and t-tests for the other variables. Three freshman students did not provide GPA information and are dropped from the GPA calculations

Twelve of the 100 subjects who were offered $7 registered while 16 of the 100 subjects offered $15 registered. The difference in registration rates between these two groups is not statistically significant (12% vs. 16%, p-value = 0.415).¹³ This insignificant effect aligns with our initial experiment, although there is nominally more separation and the overall registration rate is about twice as high.

Table 4 compares the characteristics of the people who registered with those who did not for both advertised amounts. Characteristics of those who registered and characteristics of those who did not register are similar for both advertised show-up payments.¹⁴ More importantly, the characteristics of those who registered for $7 and those who registered for $15 do not differ statistically. This suggests that small variation in show-up payments may not affect the composition of participants.

Table 4 Comparison of characteristics between those who registered and those who did not for both advertised participation amounts

	Recruited at $7			Recruited at $15			p-values of Tests for Equality
	Registered	Not Registered	p-values of Test for Equality	Registered	Not Registered	p-values of Test for Equality	Registered at $7 vs. $15	Not Registered at $7 vs. $15
	(1)	(2)	(3)	(4)	(5)	(6)	(7)	(8)
Sex assigned at birth (Female = 1, Male =0)	50.0% (0.50)	65.9% (0.48)	0.282	56.3% (0.50)	71.4% (0.45)	0.229	0.743	0.436
Major (Econ, ACCT, BA = 1, Else = 0)	41.7% (0.49)	37.5% (0.48)	0.780	37.5% (0.48)	38.1% (0.49)	0.964	0.823	0.936
GPA (Scale = 0 to 4)	3.52 (0.33)	3.66 (0.31)	0.155	3.57 (0.36)	3.70 (0.27)	0.111	0.710	0.374
Class Standing (fr. = 1, soph. = 2, jr. = 3, sr. =4)	2.50 (1.45)	1.83 (1.13)	0.065	2.13 (1.31)	2.11 (1.13)	0.955	0.480	0.109
CRT Score (Scale = 0 to 7)	3.58 (2.11)	3.35 (2.12)	0.724	3.06 (2.26)	3.35 (2.33)	0.656	0.541	0.984
Competitiveness (Scale = 0 to 10)	7.50 (2.71)	7.02 (1.89)	0.441	6.81 (1.91)	7.20 (1.84)	0.442	0.437	0.529
Willingness to take risks (Scale = 0 to 10)	6.33 (2.61)	6.05 (1.91)	0.640	6.31 (1.70)	5.93 (1.80)	0.431	0.980	0.680
Prefer enjoying oneself today (Scale = 1 to 5)	2.75 (1.06)	2.92 (0.96)	0.570	3.44 (0.89)	3.02 (0.98)	0.121	0.073	0.486
Belief in others' trustworthiness (Scale = 1 to 5)	2.50 (1.31)	2.57 (0.99)	0.831	3.25 (1.00)	2.67 (0.99)	0.033	0.098	0.515
Observations	12	88		16	84

Standard deviations are reported in parentheses. To test for equality in characteristics of the people who registered and who did not for both advertised amounts, we use proportion tests for the first two variables (sex assigned at birth and major). For other variables, we use t-tests to test for equality. Three freshman students did not provide GPA information and are dropped from the GPA calculations (one did not register at $7, one registered at $15 and one did not register at $15)

6 Discussion

In a series of experiments, we find little evidence to suggest increases in the advertised participation payment will lead to substantial increases in participation rates in laboratory experiments, at least when using typical monetary amounts and standard recruitment procedures. In our initial experiment, we sent emails to 1734 potential participants with advertised amounts varying from $5 to $15 (embedded in the text of the email per standard procedure at our lab) and the average registration rate was about 7%. The estimated increase in the registration rate for the $10 increase was about 1 percentage point and we show the insignificance of the coefficient does not depend upon the email read rate. In another experiment, we varied the advertised amount from $5 to $100 and included this amount in the subject line of the email. This is the only experiment where we find marginally significant evidence that a higher advertised amount leads to greater participation, but the overall participation rate was only 7% and marginal significance was only observed when large dollar amounts were involved. Analysis of this experiment indicates raising the $5 to $15 leads to approximately a 1 percentage point increase in participation, as in the first experiment. Increasing the advertised amount from $60 to $100 is estimated to lead to an 11 percentage point increase in registration rates, which is marginally significant in a one-sided test and is a larger effect size than was observed by Harrison et al. (2020) in a field experiment using similar advertised amounts. In a separate experiment conducted at a different laboratory with 200 subjects, we found increasing the participation rate from $7 to $15 nominally, but not significantly, increased registration rates from 12% to 16%.

Our initial experiment was designed to answer the call by Harrison et al. (2020) to vary advertised participation payments in order to induce variation in participation rates so that the impact of selection bias could be identified in experiments. Ultimately, we did not observe variation in the participation rate in our initial experiment and thus could not use the advertised payment amount as an exclusion restriction to control for selection bias in a two-stage Heckman procedure. Further, the experiment we conducted at another university enabled us to identify characteristics of the people being recruited and we did not find evidence that the advertised amount impacted who was willing to participate and who was not. Overall, our findings suggest using relatively small variation in the advertised participation payment to control for selection bias may require access to very large pools of potential participants. However, we do find marginally significant evidence that advertising different large monetary amounts may lead to variation in participation rates, akin to what has been observed in the field. Unfortunately, offering $60 or $100 as a participation payment may not be practical for many laboratory studies given the substantial increase in research cost.

While we did not observe sizable variation in participation rates, our results provide useful insights for experimental economists. First, researchers with limited budgets conducting lab experiments are likely better served keeping participation payments low and either raising the salient payoffs associated with the study, collecting data from more subjects even if that requires soliciting more potential participants, or using the funds to run additional treatments or other experiments all together. Second, our results suggest that people who are willing to participate in lab experiments are motivated by relatively small amounts of money. This is further supported by our experiment where people who had just completed one study were offered money to stay and participate in another study. Regardless of whether they were verbally offered $5 or $10 nearly everyone stayed and most were willing to complete real effort tasks for as little as $0.25 despite having already earned $30.39 on average in the prior experiment. This suggests the stakes in a typical laboratory experiment are sufficiently high to motivate the participants, consistent with induced value theory and counter to often heard criticisms against laboratory experiments. We also note that the fact varying participation payments did not have the intended effect does not preclude the existence of successful methods for inducing variation in lab experiment participation. We see exploring such alternative avenues as an important direction for future research.