Adding household surveys to the behavioral economics toolbox: insights from the SOEP innovation sample

Abstract

While laboratory and field experiments are the major items in the toolbox of behavioral economists, household panel studies can complement them and expand their research potential. We introduce the German Socio-Economic Panel’s Innovation Sample (SOEP-IS), which offers researchers detailed panel data and the possibility to collect personalized experimental and survey data for free. We discuss what SOEP-IS can offer to behavioral economists and illustrate a set of design ideas with examples. Although we build our discussion on SOEP-IS, our purpose is to provide a guide that can be generalized to other household panel studies as well.

1 Introduction

Laboratory experiments are one of the most important methods in the behavioral economics toolbox as they provide a highly controlled environment for understanding economic behavior. Another advantage lies in using monetary incentives to set up specific controlled economic environments. This allows constructs to be defined parametrically and, in particular, economic preferences to be measured.

Despite their strengths, laboratory experiments have certain limitations. They are usually not based on representative samples and lack panel dimension. There is also limited data on the participants (e.g., health, income, wealth, or family background). Some of these shortcomings are addressed by approaches such as field or online experiments. Large-scale representative surveys that include experimentally validated survey items are also widely used today. The Global Preferences Survey (Falk, 2018) is an important example in this regard. Its preference elicitation module is time and cost-efficient and is widely used to study economic preferences. Furthermore, there have also been recent attempts to re-test the importance of monetary incentives in eliciting certain preferences. For example, Brañas-Garza et al. (2022) suggests that hypothetical time preference elicitation is similar to elicitation using monetary payoffs. In their recent study conducted in Tehran, Sharafi and Kosfeld (2023) show that hypothetical versions of trust, altruism, and reciprocity correlate significantly with individual behavior. In another recent study conducted in Kenya, Bauer et al. (2020) show that quantitative survey measures of altruism, time, and risk preferences are good predictors of choices in incentivized experiments. Such findings may increase the credibility and use of low-cost preference elicitation items in survey research.

In this article, we discuss what household panels can offer behavioral economists. Of course, integrating experiments into panel studies is not a new approach. There are many important studies that have benefited from household survey data (e.g., Falk et al., 2018; Dohmen et al., 2011; Fehr et al., 2003; Bellemare and Kröger, 2007; Bellemare et al., 2008; Von Gaudecker et al., 2011; Dimmock et al., 2016) and many of them use data from the German Socio-Economic Panel (SOEP), one of the longest-running socio-economic panels in the world. However, the opportunities offered by household panels and the procedures for accessing them are not straightforward for researchers with no or limited experience in panel studies. Providing experimental economists with scientific examples and a practical guide can reduce the cost of entry into survey methodology.

Our aim is to introduce household panel studies to behavioral economists, who may be less familiar with the survey methodology, and to introduce them to SOEP-IS as a panel specifically open to behavioral economists.¹ We believe that this introduction is relevant and timely for several reasons. First, behavioral economics has become an important guide for policymakers who demand policy-relevant evidence from general population samples. While laboratory experiments are extremely valuable for testing specific hypotheses, they also have limitations in terms of representativeness and external validity. Examples include behaviors related to aging, specific socio-economic groups, health, and real-life decisions (e.g., insurance demand, occupational choices, tax compliance). Second, the social sciences are experiencing a replication and reproducibility crisis and there is a growing need for replication studies also in economics as well. Publicly available representative household panel studies are ideal for conceptual replication and address the external validity of experimental findings. Finally, during the COVID-19 pandemic, behavioral economics research moved out of the physical laboratory setting. Most likely, software and websites such as O-Tree, Prolific, Qualtrics, and Smartrix were not temporary replacements for laboratories as they continue to be used even though the laboratories are functioning again. In terms of the quality of the subject pool, Prolific is most closely related to SOEP-IS. This platform plays an important role in facilitating access to representative samples and specific groups. However, it does not have an established panel dimension, and there are potentially more selection effects in the subject pool. One of the positive outcomes of moving out of the laboratory is that the research questions of behavioral economists also have begun to move beyond the laboratory. Household panels, especially those which allow researchers to integrate their survey and experimental modules, could further fill this gap.

The paper is organized as follows. First, in Sect. 2, we discuss how behavioral economists can benefit from household panel studies and we provide a selection of studies that use SOEP-IS data with various designs. Second, in Sect. 3, we briefly introduce SOEP-Innovation Sample (IS) to behavioral economists, which offers rich, free-of-charge panel data to the research community and also the possibility to integrate economic experiments and surveys. Section 4 concludes the article.

2 What does SOEP-IS offer to behavioral economists?

In this section, we give insights into behavioral economists on the variety of research possibilities that SOEP-IS and similar household panels offer. For each characteristic, we present examples from SOEP and SOEP-IS-based studies. Note that we neither provide an extensive literature review nor do we discuss the results of these studies; rather we provide examples that illustrate the specific advantages of the use of experiments in a household panel. While our main focus is SOEP-IS, the discussion presented in this section can be generalized to other panel studies that share similar characteristics with SOEP-IS. Table 1 provides a list of similar infrastructures.

Table 1 Frequently used panel studies

	Study	Country
1	The Panel Study of Income Dynamics (PSID)	USA
2	Understanding Society	UK
3	The Swiss Household Panel (FORS)	Switzerland
4	German Socio-Economic Panel	Germany
5	DNB Household Survey (DHS)	Netherlands
6	Korean Labor & Income Panel Study (KLIPS)	South Korea
7	The Panel Data Research Center at Keio University (PDRC)	Japan
8	Russian Longitudinal Household Survey (HSE)	Russia
9	The Household, Income and Labour	Australia
	Dynamics in Australia Survey (HILDA)

2.1 The possibility to integrate new modules and incentivized experiments

SOEP-IS allows the integration of new content developed by users themselves. This does not only include single questions or question modules, but also more complex content such as experiments. SOEP-IS calls for new modules on an annual basis. Selected modules are included in the upcoming waves. While standard questions on socio-economic status, life satisfaction, and risk attitudes, constitute about half of the annual survey, the other half is reserved for innovation modules in the SOEP-IS. The combination of these two sets of variables makes it possible to investigate relationships between socio-economic variables and the module contents. Further, the panel structure and the possibility to re-run certain innovation modules enable researchers to study intertemporal relationships. Because information on respondents is available from previous years, it is also possible to distribute differently framed modules and treatments to various subpopulations.

Examples

Integrating the first economics experiment module in SOEP-IS (2012), Breunig (2021) study investment decisions of nearly 1200 respondents. In this module, respondents decide whether to invest their endowments in a safe or a risky asset where payoffs are tied to Germany’s stock market. Besides investment decisions, the module elicits respondents’ beliefs about the returns on their decisions and about the German stock market’s performance in the following year. It is a good example of how experimental data collected in SOEP-IS can be related to behavior in daily life.

Neyse et al. (2021) investigated the relationship between 2D:4D (a suggested marker for prenatal testosterone exposure) and a set of economic preferences (risk, positive and negative reciprocity, generosity, and trust). The large dataset, based on the innovative module, 2D:4D and economics preferences, provided a strong input to the 2D:4D literature as prior studies had relatively small sample sizes and lacked consistent results. Merging innovative modules with the existing data is also a common practice at SOEP-IS. The study by Lautenbacher and Neyse (2020), for example, used the 2D:4D data and the existing psychology items to show that the 2D:4D ratio is not a relevant predictor of depression.

2.2 Specific research potentials of panel data

Panel data can be used in various ways. It allows investigating the stability of behavior, it allows studying how life course events affect decisions, it provides the possibility to target studies to specific subpopulations, and it allows for causal estimation (e.g., difference in differences, regression discontinuity designs, or fixed effects estimations). Important concepts (e.g., trust) are assessed using different instruments, and over time, panels often use several of these instruments. Therefore, such data can be used to examine the relationship between the instruments and how the instruments relate to external variables. This can address issues of the external validity of laboratory experiments. In addition, the large and random sample makes generalization to the general population more credible and allows for proper investigation of heterogeneous treatment effects.

SOEP-IS allows researchers to track respondents over their life courses and to combine this information with specific complementary data from innovative modules or experiments. Thus, it allows assessments of whether the estimated parameters explain people’s decisions across the life course.

Note that panel attrition is generally low in the SOEP (below 10% per year) and the same is true for SOEP-IS.² The characteristics of dropouts can also be analyzed based on the data gathered in the survey years before the dropout.

Examples

Graeber et al. (2020) use SOEP data to study how differences in regional COVID-19 infection rates alter risk preferences. Using information from the year before the pandemic and during the first months of the pandemic in Germany, they show that higher regional infection rates imply higher reductions in individual risk tolerance. Another type of shock can be new policies affecting behavior and preferences. In a recent study Fedorets and Shupe (2021) investigate the impact of the German minimum wage reform on workers’ reservation wages. Employing a difference-in-differences strategy, the authors compare the minimum acceptable salaries of job-seekers before and after the introduction of the minimum wage reform. They find a short-term increase in reservation wages at the low end of the distribution following the reform. Such studies demonstrate how panel data can be helpful in investigating the implications of shocks for behavioral outcomes and alterations in preferences.

Changes in preferences affecting behavior are not always triggered by exogenous and wide-ranging shocks, but also by individual changes in a person’s life as getting older, marrying, having children, losing a close relative, being laid off, or changing jobs. Dohmen (2017), for example, used SOEP-Core data to investigate the relationship between risk attitudes and age.³ In addition, many life-event studies using SOEP data tackle psychological research questions. For example, Richter et al. (2019), investigate changes in sleep patterns before and after childbirth.

An example of targeting is Fossen et al. (2021) which uses SOEP-IS data to study individuals who have experience with self-employment. As another example, Fossen et al. (2020) study the risk preferences of a socio-economic group that is rarely found in household surveys but that is now covered in the SOEP: millionaires.

An example addressing the external validity is Frey (2021) who use a within-subject design to experimentally assess the predictive power of individual characteristics regarding different measures of risk-taking behavior (self-reported propensity measures as well as lottery tasks). The results suggest that demographic characteristics (age, sex) are the most reliable correlates of risk preference. Conversely, household income, fluid intelligence, and years of education were either positively or negatively associated with risk preference, depending on how risk preference was operationalized.

The recent truth-telling module of Bosch-Rosa, Neyse, and Nosenzo (2020 SOEP-IS wave) tests the socio-economic correlates of the truth-telling games of Fischbacher and Föllmi-Heusi (2013) and Gneezy et al. (2018) with about 1400 experimental subjects.⁴ In the Fischbacher and Föllmi-Heusi (2013) game, the players secretly roll a dice that determines how much money they will win. They are invited to report the number on the dice without being monitored by the experimenter. This means that they can misreport the number without being noticed. The game by Gneezy et al. (2018) is the computerized version of the former, where the die is replaced with black boxes on the screen. In this version, it is possible to identify players who misreport the numbers. The module developed by Bosch-Rosa et al. aims to test the lab results on a representative sample, to investigate the socio-economic underpinnings of lying behavior, and to investigate the differences between the two versions of the task.

2.3 Within household behavior

While a large set of experiments focus on individual decisions, there is a growing body of literature on collective decision-making and decision-making for others. These studies often depend on random matching protocols, where anonymous subjects are matched in the laboratory setting. On the one hand, the anonymity of the laboratory helps researchers to investigate economic decisions in a setting where personal interactions and biases are isolated. For example, the generosity that an anonymous dictator game elicits in the lab setting is based on an actual monetary decision that is not aimed at a particular individual but at an anonymous participant. While this is an advantage for specific research questions such as “Are people solely interested in maximizing their own profits, or do they have other-regarding preferences?”, it is relatively difficult to investigate behavior in people’s actual relationships. In real-life situations, individuals may consider how their decisions will affect relatives. Parents making investment decisions, for instance, are usually well aware that their decisions will directly affect their children. Such decisions are difficult to address in the laboratory but ideal to study in household surveys.

Examples

Engel et al. (2018) elicit risk decisions of household members who make decisions for themselves and also for other members of the household. Existence of decision patterns at the household level might also suggest that household surveys are helpful in studying generational spillovers of certain preferences, decisions, and behaviors. In another study, Bacon et al. (2014) study the correlation of risk attitudes between spouses.

2.4 Good scientific practice: replication, reproducibility, and open science

Replication and reproducibility are two important components of good scientific practice. While SOEP-IS serves as an ideal platform for testing lab results and findings gathered from small samples, it is also a good resource for direct or conceptual replication studies.

As a supporter of good scientific practice and open science, SOEP-IS invites (but not obliges) applicants whose proposals have been accepted to pre-register their hypotheses and pre-analysis plans in an online repository. In line with this view, the standard SOEP-IS module application requires certain details of the modules to be specified, including the dependent and independent variables, planned analyses, and power calculations. These requirements aim to encourage researchers to plan their study before data collection. The application form of the SOEP-IS closely follows the pre-registration protocol Open Science Framework. One of the key objectives of the SOEP-IS is that the data are not abandoned, for example, in case of a null result. To tackle this issue, the team makes all modules available to the research community after an embargo period. Analysis codes and replication materials can also be requested from the data delivery team of SOEP.

2.5 Permanent scientific data infrastructure

The SOEP-IS data are collected, processed, and archived anew every year. As a result, the time frame over which respondents can be observed expands over time. This increases the potential for intertemporal analysis. With the data, it is now even possible to implement analysis concepts in which stated preference parameters measured “today” are explained by respondents’ histories, and in which both serve as explanatory variables for decisions in later years. For example, behavioral questions such as “How is support for higher taxes shaped by income changes and risk preferences?” or “How do risk preferences and income interact over the life course?” can easily be investigated in a panel study.

Example

The Appendix Sect. 5.1 provides an illustration of an analysis based on SOEP-IS data on income redistribution.

2.6 Fostering interdisciplinary research

Although scientists often have excellent subject-specific knowledge, they sometimes overlook developments in other disciplines. In fields such as behavioral economics and psychology, as well as at the interface between educational science, economics, and sociology, a look at neighboring disciplines can often provide new impulses. SOEP-IS offers researchers from all disciplines the opportunity to jointly develop and evaluate survey content, thereby gaining more comprehensive insights than through a single-discipline approach.

3 How to use the SOEP innovation sample

The SOEP-IS is a panel study that has been running since 2012. It offers researchers the opportunity to collect data tailored to their specific research questions (see Richter, Schupp, et al., 2015). As a result, in addition to containing a broad set of standard survey questions on socio-economic and socio-demographic background, SOEP-IS incorporates data gathered through user-designed surveys and experimental modules.

As the data collection is performed by an independent company and SOEP does not share sensitive identity data with the researchers the SOEP datasets are always anonymous.

Every year, interested researchers can propose their projects to SOEP survey management. Additionally, all SOEP-IS data and the past innovation modules that are not in an embargo period are available to researchers, even if they have never proposed any innovation modules.

3.1 Sample and interview procedure

SOEP and SOEP-IS are two independent random samples of German households. The SOEP-IS is designed to enhance the core SOEP and meets the SOEP’s criteria for representative sampling of households and individuals from a cross-sectional and longitudinal perspective. SOEP-IS represents (private) households in Germany, with every adult in the household being surveyed by questionnaire (at least) once per year. Two existing longitudinal core SOEP samples (samples E and I) served as the foundation for the SOEP Innovation Sample in 2011. The longitudinal data from the existing SOEP subsamples expands the potential for the analysis of questions in earlier waves of the SOEP-IS, since the results from these surveys can be linked to newer longitudinal data. The respondents of core SOEP samples E and I left the SOEP and are interviewed with the SOEP-IS questionnaire only. In 2011, the number of respondents in SOEP-IS was 2506 (1504 households), with additional boost samples increasing this number to 5633 (3717 households) in 2018.

In 2018, the sample covered the adult age range from 17 to 97 years, comprising respondents with differing levels of education (15% low education; 56% medium education; and 29% high education), work situations (35% working full-time, 15% working part-time, 23% retired, 27% not working), and marital statuses (23% single, 56% married, 13% divorced, 8% widowed). In addition, 28% of respondents lived in households with children aged 16 years or younger, and in these households, the primary caregiver (usually the mother) provided yearly information on their children’s development.

The whole pool of SOEP-IS participants, or a subsample thereof, can participate in a newly proposed module. In either case, the sample size requirements should be reported and justified (e.g., with power analysis) in the application. Data collection is done through computer-assisted personal interviewing (CAPI). This means that interviewers visit households personally to collect the data with the help of portable devices (e.g., laptops or tablets). In the case of tasks where anonymity is required, such as economic experiments, interviewers leave the room or home, giving the respondents space to make decisions on their own.

3.2 Module submission and evaluation

Survey questions and experiments from accepted proposals are introduced into SOEP-IS at no additional cost.⁵ Yet, applicants are expected to secure outside funding to cover the costs of any incentives for behavioral experiments or the collection of biomarkers (e.g., saliva) if envisioned by the proposal. Applicants who plan to apply for outside funding for a module can ask for a support letter from SOEP-IS. Furthermore, modules that are funded by the German Research Foundation (DFG) usually have an advantage in the selection process. All proposals are evaluated according to scientific quality, suitability, and feasibility. Table 2 presents the usual timeline for SOEP-IS applications up until 2021. Note that the submission deadlines can vary across years.⁶

Table 2 Timeline for SOEP-IS Module Applications

30 November/Year 0	Deadline for e-mail to SOEP-IS team with a brief description of the proposed project
31 December/Year 0	Deadline for full proposals
April/Year 1	Notification of acceptance
September–ecember/Year 1	Data collection
April/Year 2	Delivery of raw data
April/Year 3	Delivery of final data
	(including harmonized variables and survey weights)
April/Year 4	End of the embargo period

3.3 Content

In addition to standard socio-economic survey items, SOEP-IS contains modules from various academic disciplines. With these modules, behavioral economists can address countless interdisciplinary research questions. In Table 3, we present a small set of variables that would interest behavioral economists. We also present a standard set of socio-economic variables in Table 4.

Table 3 A selection of focal variables relevant for behavioral economists

	Behavioral variables	Year	N	Notes
1	Risk attitudes	Every year	Full sample	Self-reported Likert scale
2	Time preferences	2014	Full sample	Self-reported Likert scale
3	Patience/impulsivity	2013, 2018	Full sample	Self-reported Likert scale
4	Reciprocity	2015	Full sample	Self-reported Likert scale
5	Locus of control	2017	Full sample	Self-reported Likert scale
6	Trust	2013, 2018	Full sample	Self-reported Likert scale
7	Life satisfaction	Every year	Full sample	Self-reported Likert scale in different domains
8	Cognitive abilities	2020	Full sample	3 Item CRT; procedure includes beliefs about self and others
9	Personality scales	2013, 2015, 2017	Full sample	Big-5
10	Overconfidence in different life domains	2014	Full sample	Non-incentivized overplacement questions in different domains
11	Domain-specific	2014	Full sample	Self-reported Likert scale risk preferences
12	Honesty	2020	$\sim 2200$	Dice-rolling game (incentivized)
13	Social comparison	2018	$\sim 1500$	INCOM social comparison scale

Table 4 A selection of background socio-economic variables

1	Year of birth	The birth year of the respondent
2	Gender	Binary biological sex variable
3	Education	Various variables available (e.g., the highest degree obtained, number of years in education)
4	Employment	Detailed questions such as working hours, change of jobs, industry, entrepreneurship, managerial duties, expectations
5	Income	Detailed questions such as household and individual income taxes, asset balance, inheritances
6	Family and Social	Questions on family size and changes such as birth or death, Networks number of friends, social time with friends etc
7	Marital status	Categories: Single, married (together), married (separated), same sex partnership, divorced, widowed
8	Region	State of residence, living in an urban or a rural area Born East or West Germany
9	Health	Detailed questions including current health status illnesses, stress, visits to the doctor health status of children, hospital stays, health insurance
10	Home	Questions such as size of the household, monthly rent, owned property, cleaning and household help
11	Time use	Questions on time use on work, weekends, holidays with family, children, on transportation

Since the content of the SOEP-IS is growing each year and there is a vast number of information about each module (e.g., sample size, methods, module contents, descriptive statistics), we developed two annually updated lists to complement this guide. The first one, which we created particularly to serve as an external Appendix to the current paper, is a SOEP working paper (Kara et al., 2021). This working paper contains the complete list of previous innovation modules and details of experiments conducted in SOEP-IS since the beginning. SOEP-IS will update this working paper after each data release. The second list is an extensive webpage that was developed in parallel to the current paper (http://companion-is.soep.de). The SOEP-IS-Companion is both a reference book and a practical guide. It provides information about the different questionnaires, the composition of the sample, and the structure of the data. This website is also updated regularly.

3.4 Data access and documentation

The SOEP Research Data Center distributes the SOEP-IS data as an independent dataset to researchers at universities and research institutes around the world for research and teaching purposes. The data from SOEP-IS modules are provided exclusively to those researchers who submitted the respective proposals up to the end of the embargo period of 12 months. After that point in time, the data are released to the entire SOEP user community for secondary analysis.

The direct use of SOEP-IS data is subject to the strict provisions of German data protection law. A signed SOEP data distribution contract is therefore a precondition for working with SOEP-IS data (http://www.diw.de/soep-contractmanagement). After the application for data use has been approved, users can download the data from SOEP servers through a secure data transfer system. The average duration of the process is 1 day.

3.5 Delivery of the data

The datasets can be provided to researchers in various data formats depending on their needs. The relevant socio-economic variables and standard self-reported attitude and well-being variables (e.g., subjective well-being in different life domains, risk attitudes, patience, health conditions, etc.) are already included in the datasets released to users in “long” format (i.e., data from all survey years in a single dataset). All data from the SOEP-IS modules are in a separate dataset, which is provided in a “long” format as well.

4 Conclusion and discussion

In this paper, we presented SOEP-IS, which offers rich, free, panel data to researchers along with the possibility to conduct innovative investigations with participants of a representative panel. This includes economic experiments. Based on a review of SOEP-IS-based research, we argue that it is an ideal platform to validate laboratory findings and ask certain research questions that are rather difficult to investigate in standard experimental settings.

In our view, SOEP-IS, and panel studies in general, have a big potential as an instrument in the toolbox of behavioral economists. As with all methods, panel studies also have several comparative disadvantages against laboratory studies, such as less control, longer data collection times when integrating new modules, or time restrictions in data collection. Nevertheless, adding panel surveys to the behavioral economists’ toolbox can help understand economic behaviors further.