Missing the Target? Using Surveys to Validate Social Media Ad Targeting

Abstract

Facebook ads are increasingly used by political scientists as a method of survey recruitment. A key advantage is said to be the ability to recruit targeted audiences defined by demographics, political beliefs, location, and numerous other attributes. The same feature has been decried by non-researchers concerned about potential racial discrimination and foreign influence in elections. The extent to which these ads actually reach their targets, however, is unknown. Using a series of six surveys and 20 targeted ads, I show these ads regularly fail to reach their targets. The success rate ranges from 23 to 99 percent, and ads targeted toward groups defined by self-reported data and broader geographic locations are generally more successful.

1. Introduction

Advertisements placed on social media websites are increasingly used by political scientists as a means of survey recruitment. In addition to the low cost, relative to more traditional methods, a key advantage is the power to target advertisements toward particular geographic, demographic, and political groups. The ability to target surveys toward specific subpopulations potentially allows researchers to focus on specific groups, such as voters in particular geographic locations or with particular demographic or political characteristics, who may be key to testing empirical claims.

Several recent papers in political science have taken advantage of such targeted ads. For instance, Samuels and Zucco (2013; 2014) show Facebook can be used to recruit large samples of Brazilian respondents, Hirano et al. (2015) use Facebook ads to recruit likely primary voters in particular state elections, and Sances (2018) uses these ads to recruit local voters in particular cities. Beyond geography, Zhang et al. (2018) show that demographic ad targets can be used to implement a form of quota-sampling, allowing researchers to obtain a sample that is more nationally representative than low-cost alternatives such as Mechanical Turk (Berinsky et al. 2012; see also Boas et al. 2018). And for targeting users by political beliefs, Jäger (2017) uses Facebook to recruit party activists in the US, Germany, and Thailand.

The same technology that political scientists have celebrated for research purposes is also decried by members of the broader public. Targeted campaign appeals have been at the center of alleged foreign interference in the 2016 election, during which several “Russian-linked Facebook ads” were reportedly “highly sophisticated in their targeting of key demographic groups in areas of (Michigan and Wisconsin) that turned out to be pivotal” (Raju et al. 2017; see also Rosenberg et al. 2018). Outside of campaigns, Facebook's “ability to slice and dice the habits of its hundreds of millions of users” may have enabled some marketers to engage in illegal discriminatory practices (Maheshwari and Isaac 2016). In response to the controversy surrounding such discriminatory ads, Facebook announced it would no longer allow those advertising housing, credit, or employment to target users by race or ethnicity (targeting by race/ethnicity for academic purposes is still permitted).

Despite the alleged benefits and risks of this technology, it is currently unclear just how well it actually works. In this note, I validate these ad targets using direct survey measures. Specifically, I compare the purported characteristics of targeted groups to the self-reported characteristics of survey respondents recruited via ads targeting these groups. Using a series of six studies and a total of 20 targeted ads, I show that targeted ads often do not reach their intended targets. The success rate—the percentage of respondents who report being in the group toward which the ad was targeted—ranges from a low of 24 percent (for ads targeted toward African Americans) to a high of 99.8 percent (for ads targeted toward those over 25). Higher success rates do not appear to be driven by whether the ads target demographics, political attitudes, or geography. Rather, the most successful targets appear to be those defined by characteristics that Facebook users self-report on their user profiles, with lower success rates for targets defined by statistical models and third-party data sources.

2. Data and measurement

The six studies were conducted between October 2016 and February 2018. Table 1 provides a summary. The first column assigns each study an arbitrary number for reference. The second column shows the dates the study was fielded. The third column lists the targets that did not vary within each study. For instance, all ads associated with Study II were targeted toward residents of Memphis aged 18 and older. The fourth column lists any targets that varied within each study. For instance, one ad in Study II targeted residents of Memphis who were classified by Facebook as White; etc. The remaining columns list the number of ad clicks, cost per click, and total sample recruited.

Table 1. Studies used to validate Facebook ad targeting.

“Common Targets” lists the ad targets that did not vary within each study (e.g., all ads associated with Study II were targeted toward residents of Memphis aged 18 and older). “Varying Targets” lists any targets that varied within each study (e.g., one ad in Study II targeted residents of Memphis who were classified by Facebook as White).

Each study was associated with a different Qualtrics survey, and each ad within a study used the same base URL. To record which respondents were recruited by which ads, the base link was appended with an identifier recorded by Qualtrics in an embedded data field. For Studies I–IV, the ad recruitment text was modeled after recruitment ads used in Berinsky et al. (2012), and respondents were incentivized to participate with the chance to win an Amazon.com gift card. For Studies V and VI, the text was modeled after Jäger 2017. No incentives were given in Study V, and in Study VI the promise of incentives was randomized such that the ad text in brackets was shown only to those residing in 20 of the 40 targeted zip codes.

Readers unfamiliar with the technical details of specifying ad targets may consult the papers cited in the introduction (see especially the appendix to Jäger 2017). However, it is important to discuss the details regarding certain targeted characteristics. Regarding demographics, Facebook uses the phrase “multicultural affinity” instead of “race” or “ethnicity.” When introducing this category in 2015, they described it as “the quality of people who are interested in and likely to respond well to multicultural content.” According to the marketing firm Performics, “The targeting is based on affinity, not ethnicity” (Allen 2015), and in 2017 Facebook moved multicultural affinity from the “demographics” target cluster to the “behaviors” cluster in the ad interface (personal communication with Facebook Support, 9/15/2017). However, the subcategories (as of 17 January 2018) for affinity are racial and ethnic groups: “African American (US)”; “Asian American (US)”; “Hispanic (US—All)”; “Hispanic (US—Bilingual)”; “Hispanic (US—English Dominant)”; “Hispanic (US—Spanish Dominant)”.¹ Additionally, Facebook in late 2017 removed the ability of advertisers to exclude certain affinity groups from being targeted, explicitly citing concerns that racial and ethnic groups could be excluded via the affinity targets (Guynn 2017).

When targeting by location in my studies, I always targeted those residing in a particular area, setting a radius of zero miles so as to recruit only city residents and not those living in a broader metropolitan area. City-recruited respondents affirmed they lived in a given city on the Qualtrics consent screen. This may artificially boost the success with which these ads appear to target city residents; however, only 6 percent of those who clicked through the consent screen failed to consent, suggesting that not many users outside the city were recruited into Qualtrics via the ad.

The 1,244 zip codes targeted in Study I were chosen for the purpose of reaching downstate Illinois residents living outside of large cities. The 12 zip codes in Study II were chosen for the purpose of reaching African American voters, after the multicultural affinity targeting was found to perform poorly; these are the 12 zip codes with at least 75 percent African American population according to the US Census. The 40 zip codes in Study VI come from the Miami, FL area, and were chosen for having the highest share of enrollment in marketplace insurance plans under the Affordable Care Act. Zip codes were targeted collectively in Studies I and II, such that in Study I there was one ad for all 1,244 zip codes and in Study II there was one ad for all 12 zip codes. In Study VI, there was one unique ad for each of 40 zip codes.

3. Results

Table 1 shows the results, sorted by the type of ad target category: demographics, political behavior, and geography. The first column lists the ad target, and the second column gives the study number for referring back to Table 2. For each target, the survey included a question with response options that closely matched the target categories advertisers may select on Facebook. The third column of Table 1 gives the number of responses given to the survey item used to validate a given target. The fourth column gives the percentage of these responses that match the ad target (total recruited matching the target/total recruited by the ad).

Table 2. Validation results.

For each target-study combination, this table gives the number of Facebook users recruited into the survey via the ad target, and the match rate, or the number of Facebook users recruited whose self-reported characteristics match the target, divided by the number recruited.

Starting with demographics, ads that target users based on age are highly successful. Of the 78 respondents recruited by the 18–24 target, 91 percent report an age that was between 18 and 24. Similarly, of the 551 recruited by the 25+ target, 99.8 percent report an age of 25 or above. However, education and race show much lower match rates. Just 68 percent of the 283 recruited with the “Less than BA” target report an education level lower than a BA, while 74 percent of the 350 “BA or Higher” recruits report having at least a BA. Of the 176 respondents recruited using the “White” target, 93 percent identify as White, but only 23 percent of the 547 “Black” recruits identify as Black.² It is intuitive that targets based on what Facebook calls “cultural affinity” would not successfully recruit by race; on the other hand, much public attention has been devoted to the possibility of marketers using cultural targets to discriminate based on race. To my knowledge, the current analysis is the only evidence as to whether such discrimination is possible.

Targeting users by political ideology is generally successful (see also Bond and Messing 2015). Of the 34 recruited by the ad targeting “Very Liberal” voters, 97 percent are indeed at a 1 or a 2 on a 7-point conservatism scale. Of the 34 recruited by the “Liberal” target, 80 percent are at a 3 or lower on the same scale. Partisanship is much less reliable. Of the 493 respondents recruited by an ad targeting Democratic voters, just 59 percent identify as a Democrat or a Democrat-leaning independent. And out of the 44 respondents recruited with an ad targeted toward those having donated to liberal political causes, just 39 percent report donating to liberal causes in the past two years.

The seven ads targeting users by city of residence are generally successful. Excluding those recruited by ads targeting Atlanta residents, on average 97 percent of those recruited by a given city target report living in zip codes contained in that city. In Atlanta, the match rate is 82 percent.³ Zip code targeting gives mixed results. In Study 1, the same ad was targeted toward users reportedly residing in any of 1,244 zip codes in Illinois. Out of 1,035 respondents recruited via this ad, 97 percent reported a zip code in the set of 1,244. More focused recruitment by zip code is less successful. In Study II, 12 zip codes in Memphis were targeted. Of the 243 respondents recruited using these targets, only 44 percent reported living in these zip codes. In Study VI, 78 percent of respondents’ zip codes matched their ad targets' codes.

4. Limitations

Poor match rates may result from careless or dishonest respondents, and this risk may be greater when respondents are incentivized. However, inattention and dishonesty should be problematic for all items, yet we see that some items (such as age) have significantly higher match rates than others. Note also that incentives were not used on Study V, which saw match rates as low as 39 percent.

To more directly assess these issues, Study VI included a Screener question (Berinsky et al. 2014). Respondents were asked their favorite color, then informed the question was in fact an attention check, and that to demonstrate attention they should select both “red” and “green” as their answer. Twenty out of 51 respondents, or 39 percent, passed the attention check. This is much lower than the 70 percent passage rate observed on a conventional online panel (Berinsky et al. 2014). I examine the relationship between attention, match rates, and incentives in the Appendix. To summarize, while the match rate is higher among passers, the difference is imprecisely estimated (p = 0.083); neither attention nor match rates are significantly different when incentives are used.

Poor geographic match rates could also result from disparities in access to Facebook. If certain areas are low-access, ads targeted to those areas may have less success. In the Appendix, I examine the correlation between match rates and Facebook access (number of users divided by population) in Study VI. I find discrepant access does not explain discrepant match rates.

5. Discussion and conclusion

Social media websites such as Facebook collect and store voluminous amounts of data about their users, data that may present a boon to social researchers and a nuisance to the public. However, the extent to which either of these possibilities is realized depends on the ability of researchers and marketers to use this data to reach particular groups. Using a series of surveys linked to targeted ad campaigns on the Facebook platform, I have shown the ability varies greatly depending on the target category. While some ads reach their target groups 99 percent of the time, others reach their targets only 23 percent of the time.

What explains the relative success or failure of different targets to reach their intended audience? Examining the demographics and political behavior targets in Table 1, age and ideology have match rates in the 80s and 90s, while education, race, party, and political donating have match rates as low as the 20s and only as high as the 70s. Looking further, the more accurate categories tend to have different sources than the less accurate categories. Age, for instance, is (presumably) observed directly by Facebook using the date of birth reported on an individual's profile page. The “very liberal” target, moreover, is described by Facebook on the advertiser interface as matching “People in the USA who have a very liberal political affiliation,” while the target for college graduates is described as “People who indicated their highest education level as College grad.”

Facebook's descriptions of the less successful categories are more nuanced. The description for the “Democratic Party (United States)” category is “People who have expressed an interest in or like pages related to Democratic Party (United States)”; the description for the African American multicultural affinity category is, “People who live in the United States whose activity on Facebook aligns with African American multicultural affinity”; and the description for those matching ‘Donate to liberal political causes’ is “People who are interested in donating to liberal political causes. / Source: Partner Category based on information provided by Epsilon: Multi-sourced.” Assignment to these categories is apparently less a function of self-reporting by users, and more a function of Facebook predicting categories based on user activity and third-party data.⁴

Geographic targeting is generally successful, provided the targeting is not too granular. At the level of cities and large collections of zip codes, match rates tend to be in the 90s, as high as 97 percent for the 1,244 zip codes targeted in Study I. At a finer level of detail, targeting fails: while Study II attempted to recruit respondents from 12 particular zip codes in a given city, the success rate was just 52 percent. This variation in success may be because broad geographic categories such as city are directly reported to Facebook by users via their profiles, while more granular location data such as zip code may be obtained via IP geolocation.⁵

Those interested in using Facebook for survey recruitment can adopt a validation strategy from the literature on online panels, which have faced similar issues involving “false qualifying” (Baker et al. 2010, pp. 744–745). As suggested by Phillips (2016) and implemented by Craig et al. (2013), researchers can simply compare survey responses to profile characteristics, or in the case of Facebook ads, ad target characteristics. I include instructions for implementing this strategy in the Appendix.

Facebook does indeed hold promise as a way for researchers to target specific groups. However, researchers should exercise caution, and should always validate their recruitment strategies using direct survey measures when possible. Ads targeted by age, ideology, and broader geographic areas such as city are far more likely to reach their intended audience than those targeted by race, partisanship, and sub-municipal areas.

At the same time, concerns about marketers using targets to discriminate by race, or for political actors to influence key demographics in small geographic areas, may be overstated. Since the 2016 election, there has been much public concern about foreign actors using social media to target groups susceptible to persuasion and mobilization. As one Washington Post columnist summarizes, “imagine being able to target this message with minute precision: say, telling black voters in swing counties that Hillary Clinton was an incorrigible racist, or enraging white, male gun lovers with her supposed plans to roll back the Second Amendment. Imagine how quickly such misinformation could spread and metastasize” (Emba 2017). Of course, whether such messages could actually change anyone's mind is a classic question in social science. Yet, as this note has shown, it is a debate that may not be necessary here, given the limited power of ad targeting.