Emerging trends: When can users trust GPT, and when should they intervene?

Abstract

Usage of large language models and chat bots will almost surely continue to grow, since they are so easy to use, and so (incredibly) credible. I would be more comfortable with this reality if we encouraged more evaluations with humans-in-the-loop to come up with a better characterization of when the machine can be trusted and when humans should intervene. This article will describe a homework assignment, where I asked my students to use tools such as chat bots and web search to write a number of essays. Even after considerable discussion in class on hallucinations, many of the essays were full of misinformation that should have been fact-checked. Apparently, it is easier to believe ChatGPT than to be skeptical. Fact-checking and web search are too much trouble.

1. Introduction

Much has been written about GPT including Dale (2021), the most read paper in this journal.a ChatGPT is extremely popular.

ChatGPT sets record for fastest-growing user base.b

Why is GPT so popular? The fact that so much has been written on this question suggests that the definitive answer to that question has not been written (yet). And it is unlikely that it will be written soon. That said, GPT is remarkably easy to use, and the outputs are written in an authoritative style that appears to be credible (even when it is making stuff up).

Is ChatGPT a Google Killer? Much has also been written on this question. As the cliche goes, it is difficult to make predictions, especially about the future, but the recent excitement over retrieval-augmented generation (Lewis et al. 2020) suggests there is plenty of room for synergy between chat bots and web search.

Our last emerging trends article Church and Yue (2023) on “Smooth-Talking Machines” suggested Chat bots have amazing strengths (fluency) and amazing weaknesses (trustworthiness). Search can help with trustworthiness. Chat bots have a serious problem with hallucinations. Search can be used in a fact-checking mode to mitigate some of this risk. But, as a practical matter, fact-checking is hard work, and probably won’t happen as much as it should.

2. Evaluation of prompt engineering

How do we evaluate prompt engineering? InstructEvalc (Chia et al. 2023) combines a number of other benchmarksd (Dua et al. 2019; Hendrycks et al. 2020; Chen et al. 2021; Frohberg and Binder 2022; Srivastava et al. 2023). These methods represent the current consensus on how to evaluate deep nets, but it is important to evaluate more than just the machine. We want to measure both what the machine does and what ordinary users (and my students) will do with the machine. In addition to benchmarks such as InstructEval, we should also run evaluations with humans-in-the-loop.

OpenAI, the organization behind ChatGPT, believes it is important to roll out new GPTs incrementally because it takes time for the community to learn how to use the next GPT, and because it takes time for OpenAI to appreciate how the community will take advantage of the opportunities. The process is very much a collaboration between people and machines. We would like to see evaluations that are more collaborative than traditional evaluations of deep nets.

3. Homework: use ChatGPT to write essays

This article will describe a homework assignment that I gave to my class on NLP (Natural Language Processing).e The homework asked the students to use tools such as chat bots and web search to write a number of essays. I was surprised how well ChatGPT did on some of these essays, but I was also surprised how much the students tended to believe ChatGPT, even when it was making stuff up. Even after considerable discussion in class on hallucinations, many of the student essays contained misinformation that the students should have fact-checked, as we will see.

There is considerable interest in how misinformation and toxicity spread online (Qazvinian et al. 2011; Vicario et al. 2016; Shu et al. 2017; Zubiaga et al. 2017; Cinelli et al. 2021). Work on machine learning classifiers (Poletto et al. 2020; Fortuna et al. 2021)f is unlikely to help much, given the incentives. Just as we cannot expect tobacco companies to sell fewer cigarettes and prioritize public health ahead of profits, so too, it may be asking too much of companies (and countries) to stop trafficking in misinformation given that it is so effective and so insanely profitable (Church et al. 2023). If we gave these companies a toxicity classifier that just worked, given current incentives to maximize profits, they should run the classifier in the reverse direction to maximize toxicity, since toxicity is profitable.

The homework mentioned above raises an additional concern. Not only should we be concerned about the supply of misinformation, but we should also be concerned about demand for misinformation. As mentioned above, my students could have easily fact-checked their homework, but they chose not to do so. They were prepared to believe much of what ChatGPT says, because of how it says what it says, and ease-of-use. It is easier to believe ChatGPT than to be skeptical. Fact-checking and web search are too much trouble.

4. Lay of the land

Table 1 summarizes my interpretation of what the students did with large language models (LLMs). There are also columns for what the students could have done with traditional NLP and web search. LLMs such as ChatGPT are amazingly good on metaphors, probably better than non-native speakers of English. Most of the students are international students. They did not grow up in America and they are not familiar with Americana (such as metaphors involving American baseball). Although web search also works well on metaphors and Americana, my students tended to prefer ChatGPT, perhaps because it is the new hot thing, or perhaps because bots are easier to use (and require fewer clicks than web search).

Table 1. LLMs have amazing strengths and amazing weaknesses

I was surprised how well ChatGPT did on metaphors, but I was also surprised how poorly it did on references. If you ask ChatGPT for references on some topic, it will make up papers that do not stand up to fact-checking. We will discuss some of the amazingly good cases in Section 5 and some of the amazingly bad cases in Section 6.

4.1 You have no idea how much we’re using ChatGPT

The homework assignment was inspired by Owen Terry’s I’m a Student. You have no idea how much we’re using ChatGPT. g We discussed Terry’s essay in Section 2.6 of Church and Yue (2023). In an interview on NPR,h Terry, a rising sophomore at Columbia, identified some of ChatGPT’s strengths and weaknesses. ChatGPT is good at producing thesis statements and outlines, but it does not capture the student’s style, and it is worse on quotes. If you ask for quotes, it makes stuff up.

4.2 Zero-tolerance for misinformation

Quotes and the other amazingly bad cases in Table 1 involve hallucinations. The term, hallucination, has become a nice way of referring to what we used to call bugs and computer errors:

To Err is Human; To Really Foul Things Up Requires a Computer i

There should be no excuse for making stuff up. And it should be even worse for someone to traffic in misinformation. If someone cheated on their CV, they could be fired, even years after the offense. So too, there should be little tolerance for handing in homework with misinformation, especially when it is so easy to fact-check with search.

It should not be the teacher’s responsibility to fact-check the homework. To discourage the spread of misinformation, there need to be prohibitively high penalties in the court of public opinion for trafficking in misinformation. To stop the spread of misinformation, we need to go after all parties including suppliers, users, and market makers. If we do not address the problem, misinformation could lead to a loss of confidence in all things digital.

4.3 Adversarial attacks

There is a considerable literature on adversarial attacks on LLMs (Jia and Liang 2017; Morris et al. 2020; Camburu et al. 2020; Wang et al. 2021; Li et al. 2021; Ziegler et al. 2022; Wang et al. 2023). We do not see as many papers on adversarial attacks on web search, perhaps because it is too easy to find misinformation on the web. When we look at logs of speech APIs and agents like SIRI, we find lots of kids having fun at the expense of the API, but we see less “prank calling” attacks against web search:

Do you have Sir Walter Raleigh in a can? Better let him out! j

One might view search engine optimization as an adversarial attack against web search, though Google documentation prefers to view the subject as an opportunity to teach content owners how to work with Google, as opposed to how to work against Google.k I prefer to view arbitrage as a way to teach the market makers such as Google and ChatGPT how to do better in the future.

I am not sure why there is so much literature on adversarial attacks against ChatGPT given some of the observations in Table 1. It is really not hard to generate prompts that will produce hallucinations:

1. 1. Ask ChatGPT for quotes

2. 2. Ask ChatGPT for references

3. 3. Ask ChatGPT to crawl links, quotes, references

4. 4. Ask ChatGPT to evaluate an expression with more than two big numbers

It is not hard to find opportunities for improvement. Consider Chain-of-Thought Prompting (Wei et al. 2022). This paper is widely cited because it offers a constructive workaround to an obvious weakness in ChatGPT. ChatGPT is not particularly good at decomposing complex tasks into two or more simpler tasks. ChatGPT lacks principles such as superposition,l a key principle in linear algebra and linear systems.m Intermediate representations and compositionality used to be hot topics in linguistics and computer science. End-to-end systems are easier to optimize and may perform well on benchmarks, but modularity has other advantages:

1. 1. explanation,

2. 2. capturing relevant generalizations (Chomsky 1957, 1965), and

3. 3. model size (number of parameters) (Minsky and Papert 1969).

4.4 Human-in-the-loop

Much of the adversarial literature above is focused on use cases with no human in the loop. In Church and Yue (2023), we envisioned a human-in-the-loop collaboration of humans with machines. To make this work, we need to characterize which subtasks are appropriate for humans and which are appropriate for machines. Evaluations in our field tend to focus on how well the machines are doing by themselves, as opposed to which tasks are appropriate for humans and which are appropriate for machines.

The adversarial literature above is asking the right questions:

1. 1. What ChatGPT does well?

2. 2. What ChatGPT does not do well? (Wang et al. 2023)

but we were hoping for answers that would be more helpful in human-in-the-loop settings. For example, the first answer to the second question above in Wang et al. (2023) is

The absolute performance of ChatGPT on adversarial and OOD classification tasks is still far from perfection even if it outperforms most of the counterparts.

This answer is not that useful in human-in-the-loop settings. We already know that the machine is more fluent than trustworthy (on average) (Church and Yue 2023). What we would like to know is: when can we trust the machine, and when should we ask the human to step in.

5. Amazingly good

5.1 Metaphor

I expected the following sports metaphors to be hard, especially since most of the students in the class are not native speakers of English. But I was surprised how well ChatGPT did on these.

I asked the students to use chat bots and search engines to explain what the following terms mean. Which sports are these terms from? What do they mean in that sport? How are they used metaphorically outside of that sport? If the term is more common in one English speaking country than another, can you name a country that is likely to use the term?

1. 1. cover all the bases

2. 2. drop the ball

3. 3. dunk

4. 4. fumble

5. 5. get on base

6. 6. hit a home run

7. 7. out in left field

8. 8. punt (as a noun)

9. 9. punt (as a verb)

10. 10. ragging the puck

11. 11. run out the clock

12. 12. sticky wicket

13. 13. strike out

In most cases, ChatGPT is at least as good as Google, and easier to use. Users like instant answers, where they do not have to click on anything. That said, content providers and developers of search engines depend on behavioral signals like clicks to make improvements. In the long run, without these behavorial signals, results are likely to degrade over time. Chat bots may be doing well in the short term, but it may be in their best interest to – and good for the internet ecosystem – if chat bots work with content suppliers, rather than dis-intermediate them.

For more obscure metaphors like ragging the puck, Google and Bing return a linkn with an excellent answer, whereas ChatGPT does not explain the metaphorical sense. The version of Bing chat (which uses ChatGPT) provides two definitions with links. I would not have known this idiom if I had not spent so much time reading Canadian Hansards (Gale and Church 1991). In the Hansards, ragging the puck means to waste time with great skill. It is something that you accuse the other guy of doing, or deny that you are doing, because the filibuster is not a thing in the Canadian Parliament.

Metaphor is a classic hard problem in computational linguistics. There is a considerable literature on this topic (Carbonell 1980; Fass and Wilks 1983; Martin 1990; Hobbs 1992; Gedigian et al. 2006; Krishnakumaran and Zhu 2007; Lakoff 2008; Lakoff and Johnson 2008; Shutova 2010; Mohammad et al. 2016), but most of us that have been working in computational linguistics for a few decades have rather low expectations. We grew up with certain stereotypes. The 1966 TV show, “Get Smart,” introduced a robot character, Hymie, in season 1, episode 19.o The running gag was that Hymie was too literal. When Max, the leading character, tells Hymie, “you knock me out,” the robot does exactly that.p

Given this stereotype, it is surprising how well ChatGPT does on: “you knock me out.”

1. Prompt : What does “you knock me out” mean?

2. ChatGPT : The phrase “you knock me out” is an idiomatic expression in English that is often used to convey a strong positive reaction to someone or something. When someone says, “You knock me out,” they are typically expressing admiration, surprise, or amazement. It’s a way of saying that the person or thing being referred to has a powerful or impressive effect on the speaker. For example, if someone is really impressed by a performance, a joke, or someone’s charisma, they might say, “Wow, you really knock me out!” It’s a colloquial and informal way of expressing appreciation or being captivated by someone or something.

5.2 Documentation

When I constructed the homework, I expected ChatGPT to work well on documentation. Last summer, some undergraduates taught me that they prefer ChatGPT to Stack Overflow. To test this observation on my class of masters students, I asked them to do the following with vectors from MUSE (which they had used previously):

Some people are finding ChatGPT to be useful for documentation (e.g., as a replacement for Stack Overflow). Suppose I have a set of vectors from MUSE and I want to index them with annoy and/or faiss.

1. 1. Find some documentation for how to do this.

2. 2. If possible, find some examples.

3. 3. Write a program that creates the index.

4. 4. Write a program that queries the index.

5. 5. Provide a short description of what approximate nearest neighbors does, and what is it useful for.

6. 6. Do a literature survey to find some primary papers on approximate nearest neighbors, as well as some survey papers on the topic

As with the previous two questions, in addition to answering the specific questions, I am more interested in which tools you found useful, and how you used them. Did you already know the answers? Did you use any tools, and were they useful? Which tools were useful, and which were not? What worked and what did not? Do you have any useful comments about what is good for what, and what is not good for what?

Most students used ChatGPT, as is, for this question. ChatGPT was amazingly good on questions 1–4.

6. Amazingly bad

6.1 Made-up references

But ChatGPT was amazingly bad on question 6. A number of the students returned the same awful answer:

Title: “A Survey of Nearest Neighbor Search Algorithms”

Authors: Yufei Tao, Dongxiang Zhang

Link: Survey Paper (link to https://arxiv.org/abs/1904.06188)

Note that this paper does not exist, though there are a number of papers with similar titles such as Abbasifard et al. (2014). The authors can be found in Google Scholar: Zhang Dongxiang and Yufei Tao, though I was unable to find a paper that they wrote together. Worse, the link points to Amanbek et al. (2020), a completely different paper on a different topic with a different title and different authors. I believe the students used ChatGPT to find this non-existent paper. I had hoped that the students would do more fact-checking than they did, especially after having discussed hallucinations in class, but users do not do as much fact-checking as they should. Perhaps the court of public opinion needs to increase the penalties for trafficking in misinformation.

This case is similar to a recent case where a lawyer relied on A.I. He “did not comprehend” that the chat bot could lead him astray. The bot crafted a motion full of made-up case law.

As Mr. Schwartz answered the judge’s questions, the reaction in the courtroom, crammed with close to 70 people who included lawyers, law students, law clerks and professors, rippled across the benches. There were gasps, giggles and sighs… “I continued to be duped by ChatGPT. It’s embarrassing,” Mr. Schwartz said. q

The episode, which arose in an otherwise obscure lawsuit, has riveted the tech world, where there has been a growing debate about the dangers – even an existential threat to humanity – posed by artificial intelligence. It has also transfixed lawyers and judges.

In addition to computer errors (hallucinations), there were also some human errors. I’m not sure the students (and ChatGPT) understand the difference between primary literature and secondary literature. One student confused approximate nearest neighbors with shortest paths. A survey paper is not the same as a paper whose title contains the string: “survey.” That said, I am much more concerned with trafficking in computer-generated misinformation (bogus references) than human errors by well-meaning students that are making excellent progress on the material in class.

6.2 Essays that are not only wrong but lack depth and perspective in ways that could be dangerous

Several questions on the homework asked the students to write essays:

Please use ChatGPT, Google and whatever other tools might be useful to do this assignment. The point is not so much to solve the problems, but to learn how to use these tools effectively, and to discover their strengths and weaknesses.

In retrospect, I wish I had been more explicit about asking for fact-checking. One of the student essays contained the following paragraph:

During the First Opium War (1839–1842), the British government was led by the Conservative Party under Prime Minister Sir Robert Peel. The opposition, primarily the Whigs, had varying views on the war. Some opposed it on moral grounds, criticizing the ethics of trading in opium, while others were concerned about the potential impact on international relations and trade.

This paragraph includes a number of factual errors. While the dates are correct,r the Conservatives were in the opposition at the time. Peel was the Prime Minister of England under a Conservative Government, but not at that time.s

In fact, the Opium War had little to do with opium. Neither the government (Whigs) nor the opposition (Conservatives) wanted to have anything to do with the drug trade. The Whigs had just abolished slavery and considered the drug trade to be a form of slavery. The conservatives also objected to the drug trade, though for different reasons. They viewed the drug trade as bad for business (in textiles and tea). The name of the conflict, Opium Wars, comes from an editorial on March 23, 1840, in the conservative newspaper: The Times, which argued that

The British would be saddled with the massive expense of an unnecessary foreign campaign that would cost far more than the entire value of the lost opium. Platt (2019), p. 393.

The government was put in an awkward corner because, Charles Elliot, their representative in China mishandled the situation. He convinced the drug smugglers to give him their drugs in return for British IOUs, and then he handed over the drugs to the Chinese authorities for destruction. When Parliament discovered that they could not afford to make good on the IOUs, they thought they could use force to get the Chinese to pay the 2 million pounds for the lost opium.

Here is the question that I gave to the students:

1. 1. What were the Opium Wars?

2. 2. Where did the name come from?

3. 3. Summarize the conflict from multiple perspectives, including:

   1. (a) England

   2. (b) China

   3. (c) India

   4. (d) United States

   5. (e) France

4. 4. In the English parliament, which party was in power at the time? Did the party in power agree with the opposition at the time? If not, what were the two positions? Listen to this.t Can you use ChatGPT and/or Google to find evidence to support Platt’s description of the opposition to the Opium War in England? You may want to listen to much more of this YouTube video because it has answers to many of these questions. The bigger question is what will be the future of academic historians? Will technology make it easier for historians to do research? Or will technology replace historians?

I was hoping the students would listen to the YouTube video (footnote $^{20}$ ). That video explains Platt’s description of the conservative position. More seriously, I was hoping the students would appreciate the difference between ChatGPT and an academic discussion by a historian that had just published a book on this topic (Platt 2019).

Most of the essays from the students repeated output from ChatGPT more or less as is. These essays tended to include a number of factual errors, but more seriously, the essays lack depth and perspective. In Platt (2019), p. 444, Platt argued that Napoleon understood that it would be foolish for Britain to use its short-term advantage in technology to humiliate the Chinese. Eventually, the Chinese would do what they have done (become stronger). Since the 1920s, these events are referred to as the “century of humiliation” by the authorities in China.u Platt makes it clear that the current Chinese government is using this terminology to motivate its efforts to compete with the West in technologies such as artificial intelligence. When we discussed these essays in class, I tried to argue that over-simplifying the truth, and taking the Western side of the conflict, could be dangerous and could lead to a trade war, if not a shooting war.

7. Conclusions

But despite such risks, usage of ChatGPT will almost surely continue to grow, since it is so easy to use, and so (incredibly) credible. I would be more comfortable with this reality if we encouraged more usage with humans-in-the-loop, with a better characterization of when the machine can be trusted and when humans should intervene.

We have seen that LLMs (and ChatGPT) have much to offer and can be a useful tool for students. However, there are risks. Users of LLMs, including students and everyone else, should be made aware of strengths (fluency) and weaknesses (trustworthiness). Users should be expected to do their own fact-checking before trafficking in misinformation. Laziness is inexcusable. Users are responsible for what they say, whether or not it came from a chat bot.

That said, realistically, laziness is also inevitable. Chat bots are not going away. If others are conservative with the truth and with fact-checking, then we will become conservative with belief. Credibility may disappear before chat bots go away.