Polish natural language inference and factivity: An expert-based dataset and benchmarks

Abstract

Despite recent breakthroughs in Machine Learning for Natural Language Processing, the Natural Language Inference (NLI) problems still constitute a challenge. To this purpose, we contribute a new dataset that focuses exclusively on the factivity phenomenon; however, our task remains the same as other NLI tasks, that is prediction of entailment, contradiction, or neutral (ECN). In this paper, we describe the LingFeatured NLI corpus and present the results of analyses designed to characterize the factivity/non-factivity opposition in natural language. The dataset contains entirely natural language utterances in Polish and gathers 2432 verb-complement pairs and 309 unique verbs. The dataset is based on the National Corpus of Polish (NKJP) and is a representative subcorpus in regard to syntactic construction [V][że][cc]. We also present an extended version of the set (3035 sentences) consisting more sentences with internal negations. We prepared deep learning benchmarks for both sets. We found that transformer BERT-based models working on sentences obtained relatively good results ($\approx 89\%$ F1 score on base dataset). Even though better results were achieved using linguistic features ($\approx 91\%$ F1 score on base dataset), this model requires more human labor (humans in the loop) because features were prepared manually by expert linguists. BERT-based models consuming only the input sentences show that they capture most of the complexity of NLI/factivity. Complex cases in the phenomenon—for example, cases with entitlement (E) and non-factive verbs—still remain an open issue for further research.

1. Introduction

Semantics is still one of the biggest problems of Natural Language Processing. ^a It should not come as a surprise; semantic problems are also the most challenging in the field of linguistics itself (see Speaks 2021). The topic of presupposition and such relations as entailment, contradiction, and neutrality are at the core of semantic-pragmatic research (Huang 2011). For this reason, we dealt with the issue of factivity, which is one of the types of presupposition.

The subject of this study includes three phenomena occurring in the Polish language. The first of them is factivity (Kiparsky and Kiparsky 1971; Karttunen 1971b). The next phenomenon is the three relations: entailment, contradiction, and neutrality (ECN), often studied in Natural Language Inference (NLI) tasks. The third and last phenomenon is utterances with the following syntactic pattern: “[verb][że][complement clause].” The segment że corresponds to the English segments that and to. The above syntactic structure is the starting point of our research. In order to look at the phenomenon of presupposition, we collected the dataset based on the National Corpus of Polish (NKJP). The NKJP corpus is the largest Polish corpus which is genre diverse, morphosyntactically annotated and representative of contemporary Polish (Przepiórkowski et al. 2012). ^b Our dataset is a representative subcorpus of the NKJP. The analysis of our dataset allowed us to make a number of findings concerning the factivity/non-factivity opposition. We then trained models based on prepared linguistic features and text embedding BERT variants. We investigated whether the modern machine learning models handle the factivity/non-factivity opposition.

Thus, in this paper, our contributions are as follows:

• gathering a new dataset LingFeatured NLI, based on fully natural utterances from (NKJP). The dataset consists of 2432 “verb-complement” pairs, that is sentences (denoted as Theses in the following text and examples) with their clausal/complements (denoted as Hypotheses in the following text and examples). It was enriched with various linguistic features to perform inferences of the utterance relation types, that is entailment, contradiction, neutral (ECN) (see Section 3). To the best of our knowledge, it is the first such dataset in the Polish language. Additionally, all the utterances constituting the dataset were translated into English.

• creation of an extended version of a dataset by adding negation to original sentences. This dataset contains 3035 observations and is about 25% larger than the original dataset. The purpose of assembling new sentences was, in particular, to increase the number of observations belonging to class C (from 107 to 162).

• analyzing the above dataset and presenting conclusions on the phenomenon of presuposition and the ECN relations.

• building ML benchmark models (linguistic feature-based and embedding-based BERT) that predict the utterance relation type ECN (see Section 5). ^c

In the following, Section 2 describes theoretical background of the factivity phenomenon. Then, Section 3 introduces our new dataset LingFeatured NLI with a commentary on its language background, annotation process, and features. Section 4 shows the observations from the dataset. The next Section 5 shows our machine learning modeling approach and experiments. Further, in Section 6, we analyze the results and formulate findings. Finally, we summarize our work in Section 7.

2. Linguistic background

2.1. Linguistic problems

In the linguistic and philosophical literature, the topic of factivity is one of the most disputed. The work of Kiparsky and Kiparsky (1971) began the never-ending debate about presupposition and factivity in linguistics. This topic was of interest to linguists especially in the 1970s (see, e.g., Karttunen 1971b; Givón 1973; Elliott 1974; Hooper 1975; Delacruz 1976; Stalnaker, Munitz, and Unger 1977). Since the entry of the term factivity into the linguistic vocabulary in the 1970s, there have been many, often mutually exclusive, theoretical studies of this phenomenon.

Karttunen’s (2016) article with the significant title Presupposition: What went wrong? emphasized this fact. He mentioned that factivity is the research area that raised the most controversy among linguists. It should be clearly stated that no single dominant concept explaining the phenomenon of factivity has emerged so far. Nowadays, new research on this topic appears constantly (see, e.g., Giannakidou 2006; Egré, 2008; Beaver 2010; Tsohatzidis 2012; Anand and Hacquard 2014; Kastner 2015; Djärv 2019). The clearest line of conflict concerns the question in which area to situate the topic of factivity: semantics or pragmatics. There is also a dispute about discursive concepts (see, e.g., Abrusán 2016; Tonhauser 2016; Simons et al. 2017). An interesting example from our research point of view is the work of (Djärv and Bacovcin 2020). These authors argue against the claim that factivity depends on the prosodic features of the utterance, pointing out that it is a lexical rather than a discursive phenomenon.

In addition to the disputes in the field of linguistics, there is also work (see, e.g., Hazlett 2010; Hazlett 2012) of a more philosophical orientation which strikes at beliefs that are mostly considered accurate in linguistics, for example that know that is a factive verb. These works, however, have been met with a distinctly polemical response (see, e.g., Turri 2011; Tsohatzidis 2012).

In summary, the first problem to note is the clear differences in theoretical discussions of the phenomenon of factivity-based presupposition. These take place not only in the older literature, but also in the contemporary one.

Theoretical differences are related to another issue, namely the widespread disagreement about which verbs are factive and which are not. A good example is a verb regret that, which, depending on the author, is factive or not or presents a different type of factivity from the paradigmatic in the class of factive expressions verb know that. ^d

The explosion of work on presupposition in the 1970s and the multiplicity of theoretical concepts resulted in the uncontrolled growth of terminological proposals and changes in the meaning of terms already used. The term presupposition has been ambiguous since the 1970s, and this state of matters persists today. Terms such as factivity, presupposition, modality, or implicature are indicated as typical examples of ambiguous expressions. Problems of terminology are the third problem to be highlighted in the work on factivity. It is important to note the disturbing phenomenon of transferring terminological issues to the NLI. Reporting studies analogous to ours will bring attention to this difficulty.

A final point to note is the lack of linguistic papers that provide a fairly exhaustive list of factive, non-factive, veridical, etc. expressions. ^e There is also a lack of comparative work between languages in general. This kind of research is only available for individual, selective expressions (see, e.g., Özyıldız 2017; Hanink and Bochnak 2017; Jarrah 2019; Dahlman and van de Weijer 2019).

2.2. Key terms

We, therefore, chose to put more emphasis on conceptual issues. The concepts most important to this study will now be presented, primarily factive presupposition.

2.2.1. Entailment, contradiction, neutral

Let’s start by introducing an understanding of the basic semantic relations:

• Entailment: H must be true if T is true;

• Contradiction: H must be false, if T is true;

• Neutral: H may be false or true if T is true (Chierchia and McConnell-Ginet 2000).

where T (Thesis) is an utterance and H (Hypothesis) is an item of information. In our dataset consisting of “verb-complement” pairs, H is a clausal/complement of a sentence T.

2.2.2. Information

The information we are interested in is that transmitted by means of spoken sentences, which are utterances. The mechanisms involved in this transmission may be either of a purely semantic (lexical consequences) or pragmatic nature (conversational/scalar implicatures) (Grice 1975; Sauerland 2004). Three examples of information are shown below.

<T, H> Example 1. (Entailment)

T : Only then did they realize that they had been cut off from their only way out.

H : They had been cut off from their only way out.

<T, H> Example 2. (Contradiction)

T : He was wrong to say that it was just a normal room.

H : It was just a normal room.

<T, H> Example 3. (Neutral)

T : Statists believe that people can be demanding towards the state.

H : People can be demanding towards the state.

In Example 1, the entailment is lexical in nature because it is founded on the factive verb realize that.

The contradiction in Example 2 is also lexical in nature. In Example 3, on the other hand, we have the neutrality: there is nothing in T’s utterance that guarantees either a lexical entailment or a contradiction, and there are no pragmatic mechanisms that would fund either of these two relations.

2.2.3. Negation

We take into account in our dataset the occurrence of negation, specifically internal negation. We distinguish it from the so-called external negation, which is not relevant to the phenomenon of factivity. The examples of these two types of negation can be found below.

<T, H> Example 4. (Entailment)

T : He did not know that he had opened the door.

H : He had opened the door.

<T, H> Example 5. (Neutral)

T : It is not the case that he knew he had opened the door.

H : He had opened the door.

In Example 4, internal negation was used, and in Example 5, external negation. The utterance in Example 4 implies H, whereas in Example 5 does not imply H. The source of this difference is the different types of negation used. The external negation applied to the utterance 5 makes it possible for all the meaning components of the verb to be within its range, so H does not follow from Example 5.

2.2.4. Presupposition

We understand the term presupposition as follows:

DEF. If the utterance T entails H and H is insensitive to internal negation in T, whether currently occurring or potential, then H is the presupposition of the utterance T.

We named all such information as presuppositions, regardless of their detailed nature. Thus, presuppositions can have both semantic and non-semantic grounds. In the literature, one can find lists of expressions and constructions that are classically called presupposition triggers (Levinson 1983). Below is an example illustrating a presupposition based on a factive verb.

<T, H> Example 6.

T : [PL] Ona wie/nie wie, że należy podać hasło.

T : [ENG] She knows/does not know that a password must be provided.

$\gg$ H [PL] Należy podać hasło.

$\gg$ H [ENG] A password must be provided.

Information H in Example 6 (with and without internal negation) is the presupposition of the utterance because this information is insensitive to internal negation in T. ^f Presupposition H is guaranteed by the factivity property of the verb wiedzieć, że/know that.

The relation between a semantic entailment and a semantic presupposition is shown in Example 7. ^g

<T, H> Example 7. T : The driver managed to open the door before the car sank.

$\models{/}{\gg }$ H a The driver managed to open the door before the car sank.

$\models{\gg }$ H b The driver tried to open the door before the car sank.

$\models{/}{\gg }$ H c The driver opened the door before the car sank.

$\models{\gg }$ H d The car sank.

The utterance in Example 7 semantically entails Ha-Hd because the definition of this type of entailment is fulfilled: if T is true, then H must be true. This is because in the case of the implicative verb manage that (Karttunen 1971a), some of its meaningful components are not within the scope of internal negation, for example, the information that someone tried to do something. Apart from that, the information Hb and Hd is also presuppositions of the utterance. They meet the defining conditions of a presupposition: they are insensitive to—in this case, potential—internal negation. In other words, we treat presupposition as a certain subtype of entailment.

Analyzing the difference between semantic presupposition and non-semantic presupposition, consider the following utterances in Examples 8 and 9.

<T, H> Example 8.

T : She was not told that he was already married.

$\gg$ H : He is already married.

<T, H> Example 9.

T : She didn’t know that he was already married.

$\gg$ H : He is already married.

Both utterances in Examples 8 and 9 entail information H. Moreover, in both cases, H is not within the scope of internal negation, so the information H is their presupposition. However, the foundations of these presuppositions are radically different. In Example 8, H is guaranteed by the appropriate prosodic structure of the utterance, whereas in Example 9, the presupposition H has a semantic grounding—it is guaranteed because of the factive verb know that. ^{h , i} In other words, the entailment in Example 8 is not lexical, and in Example 8, it is.

2.2.5. Factivity

It is worth noting the occurrence of the following four terms in the NLI literature: factivity, event factuality, veridicality, and speaker commitment. These terms, unfortunately, are sometimes understood differently depending on the work in question. In the presented study, we use only the term factivity, which is understood as an element of the meaning of particular lexical units. Such phenomena as speaker “degrees of certainty” are entirely outside the scope of the research presented here. We assume that presupposition founded on factivity takes place independently of communicative intentions; it may or may not occur: there are no states in between. For comparison, let’s look at how the term “event factuality” is understood by the authors of the FactBank dataset:

“( $\ldots$ ) we define event factuality as the level of information expressing the commitment of relevant sources towards the factual nature of events mentioned in discourse. Events are couched in terms of a veridicality axis that ranges from truly factual to counterfactual, passing through a spectrum of degrees of certainty (Saurí and Pustejovsky 2009: p. 231).”

In another paper, the same pair of authors provide the following definition:

“Event factuality (or factivity) is understood here as the level of information expressing the factual nature of eventualities mentioned in text. That is, expressing whether they correspond to a fact in the world ( $\ldots$ ) (Saurí and Pustejovsky 2012: p. 263).”

It seems that the above two explanations of the term event factuality are significantly different. They are also composed of other terms that require a separate explanation, for example discourse, veridicality, spectrum of degrees of certainty, and level of information.

Note that the second quoted fragment also defines factivity; the authors apparently put an equality mark between “event factuality” and “factivity” Reading the specifications of the FactBank corpus and the instructions for the annotators leads, in turn, to the conclusion that Saurí and Pustejovsky understand factivity as (a) a subset of factuality, (b) in a “classical way,” as a property of certain lexical entities (Saurí and Pustejovsky 2009).

In the presented approach, factivity is understood precisely as a semantic feature of specific lexical units. In other words, it is an element of their meaning. According to the terminology used in the literature, we will say that factive verbs are presupposition triggers. We understand the term factive verb as follows:

DEF. A verb V is an element of a set of factive units if and only if the negation-insensitive part of the meaning of V includes the information p.

The information p in the utterances with the structure we analyzed “[verb][że][complement clause]” is a complement clause. Using the category of verb signature, it can be said that factive verbs belong to the category (+/+). ^j Examples 10 and 11 illustrate presuppositions based on the meaning of the factivity verb be aware that.

<T, H> Example 10.

T : She was not aware that she had made a fatal mistake.

$\models$ H : She made a fatal mistake.

<T, H> Example 11.

T : She was aware she had made a fatal mistake.

$\models$ H : She made a fatal mistake.

Information H follows semantically from both Example 10 and its modification as Example 11. This information is beyond the scope of internal negation and is, therefore, its presupposition. The foundation of this presupposition is the presupposition trigger in the form of the verb be aware that. Neither the common knowledge of the speakers nor the prosodic properties of the utterances are irrelevant to the fact that H in the above examples are presuppositions.

In summary, presuppositions can be either lexical or pragmatic in nature. What they have in common is that they are insensitive to internal negation in T. We treat presuppositions founded on factive verbs as lexical presuppositions. If information H is a lexical presupposition of an utterance T, then T entails H. These relations are independent of the speaker’s communicative intention; it means that the speaker may presuppose something unconsciously or against his own communicative intention.

3. Our dataset

The historical background of this paper is gathered from (Cooper et al. 1996; Dagan, Glickman, and Magnini, 2005, 2013). These works established a certain pattern of construction of linguistic material, consisting in pairs of sentences: thesis and hypothesis ( $\lt$ T, H $\gt$ ).

3.1. Input material sources and extraction

The material basis of our dataset is the National Corpus of Polish Language (NKJP) (Przepiórkowski et al. 2012). We used a subset of NKJP in the form of the PCC Polish Coreference Corpus (Ogrodniczuk et al. 2014), which contains randomly selected fragments from NKJP and constitutes its representative subcorpus. We did not add any prepared utterances—our dataset consists only of original utterances found in the PCC. Moreover, the selected utterances have not been modified in any way—we did not correct typos, syntactic errors, or other difficulties and issues. Thus, the language content remained entirely natural, not improved artificially.

We automatically annotated with Discann all occurrences of the phrase że (that | to) as in Example 12. ^k

<T, H> Example 12.

T : [PL] Przez lornetkę obserwuję, że zalane zostały żerowiska bobrów.

T : [ENG] I can see through binoculars that the beaver feeding grounds have been flooded.

From more than 3500 utterances, we left only those that satisfied the following pattern: “[verb] [że] [complement clause].” It required a manual review of the entire dataset. Thus, we obtained 2320 utterances that constitute the empirical basis of our dataset.

3.1.1. <T, H> pairs

In this work, the original source of T utterances is NKJP, and H is complement clauses of T.

<T, H> Example 13.

T : [PL] Myśleli, że zwierzęta gryza się ze soba. [NKJP]

T : [ENG] They thought the animals were biting each other.

H : [PL] Zwierzęta gryza się ze soba.

H : [ENG] The animals were biting each other.

From 2320 utterances, we created 2432 $\lt$ T, H $\gt$ pairs (309 of unique main verbs). We have not paired the sentences randomly. In any particular $\lt$ T, H $\gt$ pair, H is always the complement clause of the T utterance from the pair. In some utterances, the verb introduced more than one complement—in each such situation, we created a separate $\lt$ T, H $\gt$ . For each sentence, we extracted a complement clause manually. Our manual work included, for example, removing fragments that were not in the range of the verb—see Example 14.

<T, H> Example 14.

T : He said I am—I guess—beautiful.

H : I am—I guess—beautiful.

Annotators $\lt$ T, H $\gt$ are the core of our dataset. We assigned specific properties (i.e., linguistic features) to each pair. In the following, we presented these linguistic features with their brief characteristics.

3.2. Linguistic features

3.2.1. Entailment, contradiction, and neutral (ECN)

Each of $\lt$ T, H $\gt$ pairs was assigned one of three relations: entailment, contradiction, and neutral. The occurrence of features is not balanced, that is entailment class states $33.88\%$ of the dataset, contradiction— $4.40\%$ , neutral— $61.72\%$ . The lack of balance is a characteristic of all linguistic features in the dataset. Due to the fact that our dataset is a representative subcorpus of the NKJP, we assume that the same distribution of relations is found in the NKJP itself.

3.2.2. Verb

In each utterance, the experienced linguist manually identified the verb that introduced the H sentence.

Despite appearances, this was not a trivial task. Often identifying a given lexical unit required deep thought and verification of specific delimitation hypotheses. We assume that an adequate delimitation of language entities is a precondition and also a necessary condition for further linguistic research. The difficulty in determining the form of a linguistic unit is primarily due to the difficulty of establishing its argument positions. However, there are no universal linguistic tests to distinguish between an argument and an adjuncts. In the case of verbs, it is not uncommon for two (or more) different lexical units to be hidden under a graphically equal form. For example, we distinguish between czuć, że/feel that, which is epistemic and non-factive, and czuć, że/feel that, which is purely perceptual and factive (Danielewiczowa 2002) (see Examples 15, 16, and 17). In addition to these two verbs, we also identified the verb CZUĆ, że which is epistemic, factive, and must take on a non-contradictory sentence stress.

<T, H> Example 15. (czuć $_{1}$ , epistemic, non-factive)

T : [PL] Czuł, że już nigdy jej nie zobaczy.

T : [ENG] He felt that he would never see it again.

<T, H> Example 16. (czuć $_{2}$ , perceptual, factive)

T : [PL] Czuł, że po jego ręce chodzi pajak.

T : [ENG] He felt a spider walking on his hand.

<T, H> Example 17. (czuć $_{3}$ , epistemic, factive)

T : [PL] CZUŁ, że patrza na niego, choć stał odwrócony plecami.

T : [ENG] He FELT they were looking at him, although he stood with his back turned.

Another difficulty can be caused, for example, by verbs with the shape wspominać/mention. We distinguished between wspominać, że $_{1}$ (non-factive, speech, non-epistemic) and wspominać, że $_{2}$ (factive, speech, epistemic):

<T, H> Example 18. (wspominać, że $_{1}$ )

T : [PL] W każdym razie ja kontaktowałem się tylko z nim i nigdy nie wspominał, że ktoś inny bierze udział w transakcji.

T : [ENG] In any case, I only contacted him and he never mentioned that anyone else was participating in the transaction.

<T, H> Example 19. (wspominać, że $_{2}$ )

T : [PL] Przyjechał na Ślask z Tomaszowa Mazowieckiego w 1967 roku. Wspomina, że wtedy praca była wszędzie.

T : [ENG] He came to Silesia from Tomaszów Mazowiecki in 1967. He recalls that at that time work was everywhere.

We identified a total of 309 verbs. A list of these verbs can be found in a publicly accessible repository. ^l

3.2.3. Verb type

We assigned one of two values: factive/non-factive to all verbs. From the linguistic side, it was the most difficult part. This task was done by a linguist writing his Ph.D. thesis on the factivity phenomenon (Ziembicki 2022). The list was checked with the thesis supervisor, and in most cases, these people agreed with each other, but not in all cases. Finally, 81 verbs were marked as factive and 230 as non-factive.

3.2.4. Internal negation

For each utterance, we marked whether it contains an internal negation. About 95% of utterances did not contain explicit negation words, and almost 5% of sentences did.

3.2.5. Verb semantic class

We have distinguished four semantic classes of verbs: epistemic (myśleć, że/think that), speech (dodać, że/add that), emotive (żałować, że/regret that), and perceptual (dostrzegać, że/perceive that). Most verbs were hybrid objects, for example epistemic speech. The class name was given due to the given dominant semantic component. If the verb did not fit into any of the above classes, the value other was given.

3.2.6. Grammatical tense

In each utterance, we marked the grammatical tense of the main verb and the complement H. Note that the tenses in Polish much differ from tenses in English. Polish has only three tenses: past, present, and future. It is worth adding that Polish verbs have an aspect: imperfective and perfective. Perfective verbs reflect completion of an action. In contrast, imperfective verbs focus on the fact that an action is being performed.

3.2.7. Utterance type

We labeled the type of utterance as indicative, imperative, counterfactual, performative, interrogative, or conditional. Below are examples of this type of speech.

(indicative) She knew it was raining.

(imperative) Know that it is raining.

(counterfactual) If I hadn’t looked in the window, I wouldn’t have known it was raining.

(performative) I give you the name “falling rain.”

(interrogative) Does it rain?

(conditional) If it rains, the walkways are wet.

All T utterances have been translated into English, see Appendix B.

3.2.8. Extended LingFeatured NLI dataset

For additional experiments, we added new sentences with or without internal negation. The procedure for creating new sentences was as follows: (1) in the original sentences with negation it was manually removed, (2) in sentences without negation it was manually added, (3) necessary corrections were made if the procedure of adding/removing negation caused semantic/syntax defects. For many utterances, steps (1)–(2) resulted in such a distortion of the original utterance that it was unsuitable for the dataset. The expanded version of the dataset contains 603 new utterances.

3.3. Annotation

3.3.1. Expert annotation

Among linguistic features assigned to pairs $\lt$ T, H $\gt$ the most difficult and essential to identify were factivity/non-factivity and logic relations ECN. Whether a verb is factive was determined by two linguists who are professionally involved in natural language semantics. They achieved more than 90% agreement, with most doubts arising when analyzing verbs of speaking, for example wytłumaczyć, że/explain that. The final decisions on identifying which verb is factive were made by a specialist in the field.

Semantic relations ECN were established in two stages. The dataset itself consists of single-sentence contexts, but the annotation involved an analysis of the utterances in context—annotators checked the relevant utterances in the NKJP. The first stage was annotated by two linguists, including one who academically deals with the issue of verb semantics. They achieved 70% agreement in the annotation in terms of inter-annotator agreement given by Cohen’s Kappa. Significant discrepancies can be observed for relations of contradiction, as opposed to entailment. Then, those debatable pairs were discussed with a third linguist, a professor specializing in natural language semantics. The result of these consultations was the final annotation—the gold standard.

3.3.2. Experiment with non-professional annotators

We checked how the gold standard created in this way would differ from the annotations of non-professional linguists—a group of annotators who are not involved professionally in formal linguistics but have a high linguistic competence. The criteria for selecting annotators were the level and type of education and a pre-test. Thus, the four selected annotators were as follows: (A1) cognitive science student (3rd year), (A2) and (A3) annotators with master’s degree in Polish Studies, (A4) Ph.D. in linguistics (see Table 1).

Table 1. Inter-annotator agreement given by Cohen’s Kappa (α = 0.05). Ex—an expert who made the gold standard, A1–A4—non-expert linguists

Each annotator was given the same set of $\lt$ T,H $\gt$ pairs from the set (20% of the total set). The task of each annotator was to note the relation between T and H. There were four labels to choose from entailment, contradiction, neutral, and “?.” ^m The annotation instructions included simplified definitions of key labels—as we presented in Section 2.2.

Annotators were asked to choose “?,” if (1) they could not indicate what the relation was, or (2) they thought the sentence was meaningless, or (3) they encountered another problem that made it impossible for them to choose any of the other three labels. Especially important, from our point of view, is the situation (1). The idea was to reserve it for $\lt$ T, H $\gt$ pairs whose semantic relation is dependent on prosodic features—like accent, which determines focus and topic (see Partee 2014). Let’s look at an Example 20:

<T, H> Example 20.

T : Let’s not say [that] these projects are supposed to end in a constitutional change.

H : These projects are supposed to end in a constitutional change.

There are two possible situations in Example 20: (a) the sender wants to hide the information from H (label: entailment) and (b) the sender does not want to say H because, for example he wants to make sure that this is true first (label: neutral).

Let us specify that the way in which “?” was used was, however, different for the creation of the Gold standard, which was not based on the work of annotators but of a linguist specializing in the topic of factivity: the “?” sign applied only to condition (1) and it was used only six times. Two of these are given below:

<T, H> Example 21.

[PL] No wie pani $\ldots$ jemy sobie czekoladę, a pani mówi, że będzie nowym naczelnym $\ldots$ ja w śmiech $\ldots$

[ENG] Well, you know $\ldots$ we’re eating chocolate, and you say you’re going to be the new editor $\ldots$ I’m laughing $\ldots$

<T, H> Example 22.

[PL] ale też jest fajnie pokazane właśnie to że ona nie jest dojrzała jakby no nie zupełnie

[ENG] but it’s nicely shown, too, that in fact she is not really mature, not entirely so to speak

In the first example, neither the utterance itself nor the context (several sentences before and after) make it possible to determine which ECN relation is occurring. In the second example, an analogous situation takes place, and, in addition, it is not clear exactly which main verb introduces the sentence complement: it is not known whether it is factive.

Inter-annotator agreement with the dataset gold standard was calculated using R implementation of Cohen’s Kappa. It was in the range of 61%–65%, excluding the worst annotator whose Kappa was below 52% with all other annotators. ⁿ Table 1 summarizes the inter-annotator agreement among four non-expert linguists and one of the experts preparing the dataset. The conclusions of the annotation performed and described above are provided in Section 6.1.

3.4. Related datasets

We will now review some of the most recent and similar works. Table 2 collates several analogous datasets and highlights the tasks solved in the publications.

Table 2. Datasets on NLI and their linguistic features

3.4.1. Data source

The linguistic material in our dataset was extracted from NKJP. It thus belongs to the class of datasets whose source is a linguistic corpus, such as the Corpus of Contemporary American English or the British National Corpus. A different approach to the data source is presented by for example (Jeretic et al. 2020), in which sentences are semi-automatically generated. In contrast to datasets such as (Parrish et al. 2021), we did not make any changes to the source material. The only changes we have made relate to the extended version of the dataset (see Section 3.2.8).

3.4.2. Number of observations

LingFeatured NLI has a similar number of observations (sentences/utterances) to (de Marneffe, Simons, and Tonhauser 2019; Ross and Pavlick 2019a; Parrish et al. 2021) and (Minard et al., 2016b). Compared to these datasets, the LingFeatured NLI stands out for the number of lexical units (main verbs). For example, the CommitmentBank contains 48 different clause-embedding predicates, whereas our dataset contains 309 main verbs. Because the LingFeatured NLI is a random subcorpus of the NKJP, the main verbs have different numbers of examples of use (see Table 3).

Table 3. Factive verbs in CommitmentBank and their Polish equivalents in LingFeatures NLI

As for factive verbs, we noted 82 entities of this type in our dataset. In comparison, CommitmentBank notes only 11 factive verbs (see also Table 3).

3.4.3. Annotation methodology

An important difference between our dataset and some others is the lack of use of the Likert scale. This scale is used, for example, in (Ross and Pavlick 2019a). ^o The central term of this work is veridicality, which is understood as follows: “A context is veridical when the propositions it contains are taken to be true, even if not explicitly asserted.” (Ross and Pavlick 2019a, p. 2230). As can be seen, the quoted definition also includes situations in which the entailement is guaranteed by factive verbs. The authors pose the following research question: “whether neural models of natural language learn to make inferences about veridicality consistent with those made by humans?” This is a very different question from the one posed in this paper. Ross and Pavlick used Likert scales to conduct annotations employing unqualified annotators (Ross and Pavlick 2019a). They then checked the extent to which the models’ predictions coincide with the human annotations obtained. Unlike these authors, we do not use any scales in annotation, and the object of the models we train is to predict real semantic relations, not those that occur as judged by humans. What we mean is that the task of the model is to predict the relations that objectively occur in reality, not those that occur in the judgment of the annotators. Indeed, at the operational level, these two different tasks merge, because the human being is ultimately behind the identification of the reality relations. For this reason, it does not use a Likert scale and the gold standard is created by specialists in lexical semantics.

3.4.4. Number of presupposition types and verb types

Many datasets deliberately include several linguistic phenomena, for example several types of presupposition, such as (Parrish et al. 2021) or distinguishes many different signatures of main verbs, for example (Ross and Pavlick 2019b). ^p A similar approach is in (Rudinger, White, and Van Durme 2018), that is factive verbs are one of several types of expressions that are of interest. Such a decision is dictated by the deliberate action of the corpus authors. Our set is somewhat different in this respect. Our linguistic interests are focused on one phenomenon of presupposition—factivity understood as a lexical phenomenon. In order to take a closer look at it in Polish, we decided to select a random sample of a specific syntactic construction, namely the previously described [V][że/that|to][compound sentence]. The result is linguistic material in which both the lexical and non-lexical presupposition of interest occurs. The latter occurs when the verb is neither factive nor veridical (e.g., mówić/tell), and the truth of the subordinate sentence takes place anyway because it is guaranteed by mechanisms of pragmatic nature. This allowed us to observe, in particular, the proportions of occurrence of different types of feature sets, primarily $\lt$ factive verb, E $\gt$ , $\lt$ non-factive verb, E $\gt$ , $\lt$ factive verb, N $\gt$ , and $\lt$ non-factive verb, N $\gt$ . In other words, LingFeatured NLI at the annotation level contains different sets of linguistic features with varying frequency of occurrence and this constitutes its annotative core.

We are aware that the number of verb distinctions is sometimes significantly higher in similar papers and our dataset could also include them. The decision to use only a binary distinction (factive vs. non-factive) is dictated by several interrelated considerations. First, there are no lists of Polish verbs with signatures assigned by linguists. Secondly, the preparation of such a list of even several dozen verbs is a highly specialized task. It may be added that there are still disputes in the literature about the status of some high-frequency verbs, for example regret that. Third, we are interested in the real features of lexical units, and not in the “textual” ones, that is those developed by non-specialist annotators, using the committee method. The development of signatures by unqualified annotators would be pointless with regard to the research questions posed. The type of linguistics used in this work is formal linguistics, which investigates the real features of a language, unlike survey linguistics, which collects the intuitions of speakers of a language. ^q

4. Implications from our dataset

The dataset created allows a number of observations to be made. We will now discuss the most important of these from the point of view of the issue of factivity/non-factivity opposition and ECN relations.

4.1. Frequency of ECN relations

As the LingFeatured NLI Dataset is a representative subcorpus of the NKJP, this leads directly to the conclusion that in the syntactic construction under study ([V][że/that|to][cc]), the NKJP contains significant differences in the frequency of occurrence of ECN relations (see Table 4). Particularly striking is the low occurrence of contradictions (4.4%). A preliminary conclusion could therefore be that in the syntactic construction under study classes Entailment and Neutral are most common in Polish. In various research, national corpora are used to calculate the frequency of words or collocations in language (see, e.g., Grant 2005). NKJP is the largest representative dataset for Polish which contains texts from various sources to reflect the use of the language. To the best of our knowledge, currently this is the most reliable way to verify the frequency of analyzed semantic relations.

Table 4. Distributions of features in LingFeatured NLI and LingFeatured NLI Extended dataset in Polish version

4.2. Factivity/non-factivity and ECN relations

Firstly, the distribution of features in the dataset indicates that, in the vast majority of cases, factive verbs go with an entailment relation (24.4% of our dataset), and non-factive verbs with a neutral relation (61.1%)—see Table 5. Other utterances, that is the pairs $\lt$ T, H $\gt$ , in which, for example, despite a factive verb, there is neutral, or despite a non-factive verb, there is entailment, constitute a narrow subset of the dataset (14.5% – 352 utterances in the dataset). Table 6 contains examples of such pairs. In the experiments carried out these kinds of $\lt$ T, H $\gt$ pairs will pose the biggest problem for humans and models—the best model accuracy of 62.87% (see Table 9 and Sections 3.3 and 6).

Table 5. Contingency table consisting of the frequency distribution of two variables

Table 6. Hard utterances in our dataset

Let’s recap—in 85.5% of the pairs of the whole dataset, entailment co-occurs with a factive verb or the neutrality co-occurs with a non-factive verb.

Second, if the verb was factive, then the entailment relation occurred in 97.70%. And if the verb was non-factive, the neutrality occurred in 81.50%. It means that pairs of features $\lt$ factivity, entailment $\gt$ , and $\lt$ non-factivity, neutrality $\gt$ very often co-occur with each other, especially the first pair. This means that such phenomena as cancelation and suspension of presuppositions are marginal in our dataset. ^r Below are examples of such utterances:

<T, H> Example 23.

[PL] Skad wiesz, że kupił Gosia?

[ENG] How do you know he bought Gosia?

<T, H> Example 24.

[PL] Mniejsza o incognito, może nie zorientowaliby się, że to ja.

[ENG] Never mind the incognito, maybe they wouldn’t know it was me.

<T, H> Example 25.

[PL] W polu Kryteria: kolumny Imie wpisz w cudzysłowie imię osoby, o której wiesz, że i tak nigdy nie odda pożyczonych filmów (rysunek 4.20).

[ENG] In the field Criteria: column Name, enter in inverted commas the name of a person about whom you know that they will never return borrowed films (Figure 4.20).

4.3. Frequency of verbs

In the dataset, significant differences can be observed regarding the frequency of occurrence of particular main verbs and the coverage by these verbs of the entire dataset (see Figure 1 and Table 4). According to our dataset, only 10 factive verbs with the highest frequency account for the occurrence of 60% of all occurrences of such expressions, and 10 non-factive verbs with the highest frequency account for nearly 45% of all occurrences of non-factive verbs (see Figure 1).

Figure 1. Relationship between the number of the most frequent verbs and the coverage of dataset. Left: The analysis of factive subsample. Right: The analysis of non-factive subsample.

5. Machine learning modelling: experiments and results

We used the dataset described above (Base and Extended versions) for experiments with ML models. The models we built aim to simulate human cognitive abilities. The models trained on our dataset were expected to reflect high competence—comparable to that of human experts—in recognizing the relations of entailment, contradiction, and neutral (ECN) between an utterance T and its complement H. We trained seven models:

1. (1) Random Forest with an input of the prepared linguistic features (LF)—Random Forest (LF), ^s

2. (2) fine-tuned HerBERT-based models for only main verbs in sentences as inputs—HerBERT (verb),

3. (3) model (2) with input extended with linguistic features listed in Table 12—HerBERT (verb+LF),

4. (4) fine-tuned HerBERT-based model for the whole input utterance T—HerBERT (sentence),

5. (5) model (3) with input extended with linguistic features listed in Table 12—HerBERT (sentence+LF),

6. (6) fine-tuned PolBERT-based model for only main verbs in sentences as inputs—PolBERT (verb),

7. (7) fine-tuned PolBERT-based model for the whole input utterance T—PolBERT (sentence).

We employed HerBERT (Mroczkowski et al. 2021) and PolBERT (Kłeczek 2020) models instead of BERT (Devlin et al. 2019), because they are trained explicitly for Polish. Regarding verb models, the input to the model is text consisting only of a main verb. Each model was trained using 10-fold cross-validation with stratification in order to avoid selection bias and estimate standard deviation. Due to the fact that class C is sparse, selecting cross-validation allows us to obtain more reliable results. Each fold has the same proportion of observations with a given labels due to stratification.

In addition, we have prepared a rule-based baseline model from the relations shown in Table 5 that works as follows.

• if verb is factive, then assign entailment (E);

• if verb is non-factive, than assign neutral (N).

Table 7 shows the models’ results achieved on the first-seen data (data unknown for a model). The values in the table represent mean and standard deviation of metrics, respectively. F1 score in binary setting is harmonic mean between model precision and recall. In multiclass situation, it is calculated per class and overall metric is average score. Here F1 score was calculated as weighted F1, due to large imbalance between classes. Weighted F1 score is calculated by taking the mean of all per-class F1 scores while considering each class’s support. ^t

Table 7. Classification results of entailment (E), contradiction (C), and neutral (N). Linguistic features comprise: verb, grammatical tense of verb, occurrence of internal negation, grammatical tense of complement clause, utterance type, verb semantic class, verb type (factive/non-factive). F1 score depicts weighted F1 score. LF denotes linguistic features. The results for baseline are calculated over the entire input dataset

The parameters of models and their training process are gathered in Table 8. The precise results of Random Forest for different feature sets and the feature importance plots are given in Table 12 and in Figure 2. Table 9 summarizes the results of our models for the most characteristic sub-classes in our dataset: entailment and factive verbs, neutral and non-factive verbs, and the other cases.

Table 8. Model and training parameters

Table 9. Results in the most characteristic subsets in our test base dataset: entailment and factive, neutral, and non-factive, and the other cases. Values present model accuracy [%]. LF denotes linguistic features

Figure 2. Impurity-based feature importance of feature-based Random Forest trained using base LingFeatured NLI. The chart shows the English equivalents of Polish verbs: know/wiedzieć; pretend/udawać; think/myśleć; turn out/okazać się; admit/przyznać; it is known/wiadomo; remember/pamiętać.

Table 10 presents models’ results on extended dataset. Training procedure is the same as previously. We also show the analysis of hard cases in Table 11.

Table 10. Classification results of entailment (E), contradiction (C), and neutral (N) using the extended LingFeatured NLI dataset. Linguistic features comprise: verb, grammatical tense of verb, occurrence of internal negation, grammatical tense of complement clause, utterance type, verb semantic class, verb type (factive/non-factive). F1 score depicts weighted F1 score. LF denotes linguistic features. The results for baseline are calculated over the entire input dataset

Table 11. Results in the most characteristic subsets in our test extended LingFeatured NLI dataset: entailment and factive, neutral, and non-factive, and the other cases. Values present model accuracy [%]. LF denotes linguistic features

The overall results indeed show very high performance of models in Entailment and Neutral classes: Entailment—87% to 92% in F1 score, and Neutral—91% to 93.5% in F1 score (see Table 7). Similar relation can be observed on extended dataset: F1 score of Entailments varies from 87% to 92% and from 89% to 92% for Neutral class. More precisely, the models achieved very high results (93% up to 100% in accuracy) for the sentence pairs containing lexical relations—subsets entailment and factive, neutral, and non-factive in Table 9. However, it gained very low metrics (47% up to 62.9%) on those with a pragmatic foundation, which are drastically more difficult—subset “Other” in Table 9. The results for the extended version of the dataset are similar and can be seen in detail in Table 11. Moreover, in both cases, the models’ performance is much better than the rule-based baseline.

Additionally, the overall ML modeling results show that HerBERT sentence embedding-based models are at a much higher level than non-expert linguists. Nevertheless, they did not achieve the results of professional linguists (by whom our gold standard was annotated). Feature-based models achieve slightly better results (mean accuracy across folds of 91.32%), although not for the contradiction relation (mean accuracy of 39.89%). However, the weak result for this relation is due to a small representation in the dataset (only 4.4% cases, see Tables 4 and 5). Moreover, the variance of the ML results is between 0.08% up to 2.5% across different folds in cross-validation for the overall results and the easier classes (E,N). However, the variance for C class (contradiction) is very high—from 11.93 even up to 31.97%. Once again, this occurs because the phenomenon appears very rarely, in the dataset we have only a few cases (i.e., 107 cases).

We prepared the gold standard dataset based on annotations made by experts in formal semantics (see Section 3). Additionally, we compared the model results with non-specialist annotator as an experiment. Note that the performance of models trained on such a training set achieves significantly higher results than those obtained by non-experts annotators in the test annotation performed.

Further, models with verb embedding vs the entire sentence representations are better. However, they require manual extraction of the main verb in the utterance because sometimes it is not apparent. The examples of such a difficult extraction are as follows.

<T, H> Example 26. (Neutral)

T : [PL] Czuł, że inni zbliżali się do niego, ale nie był tego pewien.

T : [ENG] He felt others getting closer to him, but he wasn’t sure.

<T, H> Example 27. (Entailment)

T : [PL] Czuł, że maszynistka nie spuszcza zeń oczu.

T : [ENG] He felt that the typist wasn’t taking her eyes off him. $\ldots$

Consideration of the difference between the main verbs in the above examples requires attention to suprasegmental differences. In Example 26, the non-factive verb czuć, że is used. In contrast, in Example 27, a different verb is used, namely the factive CZUĆ, że, which necessarily takes on the main sentence stress (see, e.g., Danielewiczowa 2002). Note that from the two verbs above, we should also distinguish the factive perceptual verb czuć, że.

<T, H> Example 28. (Neutral)

T : [PL] Lawirował na granicy prawdy, lecz przez cały czas czułem, że kłamie.

T : [ENG] He was teetering on the edge of the truth, but at all times I felt he is lying.

H : [PL] Kłamie.

H : [ENG] He was lying.

<T, H> Example 29. (Entailment)

T : [PL] – Dziękuję – odpowiedział, czujac, że policzek zaczyna go boleć.

T : [ENG] – “Thank you,” he replied, feeling his cheek starts to hurt.

H : [PL] Policzek zaczyna go boleć.

H : [ENG] His cheek starts to hurt.

In Example 28, the epistemic verb is used, while in Example 29, the main predicate is the perceptual verb. The former is non-factive and the latter is factive.

Further findings are that the models with inputs comprising text embeddings and linguistic features achieved slightly better results than those with only text inputs. The only exception to this rule is HerBERT with sentence input trained and tested on extended dataset, where the addition of linguistic slightly worsened the results (from 90.59% to 89.55% of F1 score). Besides, we can see that—in our base feature Random Forest model (1)—some features make the most significant contribution to our model, that is if the verb is factive or non-factive (see Table 12 and Figure 2). However, the indication of verb tense (see Figure 2) as relatively important for our ML tasks, that is ECN classification, appears to be misleading in the light of linguistic data and requires further analysis. It seems that we are dealing here with spurious correlations rather than with lexical rules of language connecting the verb tense with ECN relations. Deeper linguistic analysis would be advisable here, however, because the relation between the grammatical tense of the verb and the ECN relations may be the result of pragmatic rules and prosodic properties of the utterances. We hypothesize that these are spurious correlations in our dataset because, indeed, present or past tense co-occur more often with a particular class of ECN in our dataset. ^u

Table 12. Features in classification of entailment, contradiction and neutral. Random forest results with inputs of different sets of features on base LingFeatured NLI

6. Analysis of results and discussion

6.1. Expert and non-expert annotation

Inter-annotator agreement of non-expert annotators with the linguists’ preparing the dataset gold standard (Kappa of 61%–65%) indicates that the task is very specialized. We did not find patterns of errors made by the annotators. If the goal of human annotation is to identify the real relationships between two fragments, then such annotation requires specialized knowledge and a sufficiently long time to perform such a task.

Note that Jeretic et al. (2020), as part of their verification of annotation in the MultiNLI corpus (Williams, Nangia, and Bowman 2018), randomly selected 200 utterances from this corpus and presented them for evaluation to three expert annotators with several years of experience in formal semantics and pragmatics. The agreement among these experts was low. This provides the authors with an indication that MultiNLI contains a few “paradigmatic” examples of implicatures and presuppositions. ^v

Notice that the low agreement of annotators may also be the result of differences in their beliefs of theoretical nature and their research specialization. In our opinion, the analysis of the issue of human annotation process in such a task as detecting relations of entailment, contradiction and neutral in principle deserves a separate study.

6.2. Factivity/non-factivity opposition

It can be concluded that the opposition factivity/non-factivity for the prediction of ECN relations is relevant in a fundamental way. A factive verb co-occurs with relation E in 24% in our base dataset (32% in the extended dataset), while a non-factive verb co-occurs with relation N in 61% in our base dataset (53% in the extended dataset). This means that the linguistic material we collected in principle does not require labeling main verbs with other signatures (e.g., +/−; +/o) to achieve high scores for E and N relations (see Table 5 in our paper for precise counts.)

6.3. Verb frequency

It is also worth noting that in the task of predicting ECN relations in the “that|to” structure, we do not need large lists of verbs with their signatures to identify these relations reasonably efficiently. This is due to the fact that a relatively small number of verbs covers a large part of the entire dataset (see Figure 1). Data from other corpora (see, e.g., the attendance statistics for the British National Corpus) suggest that, given a list of 10 English factive verbs and 10 English non-factive verbs with the highest attendance, one might expect high model predictions if the train and test set were constructed from a random, representative sample of BNCs, analogous to our dataset, which is the NKJP sample. ^w

Given the problem of translation of utterances from one language to another, it is therefore sensible to create a multilingual list of verbs with the highest frequency of occurrence. We realize that the frequency of occurrence of certain Polish lexical units may sometimes differ significantly from that of their equivalents in other languages. However, there are reasons to believe that these differences are not significant. ^x A bigger issue than frequency may be a situation where a factive verb in language X is non-factive in language Y and vice versa. Table 13 contains lists the factive and non-factive verbs with the highest frequency in our dataset. We leave it to native speakers of English to judge whether the given English verbs are factive/non-factive.

Table 13. Top 10 verbs broken down into factive/non-factive subgroups. List of all factive and non-factive verbs is available in Appendix D

At this point, it is worth asking the following question: do the results obtained on the Polish material have any bearing on communication in other languages? We think it is quite possible. Firstly, the way of life and, consequently, the communication situations of the speakers of Polish do not differ from the communication situations of the speakers of English, Spanish, German, or French. Secondly, we see no evidence in favor of a negative answer. It is clear, however, that the answer to this question requires research analogous to ours in other languages.

6.4. ML model results

We presented benchmarks with BERT-based models and models utilizing prepared linguistic features. They are even better than the performance of non-specialist annotators. Hence, it follows that annotation of ECN relations by models on a random sample of NKJP would be better than of non-specialist annotators.

However, a few issues remain unresolved in this task, that is utterances with a pragmatic foundation. Other issues to examine are potential spurious correlations (e.g., influence of the verb tense on the model results)—further, deeper analysis of the models and their interpreting. Our results indicate the need for a dataset that focuses on these kinds of cases.

7. Summary

We gathered dataset LingFeatured NLI that is representative with regard to particular syntactic pattern “[verb][że (eng: that/to)][complement clause]” and factivity and non-factivity characteristic of the verb (in the main clause). The dataset is derived from NKJP—Polish National Corpus, which itself is representative for Polish contemporary utterances. In this study, we stated a review of the opposition factivity—non-factivity in the context of predicting ECN relations.

The four most important features of our dataset are as follows:

• it does not contain any prepared utterances, only authentic examples from the national language corpus (NKJP). Only the second version of the dataset (extended LingFeatured NLI) contains prepared utterances,

• it is not balanced, that is some features are represented more frequently than others; that is entailment class states $33.88\%$ of the dataset, contradiction— $4.40\%$ , neutral— $61.72\%$ . It is a representative subcorpus of the NKJP,

• each pair $\lt$ T, H $\gt$ is assigned a number of linguistic features, for example the main verb, its grammatical tense, the presence of internal negation, the type of utterance, etc. In this way, it allows us to compare different models—embedding-based or feature-based.

• it contains a relatively large number of factive and non-factive verbs.

We then analyzed the data content of our dataset. Finally, we trained the ML models, checking how they dealt with such a structured dataset. We found that transformer BERT-based models working on sentences obtained relatively good results ( $\approx 89\%$ F1 score on base dataset). Even though better results were achieved using linguistic features ( $\approx 91\%$ F1 score on base dataset), this model requires more human labor because the features were prepared manually by expert linguists. BERT-based models consuming only the input sentences show that they comprehend most of the complexity of NLI/factivity. Complex cases in the phenomenon—for example cases with entitlement (E) and non-factive verbs—remain an issue for further research because the models performed unsatisfactorily (below $71\%$ F1 score on the base and extended dataset).