Speaker
Description
Online dictionaries are being superseded by web search engines and chatbots. There is already a body of literature on whether the latter can assist (or replace) lexicographers (e.g., Barrett, 2023; De Schryver, 2023; Lew, 2023; Jakubiček & Rundell, 2023; Rundell, 2023). However, it is not known if these (so far) alterative reference sources help language learners in meaning comprehension and retention as much as dedicated reference works.
The study aims to determine if the source of reference affects meaning comprehension and retention. Five online sources are considered: Microsoft Copilot (powered by GPT-4), ChatGPT-3.5, Google, LDOCE and COBUILD.
The following research questions are asked:
1. Does meaning comprehension depend on whether ChatGPT 3.5, Copilot,
Google, LDOCE or COBUILD are consulted?
2. Does the source of reference affect immediate and delayed retention of
meaning?
An online experiment was based on 25 difficult words from De Schryver (2023). Prior to the study, 21 learners of English (B2) read the paper and took down the words they did not know, of which 25 most frequently listed ones were selected.
The experiment consisted of a pre-test, a main test and two retention tests (immediate and delayed). In the pretest and both post-tests, participants provided L1 equivalents of the target words relying on their knowledge. The pre-test showed whether they knew the words before the study. The post-tests checked meaning retention immediately after the main test and two weeks later. In the main test, the subjects supplied L1 equivalents after reference to the five aforementioned sources. A counterbalanced design was employed; each five 64 words were paired up with a different source. Source assignment was rotated across the words in five test versions (A-E). 109 upper-intermediate foreign learners of English (B2 in CEFR) participated in the experiment (test version A–21 students, B–23, C–20, D–22, E–23).
The results of a repeated-measures MANOVA indicate a statistically significant role of source (Wilks’s lambda=0.01, F=106.64, p<0.001, partial η2=0.784). To explore it, repeated-measures MANOVAs were conducted for each dependent variable. Significant MANOVA results were further analyzed with the Bonferroni test.
Meaning comprehension depended on the source of reference (Wilks’s lambda=0.12, F=39.23, p<0.001, partial η2=0.414). Comprehension proved the most (and comparably) successful when ChatGPT-3.5 (71.02%) and COBUILD (73.99%) were consulted (p=1.00); it was much more successful for these two sources than the remaining three (p<0.01). Copilot (61%) had a significant advantage over Google (53.66%, p=0.01) and LDOCE (53.48%, p=0.00). The latter two brought about comparably low comprehension (p=1.00).
The source referred to affected immediate retention (Wilks’s lambda=0.06, F=80.63, p<0.001, partial η2=0.503). ChatGPT-3.5 (52.64%) and COBUILD (50.96%) helped to retain meaning most successfully (p=1.00); they were significantly more useful for immediate retention than the other sources (p<0.01). Over two fifths (42.68%) of meaning explanations were remembered when Copilot had been consulted. Significantly less (p<0.00), about one third of the semantic content, was retained when Google (31.84%) and LDOCE (32.55%) had been used, with no difference between them (p=1.00).
Delayed retention depended on the consulted source, too (Wilks’s lambda=0.10, F=46.92, p<0.001, partial η2=0.437). COBUILD (32.10%) and LDOCE (29.13%) helped to retain the meaning of about one third of the words two weeks after exposure (p=0.07). Delayed retention after Google search (28.45%) was as successful as after reference to LDOCE (p=1.00), but significantly less successful than among COBUILD users (p=0.01). The least was remembered when ChatGPT-3.5 (21.43%) and Copilot (18.88%) had been consulted (p=0.19).
The study gives affirmative answers to both research questions; the source of reference affects meaning comprehension (Q1) as well as immediate and delayed retention (Q2).
In order to understand meaning and remember it immediately afterwards, COBUILD and ChatGPT-3.5 are the most recommendable, followed by Copilot. Google and LDOCE are the least helpful. However, in the long run, meaning is remembered best when COBUILD and LDOCE are consulted, and the latter is as useful as Google. Both chatbots assist delayed retention the least.
