Language allows people to communicate ideas because each person’s brain responds similarly to the meaning of words. In newly published research, my colleagues and I have developed a framework for modeling the brain activity of speakers as they engage in face-to-face conversations.
We recorded the electrical activity of the brains of two people as they engaged in non-written conversations. Previous research has shown that when two people have a conversation, their brain activity becomes linked, or aligned, and that the degree of neural coupling is associated with better understanding of the speaker’s message.
The neural code refers to specific patterns of brain activity associated with distinct words in their contexts. We found that speakers’ brains are aligned on a shared neural code. Importantly, the neural code of the brain resembles the artificial neural code of large language models.
Neural patterns of words
A large language model is a machine learning program that can generate text by predicting what words are most likely to follow. Large language models excel at learning language structure, generating human text, and conducting conversations. They can even pass the Turing test, making it difficult for someone to tell whether they are interacting with a machine or a human. Like humans, large language models learn to speak by reading or listening to text created by other humans.
By feeding the large language model a transcript of the conversation, we were able to extract its “neural activations,” or how it translates words into numbers when it “reads” the font. We then correlated the speaker’s brain activity with both the large language model activations and the listener’s brain activity. We found that activations of a large language model can predict the shared brain activity of the speaker and the listener.
In order for people to understand each other, they have a common agreement on grammatical rules and the meaning of words in context. For example, we know that we use the past tense form of a verb to talk about past actions, as in the sentence: “Yesterday he visited the museum”. Moreover, we intuitively understand that the same word can have different meanings in different situations. For example, the word cold in the sentence “you’re as cold as ice” can refer to body temperature or a personality trait depending on the context. Because of the complexity and richness of natural language, we lacked an accurate mathematical model to describe it until the recent success of large language models.
Our study found that large-scale language models can predict how language information is encoded in the human brain, providing a new tool for interpreting human brain activity. The similarity between the human brain and the language code of a large language model allowed us for the first time to observe how information in the speaker’s brain is encoded into words and transferred, word by word, to the listener’s brain during face-to-face interactions. face to face conversation. For example, we found that brain activity associated with the meaning of a word appears in the speaker’s brain before the word is articulated, and the same activity quickly reappears in the listener’s brain after hearing the word.
New powerful tool
Our study provided insight into the neural code for language processing in the human brain and how both humans and machines can use this code to communicate. We found that large-scale language models were better able to predict shared brain activity compared to different features of language, such as syntax, or the order in which words are combined into sentences and phrases. This is partly due to the ability of large language models to incorporate the contextual meaning of words, as well as to integrate multiple levels of the linguistic hierarchy into a single model: from words to sentences to conceptual meaning. This suggests important similarities between the brain and artificial neural networks.
An important aspect of our research is the use of everyday recordings of natural conversations to ensure that our findings capture real-life brain processes. This is called ecological validity. Unlike experiments in which participants are told what to say, we relinquish control over the study and let participants converse as naturally as possible. This loss of control makes data analysis difficult because each conversation is unique and involves two interacting individuals speaking spontaneously. Our ability to model neural activity as people engage in everyday conversations confirms the power of large language models.
Other dimensions
Now that we have developed a framework for assessing the shared neural code between brains during everyday conversations, we are interested in what factors drive or inhibit this connection. For example, does the language bond increase if the listener has a better understanding of the speaker’s intent? Or perhaps complex language such as jargon can reduce neural coupling.
Another factor that can affect language binding can be the relationship between the speakers. For example, you may be able to convey a lot of information in a few words to a good friend, but not to a stranger. Or you may be better wired with political allies than with rivals. This is because differences in the way we use words across groups can make it easier to connect and connect with people within our social groups, rather than outside them.
This article is republished from The Conversation under a Creative Commons license. Read the original article.
Image credit: Mohamed Hassan / Pixabay