Fresh comparisons of large language models show that AI’s ability to recognise Kremlin propaganda varies dramatically. At first glance, the leading models appear reliable, but targeted testing reveals that some of them remain surprisingly vulnerable to manipulation.
This week, the Estonian Language Institute published the results of a large-scale study evaluating more than 50 language models on their ability to identify Russian strategic narratives. Propastop contributed to the development of the benchmark by helping define the key narratives used in Russian influence operations and by assessing the models’ responses.
For Propastop, this collaboration demonstrates that disinformation research extends far beyond analysing articles, social media posts, or news feeds. The project with the Estonian Language Institute shows how expertise gained from studying Russian influence operations can also help address new challenges emerging in the age of artificial intelligence.
Propastop’s role
Propastop’s role was to help map 14 key narratives commonly used in Russian influence operations. Based on these narratives, researchers developed 75 test questions in Estonian, English, and Russian. The questions were phrased in three different ways: neutral, biased, and malicious. This made it possible to evaluate not only the models’ knowledge but also their ability to recognise manipulative claims and misleading assumptions.
Propastop experts also participated in calibrating the evaluation system. As part of this process, they reviewed a large number of model responses to verify whether the automated assessments matched expert judgment. Several experts independently evaluated the results, allowing the reliability of the assessments to be cross-checked. In total, the evaluation process lasted two weeks.
“The propaganda resilience benchmark was developed in cooperation with Propastop’s disinformation experts, who helped define the key narratives associated with Russian influence operations and whose expert assessments ensured the reliability of the results,” the Estonian Language Institute notes in its presentation of the benchmark.
What do the results show?
Language models do not perform equally well when it comes to identifying propaganda. The differences become especially apparent when a question contains a misleading assumption or is deliberately designed to bias the model.
Leading models generally handle such situations well, but the gap between models is significant. One particularly interesting finding was that several models performed worse in Russian than in Estonian or English.
For some models, the likelihood of producing a propaganda-supporting response was nearly twice as high when faced with a maliciously framed question compared to a neutral one.
According to Arvi Tavast, Director of the Estonian Language Institute, this type of work is becoming increasingly important because AI models are directly influenced by the information environment in which they are trained.
“Foreign political troll factories can produce large amounts of false content that can be used to bias AI models. This is a dangerous trend, and we must actively work to ensure that the picture of Estonia remains balanced.” — Arvi Tavast, Director of the Estonian Language Institute.
The work continues
Propaganda resilience is only the first part of a broader benchmarking program. In addition to safety, the Estonian Language Institute is also evaluating language models’ proficiency in Estonian and their knowledge of Estonia’s cultural context.
This autumn, the benchmark will be expanded with new assessments covering medicine, law, and the safety of children and young people.
The language model benchmark can be explored at mõõdupuu.eki.ee.
The language model benchmarking project was initiated by the TI-Hüpe Foundation and is supported by the Ministry of Education and Research of Estonia and the Eesti.ai initiative. The Institute of the Estonian Language is responsible for the benchmark methodology, publication of results, and the substantive analysis of the findings.
