Scientists created a fake eye disease to see if an AI chatbot could detect it, but the experiment took an unexpected turn when Gemini ChatGPT started treating the fictional disease as a real medical condition
A fake eye disease exposes a real problem with artificial intelligence.Swedish researchers deliberately invented a device called Bixsoniamania See if a popular AI chatbot can spot fake medical information. Instead, several large language models confidently described the non-existent disease as if it were real, demonstrating how easily misinformation can spread through artificial intelligence systems trained on internet data, especially on sensitive issues like health.The experiment also revealed another problem. Some researchers cite bogus scientific papers related to this imaginary disease without noticing that these studies openly admit that the papers are fabricated.
Fake illness with real purpose
The project is led by Almira Osmanovic Thunström, a medical researcher at the University of Gothenburg who is also an AI strategist and innovation manager at Chalmers Industriteknik in Sweden.She came up with the idea while teaching students how to build and train large language models (LLMs).“What’s interesting is that very few of them, even in the field of artificial intelligence, understand how large language models are built,” she told Rachel Feltman scientific americanof science hurry up podcast.To demonstrate how these systems gather information, Osmanovic Thunström and her colleagues created bixonimania, a fictional eye disease purportedly linked to excessive screen use and blue light exposure.The goal is not to fool people, but to see if the chatbot can distinguish between reliable information and fabricated material deliberately planted online.Osmanovic Thunström added: “So I really want to have a clear case that leaves traces throughout the system to show how the data is processed, how the data is generated, and how the predictive model and the training model work when distributing information.”
“Lying Loser”
In early 2024, researchers began spreading information about the fake disease on the Internet.They published two blog posts on Medium and uploaded two research papers to scientific preprint servers to share their findings ahead of peer review.The documents themselves contain some obvious clues that they are not genuine.The primary author is listed as Lazljiv Izgubljenovic, a name that translates to “Lying Loser”. The title of a study means something along the lines of “Hyperpigmentation: Real BS Design”. The newspapers thanked organizations including the Galactic Triad Lord of the Ringsand thanks also to my colleagues on the Starship Enterprise and Professor Ross Geller friends.At least one paper even states: “The whole paper was made up.”These reports have been deleted from the server.“I don’t think preprints are a tabloid in academia, because anything can end up in a database,” says Osmanovic Thunström. science hurry up.
Chatbot embraces the disease
These warnings are not enough.After false material appeared online, several leading artificial intelligence chatbots began treating bisexuality as a legitimate medical condition.according to natureMicrosoft Bing’s Copilot described it as “really an interesting and relatively rare situation.” Google’s Gemini says this is “a condition caused by overexposure to blue light.” OpenAI’s ChatGPT also hinted at the disease when users described symptoms such as itchy eyes, red eyelids, and inflammation after prolonged screen use.Several chatbots responded after users asked bixonimania directly. When people describe symptoms related to blue light exposure, others allude to this pseudodisease without even mentioning its name.The results revealed a crack that is a weakness of large language models. They generate answers by recognizing patterns in the information they absorb, rather than independently verifying whether the information is true or completely fabricated.
Even the researchers were caught
The experiment revealed another unexpected problem.Some scientists cite fabricated research papers in their own work, suggesting that they have not actually read the papers they cite.If they did, they would find countless jokes and explicit statements that the newspapers were fictional.For Alex Ruani, a misinformation researcher at University College London who was not involved in the project, the findings illustrate how misinformation spreads through similar scientific and technological systems.“This is a master class in how misinformation and disinformation works,” Rooney said nature.They added: “If the scientific process itself and the systems that support it are proficient, and they do not capture and filter out such nuggets of data, then we are doomed.”
why this is important
Large language models are increasingly used to answer health questions, explain medical conditions, and even provide emotional support to millions of users every day.Many users now view chatbots as their first source of information before consulting a healthcare professional. This could be dangerous if users try to use AI to self-diagnose illnesses without actually consulting a doctor.The bixonimania experiment demonstrates why this approach carries risks. These systems learn from large amounts of online content, and if inaccurate or falsified information is fed into their training data, it can become part of their responses.Jonathan Goodman and Mariam Rashid, social scientists at the University of Cambridge who were not involved in the study, said AI tools can be valuable, but users still need to treat them critically.While tools like artificial intelligence and the internet are helpful, “we have a responsibility to make sure we are using them and not being manipulated by them,” they argued. dialogue.They also argue that the underlying problem goes far beyond artificial intelligence.“Misinformation is here to stay,” Goodman and Rashid write in Misinformation dialogue. “What’s new is how quickly it spreads, the tools that generate it and how well it mimics the real thing.”
