When Sholto David quit his job last autumn he could have looked for another post, taken time out to travel, or grabbed his tent and hopped on his bike. But David, a biologist living in Pontypridd, Wales, threw his efforts into a somewhat obscure hobby: finding flaws in scientific papers and doing his best to have them rectified.
The work, David says, is largely thankless. Academics often got defensive about their studies or refused to respond to his criticisms. Journal editors took a similar tack, ignoring his letters, rejecting them, or investigating on timescales bordering on the glacial.
But then came an announcement. Last week, one of the most prestigious cancer centres in the US, the Harvard-affiliated Dana-Farber Cancer Institute in Boston, said it was seeking to retract six research papers and correct 31 more after David raised concerns in a blog about dozens of its studies. Many were conducted by top executives at the institute.
Of the rest of the papers David flagged, Dr Barrett Rollins, the integrity research officer at Dana-Farber, said one remained under investigation and three required no further action. Sixteen contained data generated in other laboratories and where possible, the heads of those labs had been contacted. “We will work with them to see that they correct the literature as warranted,” he said.
The move came as a “major surprise”, David said. “To the credit of the Dana-Farber Cancer Institute, I’m obviously glad they are going to do the corrections and retractions. But at the same time, it does leave a sour taste in the mouth. Most of the time this just doesn’t happen. People ignore you, institutions insist it takes years to do investigations, and journals drag their feet at every opportunity.”
According to Rollins, following usual practice to “review any potential data error and make corrections when warranted”, the institute had already taken “prompt and decisive action” in 97% of the cases David had flagged where its scientists were primary authors. Rollins himself is an author of some of the papers flagged and had been recused from any relevant investigations.
David, who did a PhD in molecular biology at Newcastle University, has long been fascinated by flaws in science. He started off finding errors in systematic reviews, meta-analyses and clinical trials, which he would flag to the scientists responsible and the journals that published them. When those efforts proved fruitless, he turned to posting on PubPeer, a website where scientists can comment on published papers.
It became more than a pastime. The biologist has flagged about 2,000 papers on PubPeer, most because of concerns about potential image manipulation. At first, he identified cases by eye, looking for duplications and questionable manipulations that stretch images, crop out specific features, or splice parts together. Now, he has help from Imagetwin, AI-based software that compares images with a database of more than 25m pictures that have appeared in open access journals.
“A lot of it is sloppiness and my thought is that people have higher expectations,” David said. “When people think about science, when they donate to science and cancer campaigns, the expectation is that the scientists who do this research have high standards and are very careful in what they do.”
The flurry of retractions and corrections follows a major project in 2021 which found that researchers could reproduce results from only half of the top pre-clinical cancer studies they looked at. In many cases, when the experiments were repeated, any positive outcomes were far smaller than originally claimed. “Maybe part of that is because the work being done isn’t high quality,” David said. “It all eventually filters through to a lack of replicability.”
David stopped short of alleging wrongdoing and stressed that he did not want to create an environment where scientists felt harassed or terrified of publishing. But equally, he said, repeated errors were troubling.
“How many errors are acceptable before we think something more worrying is happening?” he said. “If you comb through a lot of people’s papers you will find errors, but there has to be, at some point, a limit to how many sloppy mistakes you make before it’s something else, that it’s not something you can dismiss as an honest error.”
Rollins said: “The presence of image discrepancies in a paper is not evidence of an author’s intent to deceive. That conclusion can only be drawn after a careful, fact-based examination, which is an integral part of our response. Our experience is that errors are often unintentional and do not rise to the level of misconduct.”
As for David, he thinks he will return to more conventional work in a year or two. “I’m just spending some savings,” he said. “Maybe this year I’ll do some travelling. I sound well off, but I’m not. I’m just stretching my meagre savings and it’s cheap rent out here.”