In the United States, problems related to trust, transparency, and participation are affecting the scientific process. According to Pew Research Center, the percentage of Americans who say they have a great deal of trust in scientists fell 16 points between April 2020 and October 2023.
The Center for Open Science (COS), founded in 2013, seeks to address this problem by developing practices to improve the openness, integrity, and reproducibility of research. By enhancing transparency, sharing data and assumptions earlier, and facilitating critique and self-correction, Executive Director and Co-Founder Brian Nosek believes COS can accelerate knowledge sharing, solution identification, and the advancement of treatments.
The principle that animates COS is one every child learns in elementary school math: show your work.
Why trust in science and research is waning
“We don’t know what we don’t know,” said Nosek. “Part of how we learn things is through generating ideas, collecting evidence, and then putting that in the marketplace to see what survives scrutiny.”
Currently, the third leg of that stool — debating ideas and evidence in the marketplace — is hungry for innovation and improvement.
Recent peer-review research controversies highlight challenges in the academy’s existing systems to reliably produce knowledge and arrive at truth, resulting in significant costs for society and a mistrust of science. These controversies are not limited to the social sciences. Writing in the Journal of the American College of Cardiology, the Peer Review Task Force of the Scientific Publications Committee concluded, “The current process of peer review is often opaque, slow, and susceptible to conscious and unconscious bias.”
According to Nosek, there are at least two factors that contribute to the weakness of the peer review system.
Until the late 1990s, scholars had to submit paper documents to journals. The publications then sent the physical paper out for review, but never included underlying research. Even though publications have moved from analog to electronic, they still do not provide reviewers with the underlying evidence even though it easily could be shared. “The paper is just the summary of work,” explained Nosek. “The data, the analytical code — that is the knowledge — and now that we’re in a digital world, we can embrace that. Reviewers should have access to it.”
The second factor is how colleges and universities promote scholars.
“In higher education today, being published is the currency of advancement,” Nosek said. “The reward system is not about evaluation. In fact, we just assume good critique has happened.”
Encouraging scholars to prize publication over accuracy results in several negative outcomes. It compels authors to stubbornly stick to well-trod assumptions instead of testing thinking, for example, and it discourages them from inviting review by scholars who may challenge status quo paradigms.
The drive to publish also has given rise to bad actors — journals that, for a fee, will publish studies without rigorous peer review, resulting in mistakes or even fake research. These publications also pay scholars to add their names to papers. “The market is based on desperation. Elite journals in North America have checks and balances, but internationally this is a growing problem that affects hundreds of thousands of papers,” said Nosek. “And while insiders are familiar with what journals are reputable, users without system knowledge are at risk.”
The Lifecycle Journal: a prototype for a more open system
Improving the academy’s knowledge sharing system became a passion project for Nosek when he started to understand these bad incentives and how much science was lost due to the analog system. “There’s research from the ‘60s and ‘70s that’s just stuck in a drawer, but is relevant to questions we’re tackling today,” he said.
Nosek started his reform efforts from the ground floor: his own lab. His team posted drafts of papers and shared underlying data, code, and materials, for example. “Because it was so unusual, we got inordinate credit,” said Nosek. “People began using our methods and citing us. I may not have been able to put that on my CV, but there certainly were reputational benefits.”
Eventually, COS was born. Over the last decade, the organization has created technology to make open practices easier, provided community building to make open practices normative, and offered stakeholder engagement to make open practices more rewarding.
Currently, COS is building the LIfecycle Journal, a prototype for how to strengthen the research, review, and publication process. It will start taking submissions in early 2025. In the months following, Nosek and his team will gather feedback and insight, refine the prototype, and, with necessary funding, launch it more widely.
In addition to providing access to all the elements that help substantiate a paper, the Lifecycle Journal will invite review much earlier, starting from when a scholar submits plans. This input also will be available to the public. “It’s helpful for the public to know if an idea was challenged,” said Nosek. “It’s also helpful for them to know if reviewers from multiple viewpoints find something groundbreaking or valuable.”
The Lifecycle Journal and COS also will offer various evaluation services, from examining predictions and hypotheses to providing input on data sharing and the strength and reproducibility of research.
“There are lots of ways we can experiment with what evaluation means,” said Nosek. “Our goal is to move from a single point of evaluation — whether something merits being published in a journal or not — to a market of ways that evaluation can inform the process.”
The Lifecycle Journal also will try to address the negative stigma around retracting or correcting articles by allowing scholars to revise their findings after publication if a point needs to be fixed or updated. “We know errors are going to occur,” Nosek said. “The point is to make it easier to identify them and create accountability for correcting them.”
The benefits of a more open approach
Colleges and universities, individual scholars, and the public all stand to benefit from Nosek’s new model.
By allowing scholars to receive credit for all academic contributions, not just publication, the Lifecycle Journal will prioritize truth-seeking and critical evaluation over volume. For example, COS would provide a platform for recognizing when data or code was used in research. For emerging scholars in particular, the Lifecycle Journal will provide an opportunity to contribute to the republic of science and build a resume.
“Our hope is that by opening up the process and encouraging critique and engagement, we make wrestling with ideas a rewarded activity for scholars,” Nosek said.
Of course, Nosek hopes the Lifecycle Journal will result in better science. He envisions a future where all scholarly content is preserved, connected, and versioned, allowing scholars to build more easily on what came before them to foster discovery.
Because the Lifecycle Journal will allow for revision and updates and will improve transparency by providing the public with access to reviewer feedback, it could help restore trust in science. Readers also will have direct access to review primary evidence for scholarly claims, meaning everyday consumers — journalists, doctors, political and public policy officials, curious citizens — can be assured scholarship is sound.
“We address bad actors by making people show their work,” said Nosek. “That’s the basis for establishing trustworthiness in science.”
Once the Lifecycle Journal establishes that critique and challenge is the norm rather than the exception, a wider array of scholars is likely to engage, Nosek said. “Readers will be able to recognize if reviews all come from the same perspective, and talk about that,” he explained. “This conversation, along with the new evaluation modules, will provide a pathway for scholars from multiple ideological perspectives, evidence orientations, and methods of evaluation to look at the same content, but from different perspectives.”
Nosek acknowledges this project is ambitious, but that difficulty has not kept people from embracing his mission. “Some people think this task is impossible,” he said. “But very few disagree about the need for added rigor, self-critique, and transparency in scholarship. We need to try.”
