This is the second in a series of posts about each of the teams that will be attending SCI 2023 and their projects.

Research integrity and open science practices promise to build trust in scholarly work, within academia and society-at-large. Scholarly peer review, as the main quality assurance mechanism for knowledge production, is increasingly embracing these principles. Various forms of Open Peer Review (OPR) have been proposed, ranging from the disclosure of reviewer identities, to the publication of review reports, and the involvement of wider audiences in the peer review process.

These models of peer review have the potential to contribute to the trustworthiness and validity of processes and publications in diverse ways. For example, transparency of review reports enables scrutiny of review processes, while disclosing reviewer identities aims to leverage accountability of reviewers leading to more constructive, kind and reliable review reports. Inclusion of a wider set of stakeholders through allowing not only invited experts to comment on articles, has the potential to diversify reviewer pools and subsequently lead to higher trustworthiness. Therefore, combined, these models could contribute to diversity, equity, inclusion and integrity of the review and research process, potentially leading to trust and trustworthiness. However, recent research shows the efficacy of OPR is still largely unknown (Ross-Hellauer & Horbach 2022). Therefore, it is unclear whether OPR is achieving these aims.

In fact, concerns have been raised about the extent to which OPR runs the risk of having the opposite effect. There are, for example, concerns over potential repercussions for junior scholars if they are identified as the authors of negative reports, or the tacit reluctance to criticise powerful members of the scientific community. If such effects are actually at play within OPR models, this will hinder participation from members of vulnerable groups, including women, early career researchers and members of epistemic or cultural minorities. In addition, these concerns might make reviewers, especially those in vulnerable positions, dilute their critiques. Both concerns could consequently lead to a less diverse reviewer pool or a reduced capacity of the review system to deliver valid or trustworthy research. We argue that both consequences go hand-in-hand: a failure to engage a diverse and wide set of reviewers will inevitably lead to an inefficient review system and reduced trustworthiness of science among researchers and potentially also society at large. It is therefore important not only to diversify voices in review, but also to empower wider communities to shape the format and conditions of the review process.

Similarly, while transparency of review reports is thought to contribute to trust, such openness can also lead to public awareness of controversy and uncertainty within the scholarly community, which could potentially undermine public trust in research and review.

As OPR increasingly gains traction, it is urgent to investigate the implications of the dynamics described above for perceptions of ability, integrity and benevolence (the three constitutive factors of trust), for the quality of published research, and for diversity and inclusion in review.

Challenges to be addressed

Despite the high expectations of OPR, the evidence-base on which its promises rest, is thin. In particular, we think three challenges should be addressed:

(i) there is still minimal information on the fundamental question of the extent to which OPR impacts the quality and trustworthiness of publications and review processes. We consider this the most pertinent open question. Addressing it is complicated by the lack of a general understanding of what quality means and how it can be measured in diverse epistemic and cultural contexts. Our team will therefore first work on a framework to conceptualise and assess quality and trustworthiness, based on three main components: products, processes and people. The framework allows to select specific sites to study individual components and their relations, including cases with direct public policy impact, cases with highly visible research actors like the covid pandemic, or cases involving minority communities where trust was broken or never established like the aftermath of the Tuskegee Syphilis study.

(ii) implications of OPR elements for reviewers, authors and other stakeholders involved in the review process are unknown. Inequalities in peer review have long been studied, but hitherto not in the context of OPR. To address this, we aim to assess whether and how Open Identities affect the diversity and composition of the reviewer pool, and more generally how OPR elements affect power imbalances and dynamics like homophily and cronyism.

(iii) we aim to better understand how the diversity of reviewer pools affects trust in review processes and products. In particular, we aim to focus on initiatives that attempt to introduce different perspectives in the authorship and peer review process, especially the perspectives of those being studied, including groups that have been historically kept out of science. Increasingly, those being studied are offered to contribute to peer review (e.g. patient review, or initiatives to combat parachute science), but the implications for trust are unknown.

Team and Action plan

To address these challenges, we will convene a diverse core group of researchers and publishers at TriangleSCI. We propose diversity to be not only a main object of study, but also centre-stage in our approach to studying it. Building on insights from STS and feminist studies, we therefore gather a team that is diverse in terms of gender, seniority, and epistemic, cultural, institutional and geographical background. This allows us to be sensitive to the different aspects and dynamics potentially impacting trust. In addition, the team members are well-connected to a large variety of communities and stakeholders across the globe, allowing solicitation of further perspectives and opportunities to roll out planned experiments. TriangleSCI is therefore a unique opportunity to bring together this highly diverse and newly-formed team that would otherwise not be able to collaborate so intensively.

The group will map issues of equity, diversity and trust in OPR, aiming to

(i) begin work to study the efficacy of OPR models in terms of diversity and perceptions of trustworthiness in peer review’s process, people and products,

(ii) create a detailed and actionable roadmap for future research, and

(iii) develop and initiate community-awareness activities about the benefits and limitations of OPR models to foster diversity and trust in review.

This should ultimately lead to informed recommendations about how to organise the system to foster both diversity and trustworthiness in research.

Call for Community Input

To include a wide variety of perspectives in our work, we are eager to learn about your views on the way in which OPR models can contribute to diversity and trust in review. Please share your thoughts about potential benefits and risks of such models through this very brief form.

Your input will be invaluable for us to widen our understanding of the topic, to prepare for SCI and to bring community rather than individual perspectives to the Institute. If you are interested in joining our efforts and/or collaborating on experiments, please leave your contact details at the end of the form, so we can get back to you.

Our Team

The team brings together a group of researchers and scholarly communication professionals from across the globe. Team members have diverse backgrounds, particularly bringing expertise and experience with open science: peer review practices: research integrity: public trust in science: and diversity, equity and inclusion. While specifically selected to bring complementary expertise to the team itself, the diversity in team members’ backgrounds closely related to TriangleSCI’s 2023 theme and objectives, will allow the team members to contribute to wider discussions at the Institute. Moreover, as team members represent a broad range of communities, they will bring the voices of these communities to the discussions at the Institute. The team members are:

• Gowri Gopalakrishna – Assistant Professor, Department of Epidemiology, Maastricht University; Senior Research Fellow, Amsterdam University Medical Center, Netherlands
• Serge P.J.M. Horbach – Postdoc, Danish Centre for Studies in Research and Research Policy, Aarhus University, Denmark
• Veronique Kiermer – Chief Scientific Officer at PLOS (Public Library of Science), United States of America
• Tony Ross-Hellauer – Group Leader, Institute of Interactive Systems and Data Science, TU Graz, Austria
• Sonia Vasconcelos – Associate Professor, Universidade Federal do Rio de Janeiro, Brazil
• Simine Vazire – Professor, Melbourne School of Psychological Science, University of Melbourne, Australia

[ Photos by Donald Giannatti and Aaron Burden used under Unsplash free license ]