This is the fourth in a series of posts about each of the teams that will be attending SCI 2019, and their projects. This one was submitted by Daniela Saderi.

The current system of scientific peer review is flawed. Research is evaluated by a handful of unpaid reviewers selected by journal editors as “experts” in the field. Two or sometimes three reviewers are chosen opaquely, often through personal connections. Behind closed doors, they decide the fate of a research article, largely basing their evaluation on subjective criteria. Paradoxically, while peer review is a key component for scientific dissemination, very few scientists receive any formal training in it.

This arcane process is not only slow and inefficient – with waiting periods from submission to publication of six months or more – but also disadvantages researchers from under-represented groups or under-resourced institutions. When they submit their research for peer review, their work is evaluated using standards developed for research groups with vastly more resources and connections. The gatekeepers of scholarly publication are disproportionately male (Helmer et al. 2017; Lerback, American Geophysical Union, and Brooks Hanson 2016; “Nature’s Sexism” 2012; Fox, Sean Burns, and Meyer 2016) and from North America or European countries (Chawla 2018; Murray et al. 2018). Rarely is research evaluated by a diverse pool of reviewers who can provide a comprehensive and context-appropriate evaluation of the science. Thus, we need better ways to find, train, and engage researchers in peer review, and, importantly, we need to be intentional about including researchers from groups sunder-represented in scholarship in the design and leadership on any potential solutions.

Preprints are early, yet complete, versions of scientific manuscripts made freely available online before journal-organized peer review. They offer multiple opportunities to both train and diversify the pool of peer reviewers: 1) research is available immediately, instead of being delayed for months; 2) community peer review allows for the feedback of a larger and more diverse pool of experts; 3) writing and sharing reviews helps train early-career researchers in the norms of constructive peer review; and 4) as citable objects, preprints and their reviews can be used as proof of productivity and engagement for career advancement.

Our team is working to support PREreview, a grassroots initiative aimed to facilitate the uptake of preprints and community discussion around them. The idea is that all researchers should be able to engage in constructive conversations around new scientific output and be recognized for that contribution. However, in order to ensure this effort succeeds at its mission of diversifying peer review and meaningfully engage all research communities, we need to better understand the assumptions made, current workflows, and reward systems in the context of scientific publishing en large.

The 2019 Scholarly Communication Institute will allow our team to come together and discuss ways to reach a more equitable and inclusive peer-review process by fostering open practices and building infrastructures explicitly designed to fit or be adapted to different cultural contexts.

Our team aims to achieve the following goals:

1. Understand publication workflows and cultural contexts that may influence the involvement of under-represented groups and countries in the peer-review process in general and in the review of preprints in particular.

We will begin by reviewing the workflows related to scientific peer review and the publishing systems familiar to our team members. We will highlight known incentives and rewards, both overt and covert, associated with engaging in peer review and with the posting of preprints in these different contexts.

We will challenge the assumption that preprints are a valuable and safe method of sharing research for researchers from under-resourced institutions and under-represented countries in the elite landscape of scholarly communication.

We will discuss rewards and incentives that can be set in place to encourage researchers to engage with preprints, share feedback, and participate to journal-organized peer review.

In addition to our personal experiences and professional knowledge, to inform these discussions we will review anonymized results from open surveys run by groups such as ASAPbio, bioRxiv, and the Center for Open Science.

2. Challenge/test existing strategies, and brainstorm new ones to grow and nurture a diverse community of preprint reviewers.

We will focus this part of the discussion on providing contextual feedback to the following initiatives:

• • A new PREreview open-source platform is almost ready to be launched and will be ready for wide user testing during the TriangleSCI event. Some key features designed to address the concerns and needs of vulnerable research communities include: optional pseudonymity with trackable contributions that can be shared with selected parties; a strong and visible code of conduct with anonymous reporting and clear community expectations and consequences for its violation; constructive peer-review templates; specific feedback solicitations (e.g., help with language, statistics, etc.); contribution recognition badges linked to ORCID IDs. What features/resources could help make this or similar tools useful and safe for communities from under-represented groups? What unexpected consequences might emerge? How can we best monitor community interactions to minimize intended or unintended harm?
  • The PREreview team is developing a cohort-based peer-review mentorship program based on the successful Mozilla Open Leaders. Researchers with little to no experience in peer review will be paired with more experienced reviewers and invited to engage in community calls, webinars, and one-on-one calls. The curriculum includes critical evaluation of preprints, strategies to help researchers provide constructive feedback, and training in unconscious bias, licensing, and leadership. How can this program support researchers from groups under-represented in scholarship? What additional topics and resources should be included in the program? How should the mentors be selected?
  • The team is implementing live-streamed preprint journal clubs, topic-centered, interactive preprint journal clubs that are live-streamed via video conference. These events are designed to be inclusive of all researchers by allowing structured and constructive discussions around preprints and encouraging diverse methods of participation. Researchers from all over the world can join to build their network, meet globally-renowned experts, and collaborate on improving a preprint. What unseen cultural contexts may prevent global participation? How can we ensure everyone is included? How can we design these events such that they are easily adapted to fit language- and discipline-specific needs?

Output plan

The output of this work – including methods, process, and resources for user research with under-represented communities – will be published and openly shared in multiple forms:

• As a detailed report shared with the team’s associated organizations: the African Science Initiative (ASI) executive board, eLIFE Ambassadors and early-career advisory committee, the Mozilla Foundation and related Open Science groups, the steering committee of INArXiv, and the PREreview advisory committee.
• As a chapter in the Open Source Alliance for Open Scholarship Handbook, a project led by the PREreview fiscal sponsor organization Code for Science & Society (CS&S) and aimed at sharing knowledge across communities active in the global open scholarship movement.
• As a presentation at regional and international conferences including possibly OpenCon 2020 (international and satellite meetings), Force11 2020, and WACCBIP Research Conference 2020.
• As a series of blog posts published across platforms including but not limited to ASI, PREreview, Mozilla Pulse, Medium, and shared with our networks through social media.

Team Members

• Richard Abdill
  Rich is a PhD student from the United States studying computational biology at the University of Minnesota, where his research is focused on host–microbe interactions in the human gut. Prior to graduate school, he worked as a software developer at Target, USA Today, and the Minnesota Supercomputing Institute. In addition to his work on the microbiome, Rich is also interested in meta-research, especially bibliometrics and the growing use of preprints in biology. He helped build the Rxivist.org tool for searching and filtering bioRxiv preprints, and he advocates for open science practices as an ambassador for ASAPbio. He is particularly interested in understanding how different groups interact with the scientific publishing system and improving methods to quantify trends and patterns in the use of preprints. At TriangleSCI, Rich is excited to learn from different perspectives on open science and how people use scientific communication in their careers.
• Yaw Bediako
  Yaw is a Ghanaian immunologist with a broad interest in investigating immune functions to better address immunopathology associated with infectious and non-infectious diseases among African populations. His post-doctoral research (most recently at the Francis Crick Institute) focused on understanding the immunological mechanisms by which naturally acquired immunity to Malaria is acquired and maintained. In April of this year, Yaw accepted an appointment as a research fellow at the West African Centre for Cell Biology of Infectious Pathogens (WACCBIP) at the University of Ghana. He is one of five African scientists to receive a Career Accelerator Award from the Crick African Network. Yaw is passionate about developing sustainable research infrastructure and human capacity on the African continent, especially leveraging indigenous expertise in the diaspora to strengthen local academic institutions. As part of this effort, he co-founded the African Science Initiative, an online platform aimed at facilitating networking and collaboration between African scientists all over the world.
• Ruth Gibendi
  Ruth is an academic librarian from Kenya with interest and experience working with scholars to use the variety of open resources and tools in their research. She previously worked at Strathmore University where she helped establish the Institutional Repository (IR) and open access mandates before joining Meru University of Science and Technology as head librarian. Her Master’s thesis at Moi University focused on understanding how faculty organize and disseminate the digital grey literature they produce in their academic undertakings (preprints, conference papers, working papers). Ruth is passionate about open access. She is currently exploiting a retrogressive model that seeks to help faculty showcase their profiles through Google Scholar, so as to increase the deposit of preprints in the institutional repository. At TriangleSCI, Ruth looks forward to meeting a diverse and talented team of experts from whom she will learn more about open review models and how these can integrate with existing models of open access in scholarly dissemination.
• Vinodh Ilangovan
  Vinodh is a postdoctoral researcher at the Max planck Institute for Biophysical Chemistry, Goettingen, Germany. He is an early career advisor for the non-profit, open access journal eLife Sciences and an ASAPbio ambassador advocating for preprints among peers. As an early-career researcher in the biomedical sciences, he strongly advocates for responsible behavior in research through engagement with preprints and transparent practices in data sharing among peers and institutional leaders. His long-standing interests lie in changing the cultural practice around research evaluation using metrics. In order to create a community driven research output indexing system, as a Mozilla Open Leader he initiated integrated inclusive indexing, a project built on open principles. He has experience facilitating webinars and workshops on innovative forms of research outputs and creating incentives. Through TriangleSCI, Vinodh aims to bridge the dichotomy between early and late adopters of preprints and learn diverse disciplinary practices surrounding open and equitable scholarship.
• Daniela Saderi
  Daniela is the co-founder of PREreview and a Mozilla Fellow for Science 2018/2019. At PREreview she leads the development of the new open source platform and works on the growth and sustainability of the project. She has recently earned a Ph.D. in neuroscience at Oregon Health and Science University, in Portland OR, USA. During her time as a student, she became increasingly interested in how open practices can be used to empower early-career researchers in taking ownership over their work, and to facilitate collaborations within and across teams. Engagement with global communities such as OpenCon and Mozilla Science, and her participation to international ambassador programs such as those at ASAPbio and eLIFE, helped Daniela to build advocacy and community engagement skills that she then applied to a number of projects. In addition to starting PREreview, she developed programs to grow the open research community in the Portland area, including OpenCon Cascadia, Science Hack Day PDX, and Python for Neuroscience Bootcamp. In the context of TriangleSCI, Daniela is excited to work together with such a talented and diverse group of scientists with whom she hopes to develop long-lasting partnerships.
• Rizqy Amelia Zein
  Amelia is a Social Psychology instructor, working at the Department of Personality and Social Psychology Universitas Airlangga, Indonesia. She is also a researcher-in-training at the Institute for Globally Distributed Open Research and Education (IGDORE) and a member of Tim Sains Terbuka (TST – Open Science Team) Indonesia, which consists of a number of researchers from diverse scientific backgrounds and institutions. She is actively involved in the advocacy of implementing Open Science policy in Indonesia by writing a number of op-ed articles and establishing Airlangga Open Science Community. Amelia is also interested in the issues of reproducibility of psychological science, and an enthusiastic member of the Society for the Improvement of Psychological Science (SIPS). By participating in this project, Amelia hopes to learn from other participants, their experiences in engaging scientists to adopt open science practices, as well as developing networking to strengthen TST advocacy of implementing Open Science policy in Indonesia.

References

• Chawla, Dalmeet Singh. 2018. “Huge Peer-Review Study Reveals Lack of Women and Non-Westerners.” Nature 561 (7723): 295.
• Fox, Charles W., C. Sean Burns, and Jennifer A. Meyer. 2016. “Editor and Reviewer Gender Influence the Peer Review Process but Not Peer Review Outcomes at an Ecology Journal.” Functional Ecology. https://doi.org/10.1111/1365-2435.12529.
• Helmer, Markus, Manuel Schottdorf, Andreas Neef, and Demian Battaglia. 2017. “Gender Bias in Scholarly Peer Review.” eLife 6 (March). https://doi.org/10.7554/eLife.21718.
• Lerback, Jory C., American Geophysical Union, and R. Brooks Hanson. 2016. “GENDER BIAS IN PEER REVIEW AND SCHOLARLY PUBLISHING.” https://doi.org/10.1130/abs/2016am-281633.
• Murray, Dakota, Kyle Siler, Vincent Larivière, Wei Mun Chan, Andrew M. Collings, Jennifer Raymond, and Cassidy R. Sugimoto. 2018. “Gender and International Diversity Improves Equity in Peer Review.” Scientific Communication and Education. bioRxiv.
• “Nature’s Sexism.” 2012. Nature. https://doi.org/10.1038/491495a.

[ Illustration by AJ Cann used under CC-BY-SA license. Photo by Plush Design Studio used under Unsplash free license. Post was edited on 5 and 12 and 19 September to adjust the team membership. ]