Report from #review session

Starting with first principles: Broadening the definition of scholarly communication, review, and peer

 

The communications of today’s scholars encompass not only book and journal publications, but also less formal textual communications and a variety of other work products, many of them made possible by recent advances in information technology. It is not uncommon today for an academic CV or faculty activity report (the productivity report that faculty members provide each year) to list news articles, blog postings, tweets, video presentations, artworks, patents, datasets, recordings of government testimony, computer code, in-person (conference and meeting) communications, and other artifacts. Yet our entrenched systems of review and recognition are out of sync with this evolving scholarly record. New ways to track our broader scholarly record, and new methods and metrics for identifying relevant work and evaluating quality and impact of works, are needed to help legitimize a broader definition of academic contribution.

What is review? Who is a peer?

The term “review” refers to several different events or functions in the scholarly communications workflow. It can refer to evaluation of the aggregate contributions of a researcher as well as evaluation of specific publications describing specific research. It encompasses both qualitative and quantitative evaluation, at both pre-publication and post-publication stages.

 

Our vision of the future of peer review in scholarly communication sees a diminishing role for pre-publication evaluation, as the high costs associated with it have begun to outweigh the researcher incentives to contribute on a quid pro quo basis. New systems for post-publication evaluation are much better poised to leverage current and emerging scholarly workflows, and to capture multi-dimensional measures that combine qualitative and quantitative information in a way that serves the researcher-as-reader (by helping the reader to most effectively allocate limited attention to an ever more overwhelming scholarly record) and the researcher-as-author/contributor. Additionally, these new forms of review can provide valuable input into appointment (promotion and tenure) assessment processes — though we strongly oppose an over-reliance on simple quantitative measures related to publications as a substitute for thorough peer review of a scholar’s contributions within the assessment process.

Our vision of the future of peer review also sees a much broader definition of “peers” — those with a voice to participate in the review process — to include any consumer of the work, evaluating the work in any traceable context. This will add new complexity to the system, but also perhaps new transparency, and may have the effect of turning some review processes into more conversational activities.

What should be carried forward, and what should be left behind?

The current scholarly evaluation system works in many ways. It is increasingly clear that it could work better, but before we start suggesting alternatives, we need to examine what’s good about the present evaluation system. At their best, our current methods of review persist because they have been relatively successful at filtering out erroneous, duplicative, and less significant work, as well as helping us identify significant insights, advances, and discoveries. They leverage the trust we place in our peers. The hierarchy of peer-reviewed journals, supported by quantitative metrics like the Impact Factor, offers easy-to-use reputational heuristics in the academic review process.

 

These systems represented the state-of-the-art of an earlier technological era (the era of paper journals and books), and have served the community well. However, new technologies present opportunities to do better. Where legacy systems increasingly fall short is that they don’t function well across disciplines, they are subject to misuse, gaming, and misinterpretation, they can substantially slow the process of disseminating research findings, and they place significant burdens on researchers (this is particularly true with regards to pre-publication peer review). The result is that the process can be said to fail (in some measure) for the majority of published content.

 

Because the review process is deeply embedded in cultural norms for disciplines and sub-fields, innovative tools fail more commonly due to reasons of incompatibility with existing norms and behavior rather than due to technical shortcomings. Recent experiments that have failed have done so because they fail to understand disciplinary norms and boundaries (for instance, while a tool like arXivhas been very successful in certain fields of physics, the attempt to port it into certain fields of chemistry failed due to differing norms of sharing in the two fields); or they attempt to source crowds where there are no interested crowds (e.g., Facebook-like social networks for scientists).  Even the best tools will not succeed unless they harness the essential elements of existing review systems — particularly filtering and trust — and also support existing communities’ cultural parameters. Change of this nature is likely to come slowly, particularly if it is to have lasting impact.

 

Most promising technologies and practices for the future

 

Considering the diversity of research, its practitioners, and the quickly-changing scholarly environment, we should not expect a one-size-fits-all solution to the problems of review nor for a single Holy Grail metric to emerge in the foreseeable future. Rather, the future will rely on a range of tools to filter the work and assess its quality in the their disciplinary communities and amongst individual scholars.

 

Different disciplines will obviously rely on different tools but scholars must take responsibility for the accuracy and completeness of their own part of the scholarly record. While Open Access is one part of the solution, scholars need to educate themselves on the specific tools used in their disciplines and be aware of the tools used in others to maximize the possibilities.

 

These tools monitor the traces they leave to ensure their scholarship is accurately presented in the venues their disciplines value most. It is the responsibility of (and indeed, incumbent upon) the individual scholar to take control of his or her bibliographic legacy and reputation. Beyond this, scholars have vastly extended opportunities to understand how their work is being read and built upon by other scholars.

 

While the tools for consumers and producers of scholarship listed below provide an important overview of the current possibilities, there isn’t an expectation that they are appropriate or relevant for all disciplines.

Tools for consumers of scholarship: How to find the good stuff

     Multidimensional aggregators (ALM, citedin, ScienceCard, altmetric.com)

     Formalized, directed commenting systems and systems that allow for rating the raters (F1000, CommentPress, digressit)

     Aggregated researcher data: Mendeley, Zotero

     Social bookmarking tools: Delicious, Cite-U-Like, Connotea

     RSS feeds for scientists, shared through bookmarking

     Social Networks: Twitter, Google+, Facebook

     Mentions and tweets (e.g., for digital humanities where community re-tweets of relevant links are valued for review purposes; see also jasonpriem.com/self-archived/5uni-poster.png)

     Discipline-specific Twitter lists, Facebook groups, Google+ circles (making it easier to follow colleagues and colleagues’ work)

     Computing over literature: textual analysis (SEASR)

Tools for producers of scholarship: How to make sure your stuff is given its just rewards

     Keep a social record of your scholarly activities (Papers,  Mendeley, ReadCube, Zotero)

     Proactively create a broader record of scholarly communication, and co-create an institutional record (e.g., VIVO, CatalystProfiles)

     Disambiguate your own scholarly record (ORCID, MicrosoftAcademicSearch GoogleScholar, WebofScience)

     Track when you are being cited (regular vanity searches in Google Scholar, Microsoft Academic Search, academic.edu, Scopus, WebofScience)

     Monitor social network activity surrounding your research (Twitter, Facebook, Google+ searches, Klout)

     Manage, track and present your own measures of impact (citedin.org, sciencecard.org, TotalImpact,, altmetric.com)

 

Currently Unsolved Problems, Solutions, and Strategic Opportunities

 

There are several (currently) unsolved problems, these fall into 3 distinct categories:

 

Technical / Business Limitations

Problem: There are a lack of industry standards, meaning that the same metric cannot be easily compared between different platforms. Solution: Standards bodies (such as NISO) should define cross platform industry standards

 

Problem: Related to this, many corpora are not open in a machine readable fashion, making it problematic to apply a uniform metric across them. Solution: If possible, authors should mirror their content in Open Access Repositories

 

Problem: Certain metrics benefit from content being available in a specific (non-universal) format (e.g., Open Access content will gain more usage; multimedia content will receive more interaction). Hence certain content will be naturally disadvantaged in any alt-metrics evaluation of its benefits. Solution: Standard meta data sets should be attached to all output.

 

Problem: If we rely on third parties for data, then we must accept that those sources may change over time (or disappear). This means that alt-metric evaluations may never be ‘fixed’ or ‘repeatable’. Solution: Everything decays, but permalinks, and archival storage of data can limit the damage.

 

Problem: Metrics by and large only include so-called formal publications and do not capture the variety of informal media and data that are increasingly important to scholarly discourse. Solution: Providers of data need to open up their data sources, allowing tools to easily mine the widest possible variety of sources

 

Societal Limitations

Problem: Important decision makers (e.g., tenure committees) do not use alt-metrics in their evaluation process. Solution: The utility of alt-metrics needs to be demonstrated in order to persuade decision makers to use them

 

Problem: People ‘cite’ work in a wide variety of ways with a variety of semantics, resulting in difficulty automatically mining these cites. Solution: Due to the human condition, It is possible that there is no solution

 

Problem: Some work is never cited at all, but simply influences the work of others (e.g., a study which may inform a change in governmental policy). This can make automated mining impossible. Solution: Perhaps automation is impossible, but crowdsourcing this discovery is a solution

 

Adoption / Understanding Issues

Problem: Generational, geographic, and disciplinary differences mean that not all academics have adopted new methods of dissemination / evaluation to the same extent, hence disadvantaging certain sectors. Solution: Metrics that get valued the most should be the ones which have been adopted to the greatest extent. Societies and funders should encourage ‘best’ adoption of tools.

 

Problem: The notion of long-term ‘impact’ is not really well understood in traditional metrics and therefore is hard to replicate or improve upon in emerging methods. Different metrics have different value for different people and so academia will need to understand that there may never be a single metric to describe all work. Solution: We need a more nuanced and multi-dimensional understanding of impact for different groups.

 

Strategic Agenda:

 

     Propagate changes in attribution across scholarly systems (ORCID). ORCID is not in a position to handle the retrospective problem, so perhaps ‘gamify’ the problem to disambiguate author names; 

     Develop centralized (disambiguated) tools to track when you are being cited in the widest variety of possible sources;

     Develop standards to define metrics and metadata

     Open up common social media tools to develop more open environments for data interchange

     Dedicate more attention to non-Western tools and services (including multilingual tools)

     Propagate the adoption of Open (pre-publication) Peer Review

 

Interdependencies with other topics

 

The future of peer review in scholarly communication will include new methods and metrics for evaluating quality and impact that:

     extend beyond traditional print and digital outputs,

     are dependent on a broadening definition and semantic richness of literature — that is, of the communications that form the “reviewable” scholarly record.

 

Efforts to broaden and filter our scholarly record will legitimize a broader definition of academic productivity and enable new models of academic recognition and assessment that better align with the actual activities and contributions of today’s scholars.

 

Authors

Peter Binfield, PLoS

Amy Brand, Harvard University

Gregg Gordon, SSRN

Sarah Greene, Faculty of 1000

Carl Lagoze , Cornell University

Clifford Lynch, CNI

Tom McMail, Microsoft Research

Jason Priem, UNC-Chapel Hill

Katina Rogers, Alfred P. Sloan Foundation

Tom Scheinfeldt, Roy Rosenzweig Center for History and New Media at George Mason University

 


Notes for #review group

We’re taking notes and chatting here: bit.ly/sHt9OP


Rough Transcript of my Opening Remarks

Dear Colleagues:

 It is an honor to be asked to address this group, some of whom I seem to see more than my own family, to set the stage for this 2011 Microsoft Research sponsored eScience workshop on Transforming Scholarly Communication.

 The first fundamental question to ask is, do we need a transformation in the first place? Obviously we all believe we do otherwise we would not be here, but what about the majority of scholars? I use my colleagues up and down the corridor as a benchmark to answer that question. A group currently oblivious to much of what we will show tomorrow. But nevertheless a group increasingly not oblivious to the changes going on around them – data sharing policies, cuts in library budgets, open access, and our students. Let me illustrate this latter point with a recent example of something remarkable that happened to me.

 A couple of months ago I received by email a paper to PLoS Comp Biol. This happens from time to time as authors try and circumvent the standard submission procedure and contact me as Editor in Chief directly. It was a paper in pandemic modeling, which appeared to question conventional approaches to such modeling. Not being an expert here I sent the manuscript to Simon Levin in Princeton who is on our Editorial Board for his opinion. Simon is a Kyoto Prize winner and an expert in large-scale biological modeling. He indicated there was something special about this well written paper. Since the sole author was living in San Diego I agreed to meet with her and discuss the work. Simply by asking she had received a large amount of free computer time from the San Diego Supercomputer Center (SDSC), got free access to Mathamatica and had clearly benefited from the open access literature as well as resources like Wikipedia. I encouraged her to submit the work to Science, which she has done, and it is currently under review.

 

The sole author’s name is Meredith.  What makes this story remarkable, is that she is 15 years old and a senior at La Jolla High School in San Diego. She subsequently presented her work at my lab meeting, which I must say was much better attended than usual. Sitting there with my eyes closed I thought I was in the presence of a professor deep into their area of expertise. It was only when I opened my eyes and saw the braces did reality sink in.

Clearly this is an extreme case, but lets not be modest, what we are trying to do here is enable anyone with an Internet connection and a will to learn, achieve what Meredith has achieved. I cannot think of a more noble cause. While we have seen this possibility for a long time, what is new is that others are now seeing it too.

We all have our own Meredith stories or at the very least some driver that moves us in the same way. For some it is the glacial pace at which knowledge exchange takes place; for others it is the sense of unease about the lack of reproducibility in our own science; for others it is the inaccessibility of knowledge; and for others still it is the totally qualitative way quantitative scientists measure the value of scholarship.

With Meredith as our motivation, let us take a minute to analyze the path we are on towards transforming scholarship through what has happened this past year and then what might happen as a result of this workshop.

2011 may well be remembered as the year that stakeholders – scientists, publishers, archivists and librarians, developers, funders, and decision makers went from working in isolation to beginning to work together. What started with Beyond the PDF in January had become a “movement” by the time the summer meetings were over. The Dagstuhl meeting captured the spirit in a manifesto that should become a living document for us all to consider. Movements have transformed entrenched systems before and it remains an open and very exciting question as to whether that will happen here. For a small group to cause change to many requires that the many believe that change is needed and gradually get on board. I believe that time has come.

The driver of change is the ground swell towards open science. When I first heard that a group of prominent life scientists got together and agreed to start a new open access journal I was disappointed – such vision coming up with something that we had already. But if the effort by HHMI, Wellcome Trust and Max Plank does indeed compete with Science and Nature it will precipitate change. My sense is that Publishers see the writing on the wall, or more appropriately the screen and the smart ones are gearing up to a future with different business models. The winning publishers will move from serving science through scientific process and dissemination to doing that plus enabling knowledge discovery, more equitable reward systems and improving comprehension by a broader audience. Interestingly, it is not clear to me, based on my interactions with OAPSA, that open access publishers see it that way. Many simply see delivering papers as before, but with a different revenue model. Ironically even if they see the promise of change, they do not have the resources to make it happen. We must help them and that is why meetings like this one are so important. A serious example of what we must fix is the lack of consistent representation of their papers in XML. PubMed Central will come back to haunt us when developers begin to seriously try and use the content. This is history repeating itself – look at the biological databases. We should learn from history.

Open science is more that changing how we interface with the final product it is interfacing with the complete scientific process – motivation, ideas, hypotheses, experiments to test the hypotheses, data generated, analysis of that data, conclusions and  awareness. This is a tall order and I believe we need to proceed in steps. Clearly access and effective use of data is a valuable next step. Funders are demanding it, scientists (to some degree) are providing it and repositories exist to accept it. But right now it is a mess, but we have an opportunity. Ontologies exist, some tools exist and so we have the opportunity NOT to repeat the horrible loss of productivity we see in the publishing world of rehashing the same material for different publishers. Let us define and implement data standards and input mechanisms that capture the generic metadata, provide the hooks for more domain specific deeper content and allow a more universal deposition and search. We need to do this now before systems become entrenched. Otherwise Google, Bing and the like will be our tools for data discovery – we need deep and meaningful search of data.

Let me conclude with a couple of thoughts on what I believe should come from the workshop. 

1.     We will hear about some wonderful tools and innovative software developments to support scholarly communication – we must define a way to aggregate these efforts to facilitate uptake by others around a focused and shared development effort.

2.     We need to define ways to recruit to the movement – it will take more than tools to do so – are there clear wins for all concerned? If so what are they? Platforms to disseminate scholarship, new reward systems, knowledge discovery from open access content, proven improved comprehension.

What can we do so that more 15 year olds are active contributors to scholarship? This is our challenge. Thank you very much.

 

 

 

 

 

 

Bios of the Organizers

Bio sketches of the meeting organizers, who will float from group to group:

  1. Alyssa Goodman, Harvard University
  2. Alberto Pepe, Harvard University
  3. Mary Lee Kennedy, Harvard University
  4. Malgorzata (Gosia) Stergios, Harvard University
  5. Lee Dirks, Microsoft Research
  6. Alex Wade, Microsoft Research
  7.  Joshua M. Greenberg, Alfred P. Sloan Foundation
  8. Chris Mentzel, Gordon & Betty Moore Foundation

Bios of attendees in #review theme

  1. Charles Alcock, Harvard-Smithsonian Center for Astrophsyics
  2. Peter Binfield, PLoS
  3. Amy Brand, Harvard University
  4. Crystal Fantry, Wolfram Alpha
  5. Gregg Gordon, SSRN
  6. Sarah Greene, Faculty of 1000
  7. Carl Lagoze , Cornell University
  8. Clifford Lynch, CNI
  9. Tom McMail, Microsoft Research
  10. Jason Priem, UNC-Chapel Hill
  11. Katina Rogers, Sloan Foundation
  12. Tom Scheinfeldt, Roy Rosenzweig Center for History and New Media at George Mason University

Demonstrations on REVIEW on Meeting Day 1

REVIEW Standard publication-based systems, alternative rating systems, etc.
Facilitator: Amy Brand, Program Manager, Harvard Office of Scholarly Communication 

  • Peter Binfield, Publisher, Public Library of Science (Demo of Article Level Metrics)
  • Sarah Greene, Editor-in-Chief, Faculty of 1000 (Demo of F1000)