How can we achieve a fully open future?

You touch on peer review as a form of quality control for science, but you don’t mention quality control for peer review itself. Copying in a comment I made on EA Forum that seems relevant here as well:

One of the big drawbacks of peer review is the hugely variable quality of reviews that are provided. As an example simply in terms of the level of detail provided, I have had comments of one paragraph and three pages for the same article.

I think a key reason for this is there isn’t really any standardized format or expectations for reviews nor is there much training or feedback for reviewers. One thought I’ve had is that paying peer-reviewers would allow journals to both enforce review consistency and quality - although publishers have such large profit margins that it this could be feasible, they have no incentive to do so as scientists accept the status quo. In the absence of paid peer-review, I think that disclosing reviewer names and comments helps prevent ‘niche guarding’ and encourage reviewers to provide a useful and honest review (eLife does this currently, not sure if any other journals do so).

Note that while I suggested paying peer-reviewers in that comment as a way to enforce high quality reviewing standards, this could also be done via contracts to the research institutes that employee the researchers/reviewers as well, so it’s compatible with the model proposed by @jon_tennant .

2 Likes