Thanks for posting @pcmasuzzo. This is an interesting article. In the last few years, I’ve read more deeply on Philosophy and Sociology of Science (books, papers, etc.). In contrast, I’ve generally engaged with Open Science more superficially (blogs, guides, talks, etc.). As such, even though I generally consider myself as supportive of Meta/Open Science, I could relate to the perspective of this paper quite clearly.
the Metascience Symposium, twenty-two hundred miles away, another group of science studies scholars was sitting in a conference room in New Orleans discussing the same topics. The Society for the Social Studies of Science hosted three consecutive panels on the replication crisis and its implications. What is remarkable about these two events is how little visibility there is between the groups given the enormous overlap in the subject matter.
I agree with the general view of the paper is that the current iteration of Metascience should probably have more contact with qualitative research done on science from a philosophical and sociological perspective. Another interesting point is that a researcher in scientometrics also told me that they were surprised that Metascience doesn’t engage more with their field, where the Society for Informetrics and Scientometrics already hosts a conference covering similar topics on science quantification. Yet, he views there as having been little engagement between these fields.
Put simply, metascience is a scientific social movement that seeks to use the tools of science- especially, quantification and experimentation- to diagnose problems in research practice and improve efficiency. It draws together data scientists, experimental and statistical methodologists, and open science activists into a project with both intellectual and policy dimensions. Intellectually, metascience produces quantitative studies meant to describe and evaluate science on a macro scale. Metascientists then use those studies to motivate reforms in scientific practice including more elaborate reporting requirements, data deposition demands, and more replications.
…
The history of metascience cannot be understood apart from the anxiety about a possible reproducibility crisis. Metascience has produced the tools that supposedly uncovered the crisis through statistical critiques, meta-analyses, and mass replications. Together, these provided the evidence that activists have used to demonstrate systemic problems across fields. And, metascientists have positioned themselves as key players in solving the crisis by pushing for interventions in everything from scientific training to reporting.
This is interesting; I hadn’t noticed before that metascience is a field that aims to both diagnose a problem and promote a cure. Generally, I think its wise to be cautious about this as it can be quite a self-serving approach (this is somewhat ironic, given metascience’s thesis that the incentives of science distort research practices to become self-serving instead of science-serving).
statistical debates and replication scandals have occurred before. What has given metascience its potency has been its ability to attract financial and intellectual investment and, then, use that investment to create self-perpetuating cycles of research, critique, and intervention. … What Brian Nosek and other metascientists have done is to funnel the energy and funding pouring into metascience into institutions that will influence science policy long after the news cycle surrounding the crisis narrative fades.
problematically, as metascience institutionalizes—as it creates self-perpetuating institutes, conferences, journals, and departments—it creates the potential for an ongoing gulf between metascience and disciplinary science as metascientists no longer come to metascience through a discipline, but are raised intellectually in the heady mixture of big data, open science, statistical mechanics, and little knowledge how bench science is done.
These quotes also call for caution. If Metascience becomes self-perpetuating without subjecting its own existence to criticism, then it seems possible that Metascience could outlive its usefulness and/or encroach into fields where it is unhelpful (as I note below, one example could be emerging fields). Although Metascience provides a solution to the problem (that it diagnosed), it would be concerning if, having eventually solved those problems, diagnosing new problems to solve with the standard treatment becomes the core of Metascience.
(to paraphrase some common start-up advice; it’s not good to build a solution and then search for a problem - it’s better to find a problem that needs solving and build a solution)
The desire for “hard-nosed empiricism” meant that no qualitative studies of science were presented at the Metascience Symposium. There was a session which several respondents called the “skeptics panel,” but the critiques were focused on technical issues in metascience rather than the theoretical challenges that are common features of debates in STS. Jan Walleczek, Director of Research at the Franklin Fetzer Fund, helped organize the conference. He told us that avoiding these sorts of theoretical debates was a key decision regarding invitations to the symposium.
I wonder if Metascience’s focus on quantitative analysis may have come from the current quantitative practices used to assess researchers for promotion/grants (i.e. H-Index, Impact Factors) - kind of a move from quantifying productivity/quantity to trying to quantify quality (perhaps in a more nuanced way) as well. In that sense, it’s also interesting the recent UNESCO Open Science panel clearly called for more towards qualitative assessment of research/er quality (see my notes here).
Metascience poses a deep dilemma. On one hand, they are engaged in a seemingly commonsensical and uncontestable mission to make science more transparent and efficient. They attempt to formalize the loosely defined “norms” of science. … On the other hand, the project itself seems to be founded on a set of ideas that find few supporters in the philosophy or social studies of science: that “science” is a coherent entity on which to intervene, that there is a singular method for science, that “efficiency” is a meaningful concept in the area of basic research. The fundamental stance of metascience would seem to be at odds with one of the central findings of science studies, that “science is not one, indivisible, and unified, but that the sciences are many, diverse, and disunified.”`
I mostly do interdisciplinary work in what might be considered emerging or pre-paradigmatic fields. I often find myself thinking, damn, it’s tough to approach things like Open Data when there is no standardised formats or metadata, or even repositories for what you are doing. I feel some of the quantitative aspects of meta-science are going to fit best into established fields, while the qualitative approaches are likely to best for emerging fields on the frontiers of knowledge.
Relatedly, Against Method is on my reading list, which I believe basically makes the case that there is no universal scientific method (as in the quote above).
Will metascientific reforms lead us out of the replication crisis? Will they deliver us from our epistemic crisis? These questions cannot be answered. But they suggest others, equally necessary and urgent: What is a replication crisis? What is the relationship of transparency to trust? How can we measure efficiency when basic science often does not have a known objective? Metascience has been a frantically productive empirical field devoid of theory. A final question: how much longer can that last?
I think Metascience and Open Science will help with the replication crisis in established fields. Something I’ve become interested in is Scientific Prioritization which seems to lend itself to quantitative predictions (given finite resources, how can we get the most value from our research) but generally becomes difficult and qualitative as soon as your try to pin down how you actually value different outcomes. I think that addressing this point will probably require more reflection and theory.
[I didn’t get around to reading the section on Instutizaionalization of Metascience and don’t comment on that. Is IGDORE part of that process? I guess so…]
Note for others, the Metascience Symposium being referred to in the paper is: