“There is a serious danger that undue emphasis on bibliometric indicators will not only fail to reflect correctly the quality of research, but may also hinder the appreciation of the work of excellent scientists outside the mainstream; it will also tend to promote those who follow current or fashionable research trends, rather than those whose work is highly novel and which might produce completely new directions of scientific research. Moreover, over- reliance on citations as a measure of quality may encourage the formation of aggregates of researchers (or “citation clubs”) who boost each others citation metrics by mutual citation. It thus becomes important to concentrate on better methods of evaluation, which promote good and innovative scientific research. […] Evaluations must be based under all circumstances on expert assessment of scientific content, quality and excellence.” Una presa di posizione netta, sotoscritta nel 2017 da tre accademie delle scienze: Académie des Sciences, Leopoldina e Royal Society 2017. Da anni, i valutatori di stato italiani e la CRUI si tappano le orecchie, ma sull’inopportunità e i pericoli di un uso automatizzato di indicatori bibliometrici nelle valutazioni individuali dei ricercatori e dei singoli lavori scientifici c’è un consenso pressoché universale, testimoniato da diverse dichiarazioni sottoscritte da autorevoli organismi scientifici, agenzie di valutazione, premi Nobel e così via. Di seguito, pubblichiamo una antologia, necessariamente incompleta, di queste prese di posizione pubbliche. Speriamo che possa servire da promemoria al neo-Ministro, alla CRUI e – perché no? – anche all’ANVUR, il cui direttivo sarà presto rinnovato per quattro settimi. Se le cattive pratiche valutative, promosse in Italia dai più alti livelli istituzionali, continueranno a infliggere danni duraturi alla ricerca italiana, che almeno ci venga risparmiato l’alibi del “non sapevamo”.
Alcuni degli estratti dell’antologia:
- What do Nobel Laureates think of impact factors?, Nobel Foundation
- Lettera alla ministra Valeria Fedeli, firmata dai Nobel per la Fisica T. Kajita, K.S. Thorne, R. Weiss, e da altri otto scienziati, 2017
- Citation Statistics, International Mathematical Union (IMU) in cooperation with the International Council of Industrial and Applied Mathematics (ICIAM) and the Institute of Mathematical Statistics (IMS) 2008.
- Du Bon Usage de la Bibliometrie pour l’Évaluation Individuelle des Chercheurs”, Institut de France, Académie des Sciences
- Code of Practice, European Mathematical Society
- On the use of bibliometric indices during assessment, European Physical Society
- San Francisco Declaration On Research Assessment (DORA), 2013 (sottoscritta da più di 1800 organizzazioni, tra cui riviste come Science, Plos e PNAS, atenei come il King’s College London e agenzie di valutazione come l’inglese HEFCE).
- Appropriate Use of Bibliometric Indicators for the Assessment of Journals, Research Proposals, and Individuals, IEEE 2013 (l’IEEE, con più di 400.000 membri in più di 160 nazioni è la più grande società scientifica in ambito tecnologico).
- Metric Tide: Report of the Independent Review of the Role of Metrics in Research Assessment and Management, HEFCE 2015.
1. Articles in scientific journals
[1.1] D.L. Parnas, “Stop the Numbers Game – Counting papers slows the rate of scientific progress,” Communications of the ACM, Vol. 50, No. 11, 2007, pp. 19-21.
“The widespread practice of counting publications without reading and judging them is fundamentally flawed for a number of reasons: It encourages superficial research … overly large groups … repetition … small, insignificant studies … publication of half-baked ideas.
Evaluation by counting the number of published papers corrupts our scientists; they learn to “play the game by the rules.” Knowing that only the count matters, they use the following tactics: Publishing pacts. … Clique building … Anything goes … Bespoke research …Minimum publishable increment (MPI). ….Organizing workshops and conferences …
Those who want to see computer science progress and contribute to the society that pays for it must object to rating-by-counting schemes every time they see one being applied”
[1.2] D.N. Arnold (past president of the Society for Industrial and Applied Mathematics), “Integrity under attack: The state of scholarly publishing”, SIAM News. Vol. 42, No. 10, December 2009, pp. 1-4.
“The next time you are in a situation where a publication count, or a citation number, or an impact factor is brought in as a measure of quality, raise an objection. Let people know how easily these can be, and are being, manipulated. We need to look at the papers themselves, the nature of the citations, and the quality of the journals.”
[1.3] D.N. Arnold and K. K. Fowler, “Nefarious Numbers”, Notices of the American Mathematical Society, Vol. 58, No. 3, March 2011, pp. 434-437.
“Despite numerous flaws, the impact factor has been widely used as a measure of quality for jour- nals and even for papers and authors. This creates an incentive to manipulate it. Moreover, it is possible to vastly increase impact factor without increasing journal quality at all. … The cumulative result of the design flaws and manipulation is that impact factor gives a very inaccurate view of journal quality. More generally, the citations that form the basis of the impact factor and various other bibliometrics are inherently untrustworthy.”
[1.4] A. Molinié and G. Bodenhausen, “Bibliometrics as Weapons of Mass Citation”, Chimia 64 No. 1/2 (2010) 78–89
“Just as the ‘value’ of financial products is assessed by irresponsible ranking agencies, the value of scientific research is as- sessed by ill-conceived parameters such as citation indices, h-factors, and worst of all, impact factors of journals… ‘Judging the ability of a scientist by his h- factor amounts to choosing wine according to the price of the bottle, Swiss cheese by measuring the size of its holes, and choco- late by its sugar content.’”
[1.5] S. Bednarek, The focus on bibliometrics makes papers less useful, Nature News 517.7534 (2015): 245
“Science managers and politicians seem especially fond of such ways to assess ‘scientific quality’. But many scientists also accept them, and use them in hiring and funding decisions. They are drawn to the alleged objectivity of bibliometrics. Indeed, one sometimes hears that scientists should be especially ready to apply scientific methods to their own output. However, scientists will also be aware that no good science can be built on bad data,[…]”
[1.6] Biagioli, M. (2016). Watch out for cheats in citation game. Nature News, 535(7611), 201.
“Many academic fraudsters aren’t aiming for a string of high-profile publications. That’s too risky. They want to produce — by plagiarism and rigging the peer-review system — publications that are near invisible, but can give them the kind of curriculum vitae that matches the performance metrics used by their academic institutions. They aim high, but not too high.
And so do their institutions — typically not the world’s leading universities, but those that are trying to break into the top rank. These are the institutions that use academic metrics most enthusiastically, and so end up encouraging post-production misconduct. The audit culture of universities — their love affair with metrics, impact factors, citation statistics and rankings — does not just incentivize this new form of bad behaviour. It enables it.”
2. Articles in newspapers
[2.1] D.D. Guttenplan, Questionable Science behind Academic Rankings, New York Times, November 14, 2010
“.. the list [the 2010 Times Higher Education ranking of world universities] also ranked Alexandria [the Egyptian university] fourth in the world in a subcategory that weighed the impact of a university’s research — behind only Caltech, M.I.T. and Princeton, and ahead of both Harvard and Stanford. … Dr. Hazelkorn also questioned whether the widespread emphasis on bibliometrics — using figures for academic publications or how often faculty members are cited in scholarly journals as proxies for measuring the quality or influence of a university department — made any sense. “I understand that bibliometrics is attractive because it looks objective. But as Einstein used to say, ‘Not everything that can be counted counts, and not everything that counts can be counted.”
[2.2] D. Colquhoun, “Publish-or-perish: Peer review and the corruption of science,” The Guardian, September 5, 2011
“To have “written” 800 papers is regarded as something to boast about rather than being rather shameful. … The way to improve honesty is to remove official incentives to dishonesty.”
3. Institutional reports and statements
[3.1] Joint Committee on Quantitative Assessment of Research, Citation Statistics – A report from the International Mathematical Union (IMU) in cooperation with the International Council of Industrial and Applied Mathematics (ICIAM) and the Institute of Mathematical Statistics (IMS), Robert Adler, John Ewing (Chair), Peter Taylor, released: 6 November 2008, corrected version: 6 December 08
“Thus, while it is incorrect to say that the impact factor gives no information about individual papers in a journal, the information is surprisingly vague and can be dramatically misleading….Once one realizes that it makes no sense to substitute the impact factor for individual article citation counts, it follows that it makes no sense to use the impact factor to evaluate the authors of those articles, the programs in which they work, and (most certainly) the disciplines they represent.”
[3.2] Higher Education Funding Council for England (HEFCE), Report on the pilot exercise to develop bibliometric indicators for the Research Excellence Framework , released: September 2009.
“Key points. 8. Bibliometrics are not sufficiently robust at this stage to be used formulaically or to replace expert review in the REF. However there is considerable scope for citation information to be used to inform expert review. 9. The robustness of the bibliometrics varies across the fields of research covered by the pilot, lower levels of coverage decreasing the representativeness of the citation information. In areas where publication in journals is the main method of scholarly communication, bibliometrics are more representative of the research undertaken.“
[3.3] House of Commons, Science and Technology Committee, Peer review in scientific publications, Eighth Report of Session 2010–12, released: 28 July 2011.
“David Sweeney [Director HEFCE]: With regard to our assessment of research previously through the Research Assessment Exercise and the Research Excellence Framework, we are very clear that we do not use our journal impact factors as a proxy measure for assessing quality. Our assessment panels are banned from so doing. That is not a contentious issue at all.
Sir Mark Walport: I would agree with that. Impact factors are a rather lazy surrogate. We all know that papers are published in the “very best” journals that are never cited by anyone ever again. Equally, papers are published in journals that are viewed as less prestigious, which have a very large impact. We would always argue that there is no substitute for reading the publication and finding out what it says, rather than either reading the title of the paper or the title of the journal.
Professor Rylance: I would like to endorse both of those comments. I was the chair of an RAE panel in 2008. There is no absolute correlation between quality and place of publication in both directions. That is you cannot infer for a high-prestige journal that it is going to be good but, even worse, you cannot infer from a low-prestige one that it is going to be weak. Capturing that strength in hidden places is absolutely crucial.
Q256 Stephen Mosley: … a concern that the Research Excellence Framework panels in the next assessment in 2014 might not operate in the same way. Can you reassure us that they will be looking at and reading each individual paper and will not just be relying on the impact?
David Sweeney: I can assure you that they will not be relying on the impact. The panels are meeting now to develop their detailed criteria, but it is an underpinning element in the exercise that journal impact factors will not be used. I think we were very interested to see that in Australia, where they conceived an exercise that was heavily dependent on journal rankings, after carrying out the first exercise, they decided that alternative ways of assessing quality, other than journal rankings, were desirable in what is a very major change for them, which leaves them far more aligned with the way.”
[3.4] Kim Carr (Australian Minister for Innovation, Industry, Science and Research), Ministerial statement to the Senate Economics Legislation Committee – Improvements to Excellence in Research for Australia (ERA), May 30, 2011.
“There is clear and consistent evidence that the [journal] rankings were being deployed inappropriately within some quarters of the sector, in ways that could produce harmful outcomes, and based on a poor understanding of the actual role of the rankings. One common example was the setting of targets for publication in A and A* journals by institutional research managers.In light of these two factors – that ERA could work perfectly well without the rankings, and that their existence was focussing ill-informed, undesirable behaviour in the management of research – I have made the decision to remove the rankings, based on the ARC’s expert advice.”
[3.5] Code of Practice – European Mathematical Society, p. 5
“1. Whilst accepting that mathematical research is and should be evaluated by appropriate authorities, and especially by those that fund mathematical research, the Committee sees grave danger in the routine use of bibliometric and other related measures to assess the alleged quality of mathematical research and the performance of individuals or small groups of people.
2. It is irresponsible for institutions or committees assessing individuals for possible promo- tion or the award of a grant or distinction to base their decisions on automatic responses to bibliometric data.”
[3.6] On the use of bibliometric indices during assessment – European Physical Society, p. 2
“The European Physical Society, in its role to promote physics and physicists, strongly recommends that best practices are used in all evaluation procedures applied to individual researchers in physics, as well as in the evaluation of their research proposals and projects. In particular, the European Physical Society considers it essential that the use of bibliometric indices is always complemented by a broader assessment of scientific content taking into account the research environment, to be carried out by peers in the framework of a clear code of conduct.”
[3.7] Du Bon Usage de la Bibliometrie pour l’Évaluation Individuelle des Chercheurs”- Institut de France, Académie des Sciences, p. 5
“Any bibliometric evaluation should be tightly associated to a close examination of a researcher’s work, in particular to evaluate its originality, an element that cannot be assessed through a bibliometric study.”
[3.8] DORA (la San Francisco Declaration on Research Assessment – http://am.ascb.org/dora/) è stata sottoscritta da 407 organizzazioni (comprese riviste come Science, Plos e PNAS) e 9.492 individui, vedi anche https://www.roars.it/online/dora/.
“1. Avoid using journal metrics to judge individual papers or
individuals for hiring, promotion and funding decisions.
2. Judge the content of individual papers and take into
account other research outputs, such as data sets, software
and patents, as well as a researcher’s influence on policy
Di seguito alcune delle 407 organizzazioni che hanno sottocritto DORA:
– American Association for the Advancement of Science (AAAS)
– American Society for Cell Biology
– British Society for Cell Biology
– European Association of Science Editors
– European Mathematical Society
– European Optical Society
– European Society for Soil Conservation
– Federation of European Biochemical Societies
– Fondazione Telethon
– Higher Education Funding Council for England (HEFCE)
– Proceedings of The National Academy Of Sciences (PNAS)
– Public Library of Science (PLOS)
– The American Physiological Society
– The Journal of Cell Biology
– Institute Pasteur
– https://www.roars.it 🙂
– CNRS – University Paris Diderot
– INGM, National Institute of Molecular Genetics; Milano, Italy
– Université de Paris VIII, France
– University of Florida
– The European Association for Cancer Research (EACR)
– Ben-Gurion University of the Negev
– Université de Louvain
[3.9] IEEE: “Any journal-based metric is not designed to capture qualities of individual papers and must therefore not be used as a proxy for single-article quality or to evaluate individual scientists”
(IEEE statement on correct use of bibliometrics, http://www.ieee.org/publications_standards/publications/pubnews/vol6issue3/vol6_issue3_index.html
[3.10] Metric Tide: Report of the Independent Review of the Role of Metrics in Research Assessment and Management, HEFCE 2015.
“Analysis concluded that no metric can currently provide a like-for-like replacement for REF peer review”
“In assessing research outputs in the REF, it is not currently feasible to assess research outputs or impacts in the REF using quantitative indicators alone”
[3.11] Académie des Sciences, Leopoldina e Royal Society 2017:
“There is a serious danger that undue emphasis on bibliometric indicators will not only fail to reflect correctly the quality of research, but may also hinder the appreciation of the work of excellent scientists outside the mainstream; it will also tend to promote those who follow current or fashionable research trends, rather than those whose work is highly novel and which might produce completely new directions of scientific research. Moreover, over- reliance on citations as a measure of quality may encourage the formation of aggregates of researchers (or “citation clubs”) who boost each others citation metrics by mutual citation. It thus becomes important to concentrate on better methods of evaluation, which promote good and innovative scientific research. […] Evaluations must be based under all circumstances on expert assessment of scientific content, quality and excellence. Publications that are identified by the authors as their most important work, including major articles and books, should receive particular attention in the evaluation. The simple number of publications should not be a dominant criterion.”
4. Premi Nobel
[4.1] R.R. Ernst (Nobel prize in Chemistry), “The Follies of Citation Indices and Academic Ranking Lists A Brief Commentary to ‘Bibliometrics as Weapons of Mass Citation’,” Chimia, Vol. 64, No. 1/2, 2010, p. 90.
“The present hype of bibliometry made it plainly obvious that judging the quality of science publications and science projects by bibliometric measures alone is inadequate, and reflects the inadequacy of science man- agement regimes staffed by non-scientific administrators or by pseudo-scientists who failed to develop their own personal judgment.”
[4.2] What do Nobel Laureates think of impact factors?, Nobel Foundation
La Fondazione Nobel ha diffuso un video intitolato “The research counts, not the journal!” dove alcuni premi Nobel per la medicina prendono una posizione netta contro l’uso degli impact factors per valutare la qualità di una ricerca. Si noti l’uso del plurale: non solo l’impact factor di Clarivate Analytics, ma qualsiasi indicatore riferito alla popolarità delle riviste. Il video è la sintesi di una intera sezione del canale youtube della Fondazione Nobel che ospita dodici video di altrettanti premi Nobel, dedicati proprio alla critica dell’uso di bibliometria e impact factor.
[4.3] Lettera alla ministra Valeria Fedeli, firmata dai Nobel per la Fisica T. Kajita, K.S. Thorne, R. Weiss, e da altri otto scienziati, 2017
“LISA Pathfinder was led by an Italian scientific team whose members, we understand, would not qualify any more for their current positions, and actually for any permanent position in Italian universities, according to the recently revised Italian regulations. We understand that this paradoxical situation originates from a blind evaluation algorithm. […] We are confident that you, your Honor, in your wisdom, will find ways to correct such an automatic mechanism that, if unchanged, may endanger this great scientific tradition”