«Assessment of researchers is necessary for decisions of hiring, promotion, and tenure. A burgeoning number of scientific leaders believe the current system of faculty incentives and rewards is misaligned with the needs of society and disconnected from the evidence about the causes of the reproducibility crisis and suboptimal quality of the scientific publication record. To address this issue, particularly for the clinical and life sciences, we convened a 22-member expert panel workshop in Washington, DC, in January 2017. Twenty-two academic leaders, funders, and scientists participated in the meeting. As background for the meeting, we completed a selective literature review of 22 key documents critiquing the current incentive system. From each document, we extracted how the authors perceived the problems of assessing science and scientists, the unintended consequences of maintaining the status quo for assessing scientists, and details of their proposed solutions. The resulting table was used as a seed for participant discussion. This resulted in six principles for assessing scientists and associated research and policy implications. We hope the content of this paper will serve as a basis for establishing best practices and redesigning the current approaches to assessing scientists by the many players involved in that process.»

Segnaliamo un articolo apparso su PLOS Biology:

Assessing scientists for hiring, promotion, and tenure

L’articolo contiene una rassegna di documenti che criticano l’attuale sistema di incentivi.  Ne pubblichiamo un ampio estratto per quanto riguarda le “conseguenze  non volute del sistema attuale” e le “soluzioni proposte”.


Larger group proposals

ACUMEN (2014)
Europe

Unintended consequences of maintaining the current assessment scheme
The report points to five problems: 1. ‘Evaluation criteria are still dominated by mono-disciplinary measures, which reflect an important but limited number of dimensions of the quality and relevance of scientific and scholarly work’;
2. ‘Increase the pressure on the existing forms of quality control and evaluation’;
3. ‘The evaluation system has not been able to keep up sufficiently with the transformations in the way researchers create knowledge and communicate their research to colleagues and the public at large’; 4. The bibliometrics in current use ‘do not produce viable results at the level of the individual researcher’;
5. The current ‘scientific and scholarly system has a gender bias’.

Proposed solutions for assessing scientists
The consortium has developed criteria and guidelines for GEP and has designed a prototype for a web-based ACUMEN performance portfolio. The GEP focuses on three indicators for academic assessment: (1) expertise, (2) outputs, and (3) impacts. The performance portfolio is divided into four parts: (1) narrative and academic age calculation, (2) expertise, (3) output, and (4) influence.

Amsterdam Call for Action on Open Science (2016)
Europe

Unintended consequences of maintaining the current assessment scheme
Conference participants argue that maintaining the current scheme will continue to create a climate of disconnect between old-world bibliometrics and newer approaches, such as a commitment to open science—‘This emphasis does not correspond with our goals to achieve societal impact alongside scientific impact’.

Proposed solutions for assessing scientists
The conference participants’ vision is for ‘New assessment, reward and evaluation systems. New systems that really deal with the core of knowledge creation and account for the impact of scientific research on science and society at large, including the economy, and incentivise citizen science’. To reach this goal, the Call for Action recommends action in four areas: (1) complete OA, (2) data sharing, (3) new ways of assessing scientists, and (4) introducing evidence to inform best practices for the first three themes. Twelve action items are proposed: to (1) change assessment, evaluation, and reward systems in science; (2) facilitate text and data mining of content; (3) improve insight into IPR and issues such as privacy; (4) create transparency on the costs and conditions of academic communication; (5) introduce FAIR and secure data principles; (6) set up common e-infrastructures; (7) adopt OA principles; (8) stimulate new publishing models for knowledge transfer; (9) stimulate evidence-based research on innovations in open science; (10) develop, implement, monitor, and refine OA plans; (11) involve researchers and new users in open science; and (12) encourage stakeholders to share expertise and information on open science.

DORA (2012)
International

Unintended consequences of maintaining the current assessment scheme
DORA points to the critical problems with using JIF as a measure of a scientist’s worth: ‘the Journal Impact Factor has a number of well-documented deficiencies as a tool for research assessment’.

Proposed solutions for assessing scientists
DORA has one general recommendation—do not use journal-based metrics, such as JIFs, as surrogate measures of the quality of individual research articles to assess an individual scientist’s contributions in hiring, promotion, or funding decisions and 17 specific recommendations, for researchers: (1) focus on content; (2) cite primary literature; (3) use a range of metrics to show the impact of your work; (4) change the culture; funders: (5) state that scientific content of a paper, not the JIF of the journal in which it was published, is what matters; (6) consider value from all outputs and outcomes generated by research; research institutions: (7) when hiring and promoting, state that scientific content of a paper, not the JIF of the journal in which it was published, is what matters; (8) consider value from all outputs and outcomes generated by research; publishers: (9) cease to promote journals by impact factor; (10) provide an array of metrics; (11) focus on article-level metrics; (12) identify different author contributions, open the bibliometric citation data; (13) encourage primary literature citations; and organizations that supply metrics: (14) be transparent; (15) provide access to data; (16) discourage data manipulation; and (17) provide different metrics for primary literature and reviews.

The Leiden Manifesto (2015)
International

Unintended consequences of maintaining the current assessment scheme
‘The problem is that evaluation is now led by the data rather than by judgement. Metrics have proliferated: usually well intentioned, not always well informed, often ill applied’.

Proposed solutions for assessing scientists
The Leiden Manifesto proposes 10 best practices: (1) quantitative evaluation should support qualitative, expert assessment; (2) measure performance against the research missions of the institution, group, or researcher; (3) protect excellence in locally relevant research; (4) keep data collection and analytical processes open, transparent, and simple; (5) allow those evaluated to verify data and analysis; (6) account for variation by field in publication and citation practices; (7) base assessment of individual researchers on a qualitative judgment of their portfolio; (8) avoid misplaced concreteness and false precision; (9) recognise the systemic effects of assessment and indicators; and (10) scrutinise indicators regularly and update them.

Wilsdon (The Metric Tide) (2015)
United Kingdom

Unintended consequences of maintaining the current assessment scheme
The report raises concerns ‘that some quantitative indicators can be gamed, or can lead to unintended consequences; journal impact factors and citation counts are two prominent examples’.

Proposed solutions for assessing scientists
The report proposes five attributes to improve the assessment of researchers: (1) robustness, (2) humility, (3) transparency, (4) diversity, and (5) reflexivity. The report also makes 20 recommendations dealing with a broad spectrum of issues related to research assessment for stakeholders to consider: (1) the research community should develop a more sophisticated and nuanced approach to the contribution and limitations of quantitative indicators; (2) at an institutional level, higher education institution leaders should develop a clear statement of principles on their approach to research management and assessment, including the role of quantitative indicators; (3) research managers and administrators should champion these principles and the use of responsible metrics within their institutions; (4) human resources managers and recruitment or promotion panels in higher education institutions should be explicit about the criteria used for academic appointment and promotion decisions; (5) individual researchers should be mindful of the limitations of particular indicators; (6) research funders should develop their own context- specific principles for the use of quantitative indicators in research assessment and management; (7) data providers, analysts, and producers of university rankings and league tables should strive for greater transparency and interoperability between different measurement systems; (8) publishers should reduce emphasis on JIFs as a promotional tool, and only use them in the context of a variety of journal-based metrics that provide a richer view of performance; (9) there is a need for greater transparency and openness in research data infrastructure; (10) a set of principles should be developed for technologies, practices, and cultures that can support open, trustworthy research information management; (11) the UK research system should take full advantage of ORCID as its preferred system of unique identifiers. ORCID IDs should be mandatory for all researchers in the next REF; (12) identifiers are also needed for institutions, and the most likely candidate for a global solution is the ISNI, which already has good coverage of publishers, funders, and research organizations; (13) publishers should mandate ORCID IDs and ISNIs and funder grant references for article submission and retain this metadata throughout the publication life cycle; (14) the use of DOIs should be extended to cover all research outputs; (15) further investment in research information infrastructure is required; (16) HEFCE, funders, HEIs, and Jisc should explore how to leverage data held in existing platforms to support the REF process, and vice versa; (17) BIS should identify ways of linking data gathered from research-related platforms (including Gateway to Research, Researchfish, and the REF) more directly to policy processes in BIS and other departments; in assessing outputs, we recommend that quantitative data— particularly around published outputs—continue to have a place in informing peer-review judgements of research quality; in assessing impact, we recommend that HEFCE and the UK HE funding bodies build on the analysis of the impact case studies from REF2014 to develop clear guidelines for the use of quantitative indicators in future impact case studies; in assessing the research environment, we recommend that there is scope for enhancing the use of quantitative data but that these data need to be provided with sufficient context to enable their interpretation; (18) the UK research community needs a mechanism to carry forward the agenda set out in this report; (19) the establishment of a Forum for Responsible Metrics, which would bring together research funders, HEIs and their representative bodies, publishers, data providers, and others to work on issues of data standards, interoperability, openness, and transparency; research funders need to increase investment in the science of science policy; and (20) one positive aspect of this review has been the debate it has generated. As a legacy initiative, the steering group is setting up a blog (www.ResponsibleMetrics. org) as a forum for ongoing discussion of the issues raised by this report.

NAS (2015)
United States

Unintended consequences of maintaining the current assessment scheme
The authors indicate that if the current system does not evolve, there will be serious threats to the credibility of science. They state, ‘If science is to enhance its capacities to improve our understanding of ourselves and our world, protect the hard-earned trust and esteem in which society holds it, and preserve its role as a driver of our economy, scientists must safeguard its rigor and reliability in the face of challenges posed by a research ecosystem that is evolving in dramatic and sometimes unsettling ways’.

Proposed solutions for assessing scientists
The authors ‘believe that incentives should be changed so that scholars are rewarded for publishing well rather than often. In tenure cases at universities, as in grant submissions, the candidate should be evaluated on the importance of a select set of work, instead of using the number of publications or impact rating of a journal as a surrogate for quality’.
Incentives should also change to reward better correction (or self-correction) of science. It includes improving the peer-review process and reducing the stigma associated with retractions.

Nuffield Council on Bioethics (2014)
UK

Unintended consequences of maintaining the current assessment scheme
The feedback received cautioned maintaining the focus on JIFs—’This is believed to be resulting in important research not being published, disincentives for multidisciplinary research, authorship issues, and a lack of recognition for non-article research outputs’.

Proposed solutions for assessing scientists
The report suggested actions for different stakeholders (funders, publishers and editors, research institutions, researchers, and learned society and professional bodies) to consider, principally, (1) improving transparency; (2) improving the peer-review process (e.g., by training); (3) cultivating an environment based on the ethics of research; (4) assessing broadly the track records of researchers and fellow researchers; (5) involving researchers in policy making in a dialogue with other stakeholders; and (6) promoting standards for high-quality science.

REWARD
International

Unintended consequences of maintaining the current assessment scheme
If the current bibliometric system is maintained, there is a real risk that scientists will be ‘judged on the basis of the impact factors of the journals in which their work is published’. Impact factors are weakly correlated with quality.

Proposed solutions for assessing scientists
The REWARD series makes 17 recommendations covering a broad spectrum of stakeholders: (1) more research on research should be done to identify factors associated with successful replication of basic research and translation to application in healthcare and how to achieve the most productive ratio of basic to applied research; (2) research funders should make information available about how they decide what research to support and fund investigations of the effects of initiatives to engage potential users of research in research prioritisation; (3) research funders and regulators should demand that proposals for additional primary research are justified by systematic reviews showing what is already known, and increase funding for the required syntheses of existing evidence; (4) research funders and research regulators should strengthen and develop sources of information about research that is in progress, ensure that they are used by researchers, insist on publication of protocols at study inception, and encourage collaboration to reduce waste; (5) make publicly available the full protocols, analysis plans or sequence of analytical choices, and raw data for all designed and undertaken biomedical research; (6) maximise the effect-to-bias ratio in research through defensible design and conduct standards, a well-trained methodological research workforce, continuing professional development, and involvement of nonconflicted stakeholders; (7) reward with funding and academic or other recognition reproducibility practices and reproducible research, and enable an efficient culture for replication of research; (8) people regulating research should use their influence to reduce other causes of waste and inefficiency in research; (9) regulators and policy makers should work with researchers, patients, and health professionals to streamline and harmonise the laws, regulations, guidelines, and processes that govern whether and how research can be done, and ensure that they are proportionate to the plausible risks associated with the research; (10) researchers and research managers should increase the efficiency of recruitment, retention, data monitoring, and data sharing in research through the use of research designs known to reduce inefficiencies, and do additional research to learn how efficiency can be increased; (11) everyone, particularly individuals responsible for healthcare systems, can help to improve the efficiency of clinical research by promoting integration of research in everyday clinical practice; (12) institutions and funders should adopt performance metrics that recognise full dissemination of research and reuse of original datasets by external researchers; (13) investigators, funders, sponsors, regulators, research ethics committees, and journals should systematically develop and adopt standards for the content of study protocols and full study reports, and for data-sharing practices; (14) funders, sponsors, regulators, research ethics committees, journals, and legislators should endorse and enforce study registration policies, wide availability of full study information, and sharing of participant-level data for all health research; (15) funders and research institutions must shift research regulations and rewards to align with better and more complete reporting; (16) research funders should take responsibility for reporting infrastructure that supports good reporting and archiving; and (17) funders, institutions, and publishers should improve the capability and capacity of authors and reviewers in high-quality and complete reporting. There is a recognition of problems with academic reward systems that appear to focus on quantity more than quality. Part of the series includes a discussion about evaluating scientists on a set of best practices, including reproducibility of research findings, the quality of the reporting, complete dissemination of the research, and the rigor of the methods used.

REF
UK

Unintended consequences of maintaining the current assessment scheme
Not being able to identify the societal value (e.g., public funding of higher education institutions and the impact of the research conducted) of academic institutions.

Proposed solutions for assessing scientists
The REF is a new national initiative to assess the quality of research in higher education institutions assessing institutional outputs, impact, and environment covering 36 fields of study (e.g., law, economics, and econometrics). Outputs account for 65% of the assessment (i.e., ‘are the product of any form of research, published, such as journal articles, monographs and chapters in books, as well as outputs disseminated in other ways such as designs, performances and exhibitions’). Impact accounts for 20% of the assessment (e.g., ‘is any effect on, change or benefit to the economy, society, culture, public policy or services, health, the environment or quality of life, beyond academia’), and the environment accounts for 15% of the assessment (e.g., ‘the strategy, resources and infrastructure that support research’).

Smaller group or individual proposals

Benedictus (UMC the Utrecht, 2016)
the Netherlands

Unintended consequences of maintaining the current assessment scheme
Focusing on meaningless bibliometrics is keeping scientists ‘from doing what really mattered, such as strengthening contacts with patient organizations or trying to make promising treatments work in the real world’

Proposed solutions for assessing scientists
To move away from using bibliometrics to evaluate scientists, the authors propose five alternative behaviours to assess: (1) managerial responsibilities and academic obligations; (2) mentoring students, teaching, and additional new responsibilities; (3) if applicable, a description of clinical work; (4) participation in organising clinical trials and research into new treatments and diagnostics; and (5) entrepreneurship and community outreach.

Edwards (2017)
US

Unintended consequences of maintaining the current assessment scheme
The authors argue that continued reliance on quantitative metrics may lead to substantive and systemic threats to scientific integrity.

Proposed solutions for assessing scientists
To deal with incentives and hypercompetition, the authors have proposed (1) that more data are needed to better understand the significance and extent of the problem; (2) that funding should be provided to develop best practices for assessing scientists for promotion, tenure, and hiring; (3) better education about scientific misconduct for students; (4) incorporating qualitative components, such as service to community, into PhD training programs; and (5) the need for academic institutions to reduce their reliance on quantitative metrics to assess scientists.

Ioannidis (2014)
US

Unintended consequences of maintaining the current assessment scheme
Currently, scientific publications are often ‘false or grossly exaggerated, and translation of knowledge into useful applications is often slow and potentially inefficient’. Unless we develop better scientific and publication practices, much of the scientific output will remain grossly wasted.

Proposed solutions for assessing scientists
The author proposes 12 best practices to achieve truth and credibility in science. These include (1) large-scale collaborative research; (2) adoption of a replication culture; (3) registration; (4) sharing; (5) reproducibility practices; (6) better statistical methods; (7) standardisation of definitions and analyses; (8) more appropriate (usually more stringent) statistical thresholds; (9) improvement in study design standards; (10) stronger thresholds for claims of discovery; (11) improvements in peer review, reporting, and dissemination of research; and (12) better training of the scientific workforce. These best practices could be used as research currencies for promotion and tenure. The author provides examples of how these best practices can be used in different ways as part of the reward system for evaluating scientists.

Mazumdar (2015)
US

Unintended consequences of maintaining the current assessment scheme
Concentrating on traditional metrics ‘can substantially devalue the contributions of a team scientist’.

Proposed solutions for assessing scientists
To assess the research component of a biostatistician as part of a team collaboration, the authors propose a flexible quantitative and qualitative framework involving four evaluation themes that can be applied broadly to appointment, promotion, and tenure decisions. These criteria are: design activities, implementation activities, analysis activities, and manuscript reporting activities.

Ioannidis PQRST
US

Unintended consequences of maintaining the current assessment scheme
The authors note, ‘However, emphasis on publication can lead to least publishable units, authorship inflation, and potentially irreproducible results’. In short, this type of assessment might tarnish science and how scientists are evaluated.

Proposed solutions for assessing scientists
To reduce our reliance on traditional quantitative metrics for assessing and rewarding research, the authors propose a best practice index—PQRST— revolving around productivity, quality, reproducibility, sharing, and translation of research. The authors also propose examples on how each item could be operationalised, e.g., for productivity; examples include number of publications in the top tier percentage of citations for the scientific field and year, proportion of funded proposals that have resulted in 1 published reports of the main results, and proportion of registered protocols that have been published 2 years after the completion of the studies. Similarly, one could count the proportion of publications that fulfill 1 quality standards; proportion of publications that are reproducible; proportion of publications that share their data, materials, and/or protocols (whichever items are relevant); and proportion of publications that have resulted in successful accomplishment of a distal translational milestone, e.g., getting promising results in human trials for interventions tested in animals or cell cultures, or licensing of intervention for clinical trials.

Nosek (2015)
US

Unintended consequences of maintaining the current assessment scheme
With the perverse ‘publish or perish’ mantra, the authors argue that authors may feel compelled to fabricate their results and undermine the integrity of science and scientists. ‘With flexible analysis options, we are more likely to find the one that produces a more publishable pattern of results to be more reasonable and defensible than others’.

Proposed solutions for assessing scientists
The authors propose a series of best practices that might resolve the aforementioned conflicts. These best practices include restructuring the current incentive/reward scheme for academic promotion and tenure, use of reporting guidelines, promoting better peer review, and journals devoted to publishing replications or statistically negative results.

eLife (2013)
US

Unintended consequences of maintaining the current assessment scheme
The editors state, ‘The focus on publication in a high impact-factor journal as the prize also distracts attention from other important responsibilities of researchers—such as teaching, mentoring and a host of other activities (including the review of manuscripts for journals!). For the sake of science, the emphasis needs to change’.

Proposed solutions for assessing scientists
To help counter these problems, the editors discuss several options, such as repositories for sharing information: Dryad for datasets; Figshare for primary research, figures, and datasets; and Slideshare for presentations. Altmetric. com, Impact Story, and Plum Analytics can be used to aggregate media coverage, citation numbers, and social web metrics. Taken together, such information is likely to provide a broader assessment of the impact of research well beyond that of the JIF.

Nature (2016)
UK

Unintended consequences of maintaining the current assessment scheme
The journal’s perspective is that ‘metrics are intrinsically reductive and, as such, can be dangerous. Relying on them as a yardstick of performance, rather than as a pointer to underlying achievements and challenges, usually leads to pathological behaviour. The journal impact factor is just such a metric’.

Proposed solutions for assessing scientists
To help combat these problems, the journal has proposed two solutions: first, ‘applicants for any job, promotion or funding should be asked to include a short summary of what they consider their achievements to be, rather than just to list their publications’. Second, ‘journals need to be more diverse in how they display their performance’.

Send to Kindle

2 Commenti

Questo sito usa Akismet per ridurre lo spam. Scopri come i tuoi dati vengono elaborati.