Bibliometri och modeller för resursfördelning Gustaf Nelhans, Pieta Eklund University of Borås Professorskollegium December 12th, 2014 1
Aim 1. to map and describe various bibliometric models and indicators that currently are applied across a selection of Swedish universities. What? Where? How? 2. to invite for critical discussion, the pros and cons and the relative value of using such models for the allocation of funds within the academia.
Why allocate resources based on indicator models? Research policy needs tools to allocate funds without steering research directly. Though, there is also a tradition of directly funding sector research (not treated here). Evaluation based on quality Based on the Mertonian CUDOS norms Prerequisites: objective measure unobtrusive measures Quantitative models are (quite) easy to operationalize.
Possible allocation system models Australia: count of ISI-publications Thought to lead to impoverishment of research and salami slice publishing UK: RAE, panel based evaluation. From 2014: REF peer review : expensive and resource-intensive. Norway, system based on publication channel and quality (impact factor, level of internationalization) Measures at the publisher- or journal level. Average, not real impact. Sweden: Field factor and Waring distributions Only WoS-indexed journal articles low coverage in some subjects.
Sweden: present performance based funding model (2008/2012) Basic funding (80 %) Performance based share (20 %) 1. External funding (50 %) 2. Publication performance (50 %) as normalized data for publication & citation rates Performance based, (20 %) Publications & citations, (50 %) Main features Four year moving average Author fractionalization Normalization: Publications: Waring Distributions Citations: Field Normalized Citation Level Additional Weighting Medicine + Technology: 1.0; Science: 1.5; Social Sci + Humanities 2.0; Other: 1.1 Basic funding, (80 %) External funding, (50 %) Sources: Prop. 2008/09:50. A boost for research and innovation; Prop. 2012/13:30. Research and innovation Utbildningsdepartementet [Ministry of Education and Research]. Stockholm: Fritzes.s 5
Motiv till regeringens egna viktning Vi har gjort en hel del körningar när det gäller vilka effekter olika varianter av anslagssystem skulle ge. / / Jag kan bara säga att ett stort problem för mig var att detta system rent och naket skulle slå negativt mot humaniora och samhällsvetenskap. Vi införde denna fördubblingsfaktor för att se till att ha en kupa, en skyddande hand, inte minst över humaniora. Vi får väl se hur det blir. Vår bedömning är att denna multiplikator 2 är tillräcklig för att skydda humaniora. Jag kan inte garantera att det är så. Naturligtvis får vi följa det. Men jag tror att det kommer att vara positivt. - (Applåder) Lars Leijonborg, utdrag ur riksdagsdebatten inför omröstning om proposition 2008/09:50, Internet: http://www.riksdagen.se/webbnav/index.aspx?nid=101&bet=2008/09:64 (hämtad 2010-10-14) 6
Research and innovation bill 2013-2016 Research and innovation (Prop. 2012/13:30) Key points: Performance based share doubled (20 %) from 2014 Peer review instead of bibliometrics? Cf, the British RAE/REF system or Univeritetskanslerämbetets kvalitetsutvärderingssystem för högre utbildning But, implemented not before 2018 Meaning: two general elections (2014, 2018) one innovation bill (expected in 2016) will pass before the new model is implemented. 7
Vetenskapsrådets FOKUS-modell Forskningskvalitetsutvärdering i Sverige VR 2014 December (soon) Peer review 6-year interval Vetenskaplig kvalitet 15% Kvalitetsutvecklande faktorer Genomslag utanför akademin 15% 70% Vetenskapsrådet (2014b). Modell för fördelning av forskningsresurser till lärosäten (Fokus). http://vr.se/omvetenskapsradet/regeringsuppdrag/ regeringsuppdrag/modellforfordelningavforskningsresursertilllarosaten.4.5a947f0d145b21c1709a7.html
Norwegian model (variants also used in Denmark and Finland) Two dimensions: publication channel level of the channel (0: not scientific) 1: ordinary scientific 2: highly prestigious publication channels Publikationskanal Nivå 1 (80%) Nivå 2 (20%) Monografi 5 8 Artikel i periodika eller serier 1 3 Artikel i antologi 0,7 1 Grupp A: Grupp B: Grupp Arguably: C Secondary peer review Impact factor based system Nivåindelning naturvetenskap, medicin ekonomi, informationsvet en-skap, teknik mjuk samhällsvetens kap humaniora Publikation er som kan nomineras Tidskrifter Alla Alla Kvalitetsmått Impact factor + substitution - - + supplement - - + supplement Indikator Doktorgradskandidater 30 % EU-midler 20 % Forskningsmdsmidler 20 % Publiseringspoeng 30 % Vekt
Comparison Sw/No model Swedish model Transparency: Variables in the calculated model are relative Selection: Only published material that is indexed in WoS ISI Measure of quality Citation measures, field normalized Source of data: Already available data (WoS ISI) Norwegian modell Transparency: Pre-determined point system Selection: More research channels (Monographs, conf. Proc, journal articles) Measure of quality: Secondary peer review Sources of data: An authorization index must be created (Cristin, NSB) and publication lists must be updated..
The citation Quality indicator or measure of performance?
The citation as an indicator of quality Eugene Garfield (1963): One purpose of this communication is to record my forewarning concerning the possible promiscuous and careless use of quantitative citation data for sociological evaluations, including personel and fellowship selection Impact is not the same as importance or significance At the same time, he also argued SCI to be used to evaluate Journal performance Journal Impact Factor (JIF) Garfield, E. 1963. Citation Indexes in Sociological and Historical Research. American Documentation 14:289-91. 16
Kessler and Heart The warning reads: CAUTION! Any attempt to equate high frequency of citation with worth or excellence will end in disaster; nor can we say that low frequency of citation indicates lack of worth. Kessler, M.M., and F. E. Heart (1962) Concerning the probability that a given paper will be cited, Report (Massachusetts Institute of Technology, Cambridge).
Argument for the use of citation analysis as a quality indicator: The observation that citations indicate use, and therefore usefulness as well as impact, is the basic argument for using them as an indicator of quality. Gläser, Jochen, & Grit Laudel. 2008. The Social Construction of Bibliometric Evaluations. In The Changing Governance of the Sciences, edited by R. Whitley and J. Gläser. Dordrecht: Springer. 101-123.
Citations as performativity - being cited Traditionally: Citations as reward, (passive) Citation Index as representation of publication patterns My proposal: Performativity of being cited What research work do citations do? Citations as construction and epistemological networking The citation viewed as an outcome of active achievement or performance Reflexive actors (researchers are active) Citation index as a performative arena for publishers, authors, citers, publications and articles; indeed the whole citation culture (Wouters, 1999) Authors actively position themselves by choosing journal/field to publish in & research problems to publish on Making themselves cite-able 20
This study: Goal-oriented selection of 13 universities at different levels and in different locations. + Borås = 14 HEI bodies Spread: Size Geographic Large coverage/specialized universities Different sectors Qualitative survey Additionally: en sökning på dessa lärosätens hemsidor, samt fria sökningar på sökord som resursfördelning, fördelningsmodell, prestationsbaserad fördelning, bibliometri, fakultetsmedel, samt lärosätenas namn, namn på fakulteter och andra organisationsformer
Themes 1. General description of models used 2. Amount of funds that are allocated, as well as on what level within the HEI allocation is performed 3. Description of how co-authorship is treated 4. Planned changes in the allocation models 5. Ongoing discussions within the university
Preliminary findings - overview All universities in our set with the exception of Chalmers use bibliometric measures to some extent for resource allocation at one or several levels. The types of measures and models used differs considerably, but two types stand out: Actual impact calculated as amount of publications and citations (as is used in the Swedish national allocation model).) Number of publications combined with a evaluation of the publication channels average impact (as is used in the Norwegian model). State of the art: Field normalization (in both models) The largest and most diversified universities often use a range of measurements depending on faculty.
Participants, bibliometric model Bibliometric distribution model Uses model Pub based Chalmers tekniska högskola No Point-based (NO) Citation based (SW) Combined Göteborgs universitet Yes X X Kungliga tekniska Högskolan Yes X Linköpings universitet Yes X Linnéuniversitet Yes x Luleå Tekniska universitet Yes X Lunds universitet Stockholms universitet Yes X Yes Sveriges lantbruksuniversitet Yes X Högskolan i Borås Yes X Högskolan i Halmstad Yes X Högskolan i Skövde i.u.* Malmö högskola Yes X Mälardalens högskola Yes X
Impact: Högskolan i Halmstad Tabell 4: Poängsättning för publiceringstyper vid Högskolan i Halmstad Publikationstyp Poäng A Artikel i tidskrift + artikel 8 forskningsöversikt B Konferensbidrag 1 C Kapitel i bok + del av antologi 2 D Bok (monografi) 8
Konstnärlig verksamhet Tabell 3: Poängsättning för publiceringstyper vid konstnärlig fakultet vid Göteborgs universitet (UR: Regler för registrering av forskning och konstnärliga verk i GU:s open-access-databaser, 2012) Publiceringsform Bok (eller motsvarande) utgiven på nationellt eller internationellt förlag Konstnärligt arbete, refereegranskat Artikel, refereegranskad (vetenskaplig/konstnärlig) Konferensbidrag (vetenskapligt/konstnärligt) refereegranskat Artikel (vetenskaplig/konstnärlig) Forskningsöversikt Kapitel i bok (eller motsvarande) utgiven på nationellt eller internationellt förlag. Även redaktörskap för bok Konstnärligt arbete Poäng 5p 5p 3p 2p 1p 1p 1p 1p
Karolinska institutet: Combined indicators Summa Journal Impact Factor, JIF Tidskriftskvalitet (kvantitetsberoende & kvalitetsberoende, ej fältnormerad) 50 procent Artikelkvalitet Summa citeringar, C (kvantitetsberoende & kvalitetsberoende, ej fältnormerad) 20 procent 5-year average Antal publikationer mats inte. Fältnormerad Citeringsgrad, Cf (kvantitetsberoende & kvalitetsberoende, fältnormerad) 30 procent Externa medel Bibliometriskt utfall Antal disputationer Antal post-docmånader
Levels Tabell 6: Andel fördelade medel baserat på bibliometriska indikatorer, samt nivå inom lärosätet på vilken fördelning baserat på bibliometriska indikatorer sker. * i.u.= ingen uppgift. Lärosäte Share of bibliometrics Nivå 1 (Fakultet) Nivå 2 (Institution) Nivå 3 (Individ) Chalmers tekniska högskola - - - - Göteborgs universitet 7,5 % x x Kungliga tekniska högskolan ett par % x x Linköpings universitet i.u. x Linnéuniversitet 2,5 % x x Luleå tekniska universitet (42/45 milj kr) x x Lunds universitet 28 % vid Samfak x Stockholms universitet i.u. x Sveriges lantbruksuniversitet 5 % x Högskolan i Borås 15 % x Högskolan i Halmstad ~25 % x Högskolan i Skövde i.u. i.u. Malmö högskola i.u. x x Mälardalens högskola 9 % x
Individual models Umeå universitet sedan 2008 tillämpats vid humanistisk fakultet individuell komponent forskare och lärare själva ansöker om medel i konkurrens med sina kolleger Publiceringsmått uppfattas som viktiga hos forskarna, men andelen peer reviewad internationell publicering spelar liten roll för utfallet. Påfallande många anser dock att systemet har påverkat arbetsklimatet negativt många upplever en ökad individuell stress och press i arbetet. (Sjögren, 2011) Linnéuniversitetet 2,5 % av medlen fördelas på individnivå fältjusterad publikationspoäng Publikationspoäng motsvarande > 8 000 (- 150 000) kr: Fördelas direct till forskaren Publikationspoäng < 8 000: Fördelas till institutionen. 8 tkr= skamgräns? Excellenspott: 20 procent av forskarna med högst publikations-poäng erhåller därutöver 15 000 kr per individ. Luleå tekniska univ Price tag: Level 1: 35 000 kr Level 2 or indexed in WoS: 70 000 kr
Author fractions 50 % yes 50 % no Usual models: 1/n; sometimes down to 1/10 No: Each author receives full credit Fractionalization within the university. External authors not counted: E.g. 3 authors, two internal + one external: -> Internal authors receives 50 % each, external author is discarded..
Planned changes Sahlgrenska akademin: scaling down performance based model (2014). Linköping University: discusses the introduction of a more concisive model: e.g. (field normalized citation measures) KTH: Too much lag, needs a quicker model. Evaluates a point based model using field normalized journal impact (journals and conference proceedings). One-year window. Chalmers: A performance based allocation model has been launched since the survey was made.
Discussions within the universities hur publikationer poängsätts de olika disciplinernas särart att redaktörskap inte räknas att tröskelvärdet för synlighet uppfattats som för högt vad som skall bedömas som kvalitetsgranskat att förtydliga riktlinjerna för hur publikationer ska räknas möjligheten att låsa föregående års publikationer för att underlätta evalueringsarbetet framöver i vilka tidskrifter forskare uppmanas att publicera sin forskning att öka incitamenten för kvalitetsutveckling att förbättra utfallet i regeringens kvalitetsbaserade resurstilldelningsmodell
Discussion Bibliometric indicators: historical merits Averaging factors tend to limit impact of changes -> leads to long lag Effects on innovative research? Resource allocation models are (potentially) performative on researchers At the governing level: audit society, the fear of New Public Management (NPM) HEI:s risk loosing self government, when decisions are allocated to external funding models. Hands tied What is the role of those who are put to do the evaluations? Bibliometrics function often based in the library.
Downside of the performative idiom curriculum vitae AND h-index Gaming the system Techniques self (collegue) citation editor coercion citation cartels Research policy advice: Division of Analysis and Evaluation, GU In response to university rankings: another way of advancing on the list would be to appoint highly cited researchers, since they bring with them their earlier citations (Gunnarsson 2013, my translation) False! 39
Discussion (meta level) Consequences for: Research policy: the occurrence of bibliometric models have been regarded as a supposedly objective tool to tap the research system for information about its intrinsic qualities, but without influencing the research analysed. Unobtrusiveness questioned Research: citation based model used in Sweden has suited some disciplines better, while others fared worse. the introduction of performance-based models created incentives for researchers to publish according to the yardstick used Researchers: impact down the hierarchy, as performance-based models have trickled down at all levels in the research practice. In conclusion, the impact on individual researchers is discussed as they grapple with adapting their performance to different and sometimes contradictory quantitative benchmarks. 40
University of Borås in the Resource allocation model in Sweden 42
43
Renegotiated shares of Gov t performance based funding (2014) Larger universities: Smaller universities: 6.19 % University of Gothenburg ~10 % University of Borås 0.23 % 44
Research groups at HB
Students impressions Comments: Language limitations. Limitations within the subject area. Suboptimal pointwise, research wise: unknown. They could have published in more scientific channels, that are accepted by the Norwegian list.
General conclusion Bibliometrics in research evaluation: Quantitative or Qualitative solutions? Prevalent both in citation & impact factor based models. Field normalization and other bibliometric techniques solves quantitative aspects, but what about qualitative differences in citation practices? Individual performativity incentives to publish E.g. being cited how well researchers make themselves citeable in citation based metrics. Citedness 47
Tack! g u s t a f. n e l h a n s @ h b. s e http://www.adm.hb.se/~gune