Experiences with the rating of publication channels for the Nordic Model. With a response to a proposal for automated ratings from Saarela et al. (2016)

Norway, Denmark and Finland use the “Norwegian model” of block-grant allocation that links national publication data to a weighted quality index of publication channels. In each country, the rating of the publication channels is the responsibility of experts in the field. Nordforsk is presently funding a Nordic collaboration project where we merge the lists, compare ratings and study the possibility of sharing information, thereby reducing the workload and improving the data quality. In a parallel project in Finland, Saarela et al. (2016) have used new techniques to demonstrate that Scopus-based IPP, SNIP and SJR, in combination with the Danish and Norwegian levels, allow for good prediction of the Finnish expert-ratings. They suggest that automatic rules could replace or assist the expert qualitative judgment to improve the transparency and objectivity and to save man-hours and money in the Norwegian model. In this presentation, we will present the experiences with expert-based assessments and Nordic collaboration so far and discuss the following topics: 1. Why involve expert panels in the evaluation and what is their task?2. How large part of the publication channel lists is covered by metrics for journals?3. Under what conditions can expert based ratings be expected to be correlated with citation based journal metrics. Where do we find exceptions? 4. What metrics are already used to inform the panels, and what other information could be appropriate?5. What are the main challenges with involving panels in the rating process?6. To what degree would automated rankings improve the evaluation process?7. To what degree would the creation of Nordic versus national panels improve the process?ReferencesAhlgren, P., Corriander, C., & Persson, O. (2012). Field normalized citation rates, field normalized journal impact and Norwegian weights for allocation of university research funds. Scientometrics, 92(3), 767–780.Saarela, M., Kärkkäinen, T., Lahtonen, t., Rossi, T. (2016) “Expert-based versus citation-based ranking of scholarly and scientific publication channels”, Journal of Informetrics, 10 (2016), 693–718.