Research data must be managed and assessed responsibly
Extensive amounts of research information from various countries and disciplines are gathered in numerous databases, systems, and digital services. They assist researchers and other users in locating research information as well as in planning, monitoring, and enhancing the operations of research organizations, and in assessing research. The progress has been rapid in recent decades.
Over the past 20 years, all higher education institutions and most research institutes in Finland have adopted a CRIS system (Current Research Information System) to support the management of research information. These systems gather information on publications, research data, researchers, organizations, research groups, projects, and other research activities.
In Finland, all universities use a system produced by a commercial operator: Elsevier Pure, Clarivate Analytics Converis or Solenovo SoleCRIS. Universities of applied sciences and some research institutes use the Justus service developed by CSC to enter and manage similar information.
CRIS systems collect research data
Traditionally, organizations’ CRIS systems have had two tasks. They display and present the research produced by the organization on the website or on a portal included in the system. Furthermore, they offer numerical data on research to assist in monitoring, decision-making, and planning within the organization. This information can be combined with other information outlining the organization’s activities.
Several countries have also introduced national CRIS systems that either act as primary data collection systems or compile organizational data. In Finland, the national research information hub was introduced in 2020. It compiles information from research organizations and funders, and the information is available to everyone on the Research.fi website. Its objectives include serving as a central source for research information, while also facilitating the smooth transfer of information between research institutions and funders.
Attention to responsible assessment of research
The systematic gathering of research information and the development of CRISes have facilitated the development of different monitoring indicators, allowing for an increasingly quantitative assessment of research activities. That said, it is essential that the collection and use of information is conducted in a responsible way especially when supporting decision-making. The scientific community has become aware of the challenges of using quantitative indicators, leading to the development of various principles and guidelines for responsible assessment of research in the last ten years.
The latest and most significant addition to this discussion is the European Agreement for Reforming Research Assessment (the so-called CoARA agreement). It has been possible to commit to the agreement since September 2022. By the end of April 2024, more than 700 organizations had signed it globally. In Finland, all universities and most universities of applied sciences, the Research Council of Finland and a few other organizations have committed to the agreement.
Upon signing the agreement, the organization commits to considering research outputs, practices and activities in a versatile manner when assessing research or researchers. Broad examination of research activities requires that the assessment is primarily based on qualitative consideration. Peer review plays a central role.
In Finland, the funding model encourages open reporting of publications
While it is clear that comprehensive and high-quality information will be needed for assessing and monitoring research in the future, the various declarations and principles of responsible assessment provide little guidance on how research data should be collected and managed. Publication data is readily accessible across multiple international databases and are already systematically compiled within organizations’ CRIS systems and at the national level. In Finland, this has been facilitated by the fact that publications play a central role in the funding model of higher education institutions. This provides a strong incentive for higher education institutions to report their publications in a comprehensive manner. Research data is also collected in different services and archives, although the field is still fragmented and the knowledge base is not very comprehensive.
However, the broader assessment of research activities also encompasses the societal impact of research and other scientific merits, which are described by a range of societal and scientific expert activities, statements, and achievements in teaching. However, this type of data cannot be readily found in any database. In many cases, only the researcher is aware of such information. However, most CRIS systems enable the collection of such data. In fact, this type of data could prove to be crucial in the future when building a more diverse knowledge base to support the responsible assessment of research.
The transparency of data collection and source systems is important
The principles of responsible assessment include that the knowledge base of research assessment must be built on an open, transparent and uncomplicated data collection process, and the targets of assessment must be able to verify the data collected on them. CRIS systems excel in supporting these principles when compared to systems such as Google Scholar or OpenAIRE, which collect information as comprehensively as possible, but there is no information available on the coverage or data collection principles of the source systems.
Considerable resources are allocated to data collection, system maintenance and data verification. One aspect of the responsible research assessment is that the costs of data collection, both direct and indirect, should be proportional to the advantages gained from the data collection. For example, researchers should not be obliged to disclose details about expert activities without knowing in advance how the information will be used. Conversely, the development of interoperable digital services can support researchers by enabling data to be shared and used elsewhere once it has been entered to a system. This can significantly reduce administrative work while enabling information to be extensively used in a compatible and reliable format.
The Researcher’s Profile Tool introduced in the Research.fi service in 2023 enables researchers to compile their own data coming from several organisations into one place. The future goal is to enable researchers to automatically transfer their data to the funder or other service they use. In addition, a CV tool is currently being developed to make researchers’ work easier.
Hanna-Mari Puuska, Director, CSC
Laura Himanen, Project Manager, CSC