We are interviewing Professor Yong Zhao, the director of Information Research Center of China Agricultural University, a Research Librarian and doctoral supervisor at the CAU Library. He has a PhD degree in Management and he engaged in postdoctoral research in Information Science in Institute of Scientific and Technical Information of China (ISTIC), the research direction of which was “Evaluation of Scientific Research”. Moreover, he is the deputy director of Development and Planning Office in CAU, managing the subject evaluation reform, the talent evaluation reform, etc.
He mentioned that evaluation of scientific research is a very important area of Library and Information Science, which is not only the evaluation of scientific research, but also the evaluation of all aspects of science, including the evaluation of science & technology, the evaluation of people, the evaluation of institutions, etc. From theoretical research to practical application, he has been practicing the policies and documents from the government and the Central Committee of the Communist Party of China (CCCPC).
(Archivoz) What is the current situation of the evaluation system of scientific research in China Higher Education?
(Yong Zhao) In the current situation of higher education evaluation system in China, 2016-2017 is a very important period for China’s policy reform. Why? There are two big backgrounds.
2016 is an important year, when China issued a series of documents on talent evaluation reform, including one document ‘Opinions on Deepening the Reform of the System and Mechanism for Talent Development’. This document first proposed “Overcome the orientations of ‘Academic Credentials Only’, ‘Professional Titles Only’, ‘Papers Only’, etc., and paper is not regarded as the restrictive condition to evaluate the application-oriented talents”. These documents were against “SCI Supremacy”, and caused strong repercussions in Chinese academic community.
In 2017, Springer announced the retraction of 107 papers published in Tumor Biology. Most of these papers were from China. In fact, there were some retractions in 2016, which were not as many as 2017, and had not caused enough attention in the society at that time. Why did this happen? Because “SCI Supremacy” has distorted the whole direction of scientific research. Therefore, the reform of scientific research evaluation closely relates to the construction of scientific research integrity system. This is another important background in China.
Policies and documents of CPC Central Committee are instructive and have the legal effect. It is the universities’ responsibility to implement these polices. In October 2020, the CPC Central Committee and the State Council of PRC issued “The Master Plan to Deepen the Education Evaluation Reform in the New Era” (English), which is currently the most authoritative document. The master plan put forward the reform goals for China’s education field, including improving evaluation of higher institutions (universities and colleges). Inheriting the previous policies, it proposed that “effectively address deep-rooted problem ‘Five Only’ (Scores Only, Enrollments Only, Academic Credentials Only, Papers Only, Titles Only)”, “promote the classified evaluation of higher institutions, guide the scientific orientation of different types of universities, and establish their own characteristics and levels”, etc. This is current situation and strategic orientation.
The so-called “classified evaluation” is to adopt different evaluation system according to different disciplines and industries. The medical evaluation reform is the first and fastest in the whole process, due to the high frequency occurrence of retractions and academic misconducts in this field (especially oncology and tumor biology).
The reform of university evaluation system also relates directly to the welfare of researchers. For example, China Agricultural University, for the first time in history, did not conduct professional title evaluation by the end of 2020, but in March 2021. The reason was that the relevant documents continued to be put off (a delay of 3 month) due to heated discussion. First of all, the professional title evaluation should break with “Five Only”, including “Papers Only”, “Titles Only”, “Academic Credentials Only”, “Prizes Only”, and “Projects Only”. The title evaluation system constructed by CAU reflects the diversification, instead of focusing on one indicator. Paper is just one aspect, and not a prerequisite or a mandatory condition, which is a significant change. This change brings new challenges and problems to the researchers, and determines the focus of their future work. On February 26, CAU issued a document on the evaluation of professional titles, and everyone is talking about the details of it.
In conclusion, scientific research evaluation is a major event in China’s higher education, which relates to the personal benefit and vital interest of every scientific researcher. It is the responsibility of scientific research management to make the achievements of scientific researchers be fairly recognized.
(Archivoz) What are the main problems of the current evaluation system?
(YZ) One of the most important problems of the previous evaluation system is “Papers Only”, which is not the main problem anymore. What problems will we encounter after the reconstruction of the scientific research evaluation system?
- One aspect is about global common issues.
The scientific research evaluation system is newly constructed and needs time to adapt. There definitely will be some problems, especially in some evaluation methods. For instance, the evaluation of social services. There are mature bibliometric indicators for the evaluation of general scientific research, such as papers, patents, funds, etc. However, how to measure the scientific research contribution of rural revitalization, social services, regional development, and national strategic support? This kind of evaluation and methodology about social dimension and measurement are challenging, not only in China, but also in the whole world. From the perspective of Library and Information Science, it is indeed an invisible global challenge to evaluate social contribution.
The university has three functions: education function, scientific research function and social service function. For a long time, people equate the function of scientific research with everything. This is partly cause by some university rankings, like the top four rankings: US News, THE, QS, and ARWU. Besides scientific research, the evaluation criteria of these ranking lists also take into account the teaching, but does not include any indicator about social service. There is no weight for social service indicator.
How to measure the social service function? This is a very important methodological issue. There is no quantitative indicators in the current evaluation system. In the mature scientific research evaluation frameworks such as UK REF and Australia ERA, social service evaluation relies on cases and expert evaluation. These qualitative evaluations lack quantitative evaluation methods, and are easily questioned by participants.
- The other aspect is about our Chinese situation.
We often directly apply the international ranking system, like the top four rankings, and simply equate the international excellence of the universities with their contribution to our country, without considering the national conditions. This is logically problematic.
The indicators of the international ranking lists are relatively rough, because they need to ensure an international lateral comparison. Thus, these indicators are internationalized, unified and standardized, and cannot be refined, differentiated and personalized. Some localized indicators and regional unique indicators cannot be included in the international scientific research evaluation system. However, it is these personalized indicators that actually play a crucial role in measuring the contribution of the universities.
The reality is that the local evaluation system, although combined with national conditions, such as Golden Apple Ranking, is immature and not highly recognized by the society. Most university presidents still focus on the international rankings. Therefore, many Chinese universities adopt the international evaluation indicators to implement the “Double First-Class Plan (World-class Universities and First-class Disciplines)”, which is one of the main problems of the scientific research evaluation in China.
In the evaluation system, we also lack certain scientific methods and data.
To build localized scientific a research evaluation system in China, we need the guidance from authoritative institutions, as well as fair, just and open data sources. Here is an example. The Ministry of Education has a statistical yearbook, which published the annual data of transformation of scientific and technological achievements, projects, etc. of all Chinese universities. After 2017, the data dimension of the yearbook has been changed from “University” to “Region and Category”. Now we can only find the transformation of S&T achievements in one region or one category, instead of one university. The Ministry of Education should have the universities’ data, but it did not consider the application value of these data in scientific research evaluation. It published the data from their own perspective, which invisibly expanded the data dimension and had a certain impact on our statistical analysis.
In terms of method, there is no distinction between comprehensive university and professional university in scientific research evaluation. People realize that this is a problem but have not found a solution yet. You cannot compare Peking University (comprehensive university) with China Agricultural University (professional university), which is unreasonable. The two universities have different goals and different disciplines. Let me make another example: China Agricultural University and Beijing University of Posts and Telecommunications. CAU mainly serves agricultural industry, while BUPT mainly serves information technology industry. Compare the transformation of S&T achievements of the two universities, CAU is not as good as BUPT. Nonetheless, CAU also bear the responsibility for rural revitalization and the rural modernization in China. Many teachers are engaged in public welfare and social services, such as Professor Xiaoyun Li’s poverty alleviation project in riverside village. It is easy to ignore the social service function of university, which is an urgent problem to be solved.
In addition, the research evaluation system is more about effectiveness evaluation, and lacks of performance evaluation. It is not good to only focus on the output. We should comprehensively consider the input and the output. For example, the input of a SCI paper in CAU and the input in Peking University are quite different. The results of scientific research evaluation directly relate to the national financial allocation of colleges and universities, so it is very necessary to distinguish the evaluation object and adopt the performance evaluation method.
Interview conducted by: Ke Wu
