“AJ branch-off edition” is a category of articles published in “Acaric Journal,” a career magazine for graduate students and researchers published by Acaric Co., Ltd., or fresh articles only available on the web. This time, we bring you the article from vol.1.
Impact Factor (hereafter, IF) is an index used to rank journals. IF expressed as a numerical value is easy to compare and, therefore, is easily misused. In response to the overuse of IF without knowing the calculation method and background, the humanities and social sciences have proposed a new evaluation index.
— How do you evaluate research results such as papers in the humanities and social sciences?
Prof. Ikeda: There are two types of evaluation: quantitative and qualitative. In any research, the qualitative part is the most important, and there is some consensus among experts. However, because it is qualitative, it has the aspect of being vague. Therefore, it is easy to evaluate if the results are outstanding, but if they are in the middle of the range, it is difficult to make a precise evaluation. On the other hand, quantitative is an indicator of “number,” so in the humanities and social sciences, we can only use the number of journal papers and publications. Generally, we use a combination of these for evaluation.
— “Impact Factor” is often used as an evaluation index in science and engineering research. What are the characteristics and challenges of this indicator?
Ikeda: The “Impact Factor” (IF) is calculated mainly for natural science and social science papers. It is calculated by dividing the number of times cited two years ago and one year ago (the number of citations) by the total number of articles published in the same journal. It does not measure the quality of the research, but it measures the impact. I believe that many people misunderstand this point.
Although publishing a paper in a high-impact journal is meaningful, those pursuing science may not be concerned about IF that much. It is crucial that “IF cannot measure the quality of research.” For example, the IF figure will go up even if others cite the research as a target of criticism. In addition, it is impossible to know from the outside how the IF has arrived at such a value. In other words, it is a black box.
— IF is calculated based on the Web of Science but does not include many Japanese-language papers in the humanities and social sciences. Is this an appropriate index for evaluating researchers?
Ikeda: Web of Science is a database that targets specific journals, so its sampling is biased; the same is true of SCOPUS, which is the basis for the journal metric CiteScore, which is different from IF. Because Western companies operate these databases and metrics, they contain few Japanese-language papers. They primarily target journal papers written in English.
In addition, since they are overwhelmingly biased toward science and engineering, they cannot be said to provide a fair evaluation across research fields. However, if you limit yourself to areas such as medicine, life sciences, and physics, where most papers are mainly in English and influential journals are covered. In that case, you may find a certain validity in using these journal metrics to evaluate your research.
— I understand that the University of Tsukuba has developed its evaluation index called “iMD,” but could you tell us about its compatibility with IF and how it differs?
Ikeda: iMD (index for Measuring Diversity) is an attempt to quantify the value of academic papers, including those in the humanities and social sciences. First, it has two points in common with IF: it is a journal metric that does not evaluate each article individually or measure the research quality.
The second difference is that IF is processed in a black box database, whereas iMD can quickly calculate manually. In addition, while IF measures how much attention a finding has received in terms of “number of citations,” iMD measures the “diversity of authors’ affiliations.
When looking at diversity in terms of the organization and country to which the author belongs, for example, if 10 people co-authored a paper and all of them were researchers at the University of Tsukuba, then “organization: 1” and “country: 1. However, if all the authors belong to different Japanese universities, then “Organization: 10” and “Country: 1. Furthermore, if all authors belonged to different universities and countries, it could be “Organization: 10” and “Country: 10. We believe that researchers across disciplines can reach a consensus that a national journal is “better” than an intramural bulletin and an international journal is “better” than a national journal.
There is a correlation, as journals with higher IFs have more significant numbers in iMD. It is because authors in journals with high IF, such as Nature, Cell, and Science, have diverse affiliations.
— What was the background of the development of iMD?
Ikeda: Papers written by people affiliated with companies, even in the engineering field, are often written in Japanese, so they usually do not have an IF. We also believe that there should be a variety of evaluation methods and that we can adequately understand by measuring with various indicators. Therefore, we aimed to base our evaluation on factors other than citation counts.
When evaluating research papers in science, engineering, and literature, it is difficult to compare them by IF, but iMD allows us to do so. We can use it not only for straightforward comparison of numbers but also for growth rates.
The starting point for iMD was the background that when evaluating something within the humanities and social sciences in a given university, there are few people in the same field of expertise, making it difficult to assess each other and making qualitative evaluation difficult. In quantitative evaluation, the number of articles published in academic journals and books is essential. Still, the level of the academic journals varies, there are differences between academic and general books, and you can publish some books if you pay for them, such as self-published books. We had a problem of not being able to distinguish between these different types of publications clearly.
Dr. Morimoto: Professors who serve as reviewers for Grants-in-Aid for Scientific Research also make judgments based on the names of journals. Their perception of journals has much to do with the qualitative evaluation of research content and researchers. However, the criteria are only in the minds of researchers. Therefore, before developing iMD, we tried to make a list and ask them to rate the journals on a 5-point scale from S to D. With the cooperation of URAs (University Research Administrators) in the humanities at Kyoto University and Osaka University, we surveyed three universities, including the University of Tsukuba.
Focusing on fields in which many professors write articles in Japanese (e.g., Japanese linguistics, Japanese history, Japanese literature, etc.), we asked professors at each university to tell us what journals and books they thought were “good.” However, the result was that the academic clique controversy is still dragging on. While the “university” cannot give a high rating to a specific journal, the faculty members “individually” rate it as “good.” The fact that the same journal was rated differently by different people made us realize once again the difficulty of qualitative evaluation.
— What activities are you doing to promote the use of iMD at other universities and in a wide range of research fields?
Morimoto: We are trying to get people to understand and learn about iMD by talking to them repeatedly like missionaries in various places. For example, I was recently asked to speak about iMD by the “Committee to Study Journal Issues” established in the Information Committee of the Council for Science and Technology within the Ministry of Education, Culture, Sports, Science and Technology. Currently, we are working to spread the word with the participation of professors from Hitotsubashi University and Doshisha University.
We also utilize the URA network to disseminate information. The URA industry has a place called the RA Council, which is like an academic conference for URAs. We have made presentations there every year since 2015 and continue exchanging opinions with the participants. We also hold a large forum called the “Humanities and Social Sciences Research Promotion Forum” once a year with URAs from Kyoto University, Osaka University, and other universities. We have had several discussions there as well. We are trying to promote iMD through these horizontal connections. In fact, some private universities have already begun to evaluate research using iMD on a trial basis.
— What are the advantages of “evaluating research” with iMD?
Ikeda: The extraordinary thing about iMD is that it produces different figures for the university bulletin, national-level, and international journals. Wouldn’t this motivate us to try more diverse journals? Without this number, it would be “one for one” anyway, and it would be possible to increase the number only with intramural journals. We expect iMD to improve the dissemination of research.
Also, until now, researchers have had to decide what languages to use and who to read them, but I believe that it will allow them to make their own decisions about where to submit their work. We believe this will stimulate scholarship and lead to the development of science.
I believe we could create this iMD because we are both university faculty members and URAs conducting research together.
Morimoto: The URA position has many uncertain and challenging aspects, such as the fact that employment and salary patterns vary significantly from university to university, and the career path is not yet firmly established, but it is a fascinating job. I think the position requires the most entrepreneurial mindset in a university.
Translated with www.DeepL.com/Translator (free version) and Acaric Journal Editorial Board
Profile (at the time of the interview)
Prof. Jun Ikeda
Executive Officer of the University of Tsukuba (in charge of Tsukuba Conference), Director of the Office of the Assistant to the President, and Professor of Humanities and Social Sciences at the University of Tsukuba. After working as an assistant professor at the Faculty of Foreign Languages, Kansai University of Foreign Studies, he joined the University of Tsukuba in 2000. He specializes in Biblical Hebrew and other ancient Semitic languages, linguistics, digital humanities, research assessment indicators, etc. Ph.D. (Tel Aviv University).
Dr. Yukihito Morimoto
Research Administrator, URA Research Strategy Office, University of Tsukuba. After working as a research coordinator (URA) at the Office of the President, Kansai University, he has been in his current position since FY2013. He was seconded to the Cool Japan Policy Division, Ministry of Economy, Trade and Industry as an assistant director in FY2019. He is researching and developing methods to evaluate academic journals in terms of diversity. Ph.D. in Economics.
