Research | Specialists discuss the criteria that guides the selection and the consequences of evaluations like the one conducted by Stanford University, which reveals functional aspects of the academic and scientific fields
*By Elstor Hanzen
*First published December 15, 2022
*Photo: Flávio Dutra/UJ
A select group usually appears annually in the ranking of most influential researchers in the world. Conducted by Stanford University (United States) and published by Elsevier in November 2022, the latest survey presents the 100.000 top-cited scientists in 2021. Among the mentioned, 46 are affiliated with UFRGS and are distributed across 10 graduation areas, representing 10% of the faculty in those fields. Considering all graduation faculty at the University, this group accounts for 1,5%. In the previous list, published in 2021, 32 of members of the list were from the university.
The ranking presents a way to value scientific production and acknowledge the researchers and institutions involved, as it often becomes a subject of media attention. Despite the undeniable degree of influence of those ranked and the excellence of their graduate programs in their respective fields, specialists in the analysis and dissemination of scientific information warn about the importance of considering the data, evaluation criteria, and the topics of interest that guided the compilation of the list. In order to select the top 100.000, data extracted from Scopus, a bibliographic database, of around 200.000 scientists was analyzed. The metric used was based on the number of citations received by a specific researcher, including self-citations.
“Due to its sample, the analysis has biases considering the countries involved, languages of the publications, and selected metrics, which favor a specific group of commercial journals”, points out Washington Segundo, coordinator of Treatment, Analysis and Dissemination of Scientific Information at the Brazilian Institute of Information in Science and Technology (IBICT). He also emphasizes that Scopus has around 81 million documents in its database, while other more inclusive databases present two to five times more studies. Examples include OpenAIRE, with 146 million articles, Core UK, with 207 million articles, and BASE Search, with 316 million.
“Furthermore, the raw data of the citations are not openly available due to the reinforced commercial specificities, making it difficult for third parties to reproduce the study and conduct the ranking,” Washington adds. Elsevier is one of the main commercial publishers at the international level, with publicly open trade on the London stock exchange. “Their interests are usually more aligned with higher financial return than with the best practices of the scientific world”, evaluates the coordinator.
Exact sciences and humanities
Among the 46 most influential researchers affiliated with UFRGS, the majority are in Engineering (12) and Medicine (7). The remaining 27 names are equally distributed across areas that are based on exact and applied sciences. Humanities fields, such as History or Languages and Literature, have never been ranked in this type of international selection, even though the graduate program in these areas at the University have received the highest rating from the Coordination for the Improvement of Higher Education Personnel (CAPES).
There are several justifications for such a scenario. The Scopus database is predominantly composed of articles from scientific journals, the majority of which are commercial. However, certain fields do not primarily use this medium for scientific communication. Such is the case with humanities studies, which usually tend to use books, essays, and conference papers as means of dissemination. Furthermore, the majority of research in this field in Brazil takes into account a specific regional context that may limit international interest. These factors make it difficult to include certain subjects in scientific journals, as they tend to focus on issues in global scale.
Thus, as Fernando Nicolazzi, coordinator of the Laboratory of Studies on Political Uses of the Past (LUPPA/UFRGS) and researcher (level 2) for the National Council for Scientific and Technological Development (CNPq), points out, it is difficult to establish a universal metric that can measure – both in quantity and quality – scientific research. This difficulty occurs due to the specificities and conditions of research production, that vary greatly among different areas of knowledge and geographical region, both from a local perspective (regional variations in a vast country such as Brazil) and a global perspective (North and South). “For example, the choice, or imposition, of the language in which a text is written is decisive to define its reach and the possibility of it being used and cited. There is no doubt that publications in English have a wider range of circulation than those in Portuguese,” compares the History researcher.
According to Nicolazzi, there are also areas, many of which are part of the exact sciences, in which scientific production happens in a collaborative way – scientific articles with multiple authors resulting from research conducted by large teams. “In many cases, the coordinator of these laboratories ends up appearing as an author on different ‘products’ involving this research”. Some areas also tend to prioritize the publication of scientific articles in specialized journals over books.
Another aspect highlighted by the researcher is the different timeframes between the research and the publication of partial or final results, which varies on each field.
“We can consider the agility with which science identified the virus responsible for the COVID-19 and its mutations and developed specific vaccines against it. All of this was supported by studies that circulated very rapidly and had global repercussions.”
Fernando Nicolazzi
In the humanities, particularly in History, such agility is not a possibility, and the production of articles with multiple authors is uncommon. “The vast majority are texts with an individual author, with only a few exceptions. Advisors usually don’t appear as coauthors in their students’ productions”, says Nicolazzi. Furthermore, there is a strong emphasis on the publication of books, which have a significantly different production, dissemination and circulation process from articles in journals – not to mention the time required from the beginning of the research until the publication of its results.
All these factors lead to citations in the humanities having a longer lifespan. In other words, to consider only short periods, such as three or five years, in the evaluation of these areas is unreliable. “One of the most cited academic articles in my field was published in 1988”, mentions Nicolazzi.
International projection
According to Júlio Barcellos, Executive Vice-President for Graduate Studies at UFRGS, the university is growing with each new ranking and is establishing itself as an internationalized institution. Furthermore, Barcellos emphasizes that the systems for capturing scientific productions, science communication, and dissemination are continuously improving, leading faculty members to become more experienced in communicating their scientific achievements.
He adds that international rankings are complex and combine a multitude of data. Nonetheless, the mentioned researchers are part of excellent graduate programs. “This is of utmost importance for the rankings since graduate programs are one of the main components responsible for the university’s exceptional collocation, both national and internationally”.
Andréa Bernardes, professor at UFRGS since 1990 and researcher for the Graduate Program in Mining, Metallurgical, and Materials Engineering (PPGE3M) since 1997, the same year that she returned from her doctoral studies at the Technical University of Berlin (Germany), integrates the group of 12 engineering researchers from the university present in the ranking. “Since my doctoral studies, my research has been focused on what is known as circular economy, or in other words, an emphasis on material recycling processes and water and effluent treatment processes for reuse,” she explains.
For the researcher, this recognition signals what is happening with the careers of the listed researchers. “This is due to the fact that I have had the privilege of always counting on colleagues and students to conduct research, which provided me with the necessary intellectual and infrastructure foundation. Thus, I believe my name is in the ranking because I am associated with an excellent group of people who already have a consolidated history of working together,” she attributes. She also mentions that the PPGE3M has an exceptional rating of 7 from CAPES.
The only researcher from the Graduate Program in Human Movement Sciences (PPGCMH) to be listed among the 46 most influential UFRGS researchers, but present in the ranking since 2019, Eduardo Lusa Cadore attributes the quality of the publications, the collaboration between research groups, and the application of human resources in his performance as a researcher as the key factors for his ranking. “We focus on relevant knowledge when prescribing exercises, so that students feel motivated and capable of performing activities practically. Thanks to this, we can see the results of our group effort and establish relations with international institutions”, emphasizes the professor, who has 20 years of experience in academia. PPGCMH has a rating of 6 from CAPES.
Below is the list of UFRGS researchers included in the ranking, organized by unit:
Although the graduate programs are of excellence and the researchers’ influence is recognized, the purely quantitative evaluation system based on citations has distortions and deserves criticism, as previously pointed out by the coordinator of IBICT, even though it seeks to extrapolate a measurement of quality. “Citation indexes capture the popularity of an article and leave out the assessment of its quality and scientific content, as well as other issues of great relevance, such as the social impact of the research. The main problem is that research funding distribution is often based solely on citation indexes, neglecting other important aspects of the investigatory capacity of those proposing the research,” analyzes Washington.
(Photo: Flávio Dutra/UJ)
Consequences
This scenario has consequences in the economy of academic citations, and if this is the only metric to be used in the evaluation of researchers, there are certain areas that, due to their production methods, may not achieve equivalent numbers compared to others. On the other hand, it would be difficult, even dangerous, to attempt to simply quantify the social impact of research in the Humanities, warns Nicolazzi. Some research projects can develop social effects years or decades after their publication. Examples include the research on the Sociology of violence – which contributes to the development of public security polices – and historiographical studies about the 19th-century slave society, which help in understanding authoritarianism and racism in the contemporary Brazilian society.
“Is it possible to evaluate that in terms of social impact and when is it appropriate to measure it?”, questions Nicolazzi. According to him, such questions lead to the problematization of instituting universal metrics for vastly distinct areas and, in the same way, for social contexts that do not always present the same conditions for scientific production and dissemination.
Furthermore, these factors draw attention to the dangers in linking these indexes and the quality of researchers, or more importantly these indexes and the creation of public policies for science and allocation of resources for research funding. “All of this does not devalue the influence of the ranked names”, Nicolazzi emphasizes.
Even though it’s not a rule – because it depends greatly on the impact, innovation, and future contributions of the research – being among the most influential scientists in the world can have significant repercussions in the researcher’s life, Barcellos acknowledges.
“Obviously, an international research leader is much more competitive regarding resources, since such researcher has demonstrated the necessary competence to make good use of those resources and has transformed research into advancements for society. Therefore, if a university is measured by its national and international standing, naturally, there are advantages for a researcher to be at the top of the list”
Júlio Barcellos
Furthermore, Barcellos emphasizes that many international public notices are aimed at this select group. “Having these leaders in our university is an opportunity for young researchers to associate themselves, form teams, and this leadership generates positive succession,” he adds.
However, there are cases where this influence can go too far, as pointed out by Wolnei Caumo, a researcher for the Graduate Program in Medical Sciences (PPGCM) at UFRGS and project consultant for Harvard University (United States). With over 20 years of experience in academia, he warns that there should be safeguards against biased evaluations: “When people who are prominently featured in the field reach for public notices of funding availability, evaluators may tend to think that the reputation of those applicants precedes them and have to fight such preconceived idea. The best alternative [for such evaluators] would be a blind assessment, which is when both applicants and/or the institution behind the project have their identities omitted.” The professor emphasizes that, in such cases, the evaluation should not be based on the author’s resume or the weight of the institution but solely on the quality and importance of the research. “Otherwise, we won’t be opening opportunities to new researchers, and that is unfair,” he says.
The cherry on top
Nicolazzi warns that the naturalization of the idea that academic evaluations are like Formula 1 races – where only a few can occupy the top positions on the podium – reproduces the rationality that privileges a competitive model of society, in which individuals and institutions are reduced to competitors against each other. “In the end, we are under the risk of reaching a situation where many people contribute to making the cake, but only a few will be able to enjoy it, and even fewer will have the privilege of eating the cherry on top,” he compares.
Even though they are living under the capitalist rationality and the logic of academic productivity, Nicolazzi emphasizes the importance of reflecting on to what extent metrics are a cause or a consequence of scientific investigations. “In other words, to what extent researchers are acting solely to achieve a specific result in quantitative evaluations, or to what extent this result is a consequence of the quality of the research conducted,” he explains.
The answer, he acknowledges, is neither easy nor simple and involves considering the social and political conditions of knowledge production. “Nevertheless, my point of view is that what prevails among researchers in public universities is indeed the commitment to science and public welfare. The rankings are a consequence of the hard work and dedication carried out in our institutions,” he concludes.
The current logic of evaluating applied science promotes productivity and favors commercial groups that dominate the scientific publishing market, states Washington. For this reason, IBICT coordinates a commitment with the Open Government Partnership, led by the Comptroller General of the Union (CGU), which aims at building new evaluation mechanisms to advance Open Science in Brazil. “It is necessary to raise awareness among evaluation committees and review the criteria adopted, prioritizing a global analysis of the impact and results obtained by science,” stresses the IBICT coordinator.
Translated into English by Enzo Sezar de Assis, undergraduate student enrolled in the course “Supervised Translation Training I (English)” of the Undergraduate Program in Language and Literature, under the supervision and translation revision of Professor Elizamari R. Becker (P.h.D.) – IL/UFRGS.
:: Read in portuguese:
Áreas da saúde e exatas reúnem maioria dos 46 cientistas da UFRGS em ranking dos mais influentes do mundo