Scientific Journals | With the changes in how the production of knowledge by graduate programs is evaluated, each journal only gets one classification, regardless of the number of areas it was cited in, based on bibliometric indicators
*By Elston Hanzen
*First published March 2, 2023
*Image: Flávio Dutra/JU
The new Qualis invests in bibliometric indicators as the main way to classify journals. The three indicators – Impact Factor, Scientific Journal Ranking (SJR) and h5-index – now take into consideration the number of citations in three databases: Web of Science, Scopus and Google Scholar. These indicators were considered in every area of evaluation and are the main points of controversy regarding the change.
Additionally, the strata went up from eight to nine. In the 2013-2016 quadrennial, they were A1, A2, B1, B2, B3, B4, B5, and C. Later, in 2017-2020, they were A1, A2, A3, A4, B1, B2, B3, B4, and C – in a descending order of relevance.
There is also the new evaluation process by main area, Reference Qualis, that considers the area with the most publications between the years of 2017 and 2018, with its quantity obtained from Brazilian graduate program’s evaluations reports.
If scores are still tied, the main area will be the one with the most representative number of publications in relation to the total productions of the area (i.e., a journal will not have one classification per area like it used to). Thus, a journal with no publications of graduate programs’ faculty and students will not be considered.
In previous versions, each area had the autonomy to define its classification criteria based on bibliometric indicators, editorial characteristics or prestige. This allowed the same journal to be considered A1 in one area and B4 in another.
With the new system, the definition is on the area with the most productions (main area), and every other receives the same classification – even if their journals have different profiles of distribution of bibliometric indicators.
Washington Segundo, coordinator of the Area of Treatment, Analysis and Dissemination of the Brazilian Institute of Information in Science and Technology (IBICT), says this means that publications with distinct indicators can have the same Qualis if they end up at different main areas.
Ana Gabriela Clipes Ferreira, coordinator of the Advisory Committee of Support to Editing Scientific Journals from UFRGS, highlights that, until recently, areas like Education, Linguistics, Arts, and Language and Literature did not have a strong presence in indexers and databases, such as Scopus and Web of Science – therefore, those were not considered on the assessment of the area. The reality was different in areas such as Agrarian Sciences and Biological Sciences.
“Depending on the area of knowledge, the impact can be either negative or positive. Although the new Qualis was happily welcomed by some journals, others joined the protests against the system’s changes.”
Ana Clipes
CAPES sees these changes as improvements. The agency states that Qualis is a specific tool to evaluate Brazilian graduate programs, considering very relevant national journals due to their large coverage and power of dissemination – which creates a positive impact on the programs and society.
“Some areas were allowed to subdivide groups of national and international journals – a request from the very own coordinators of the areas of evaluation. This reinforces the importance of continuous dialogue with representatives of the academic community,” says a memo from CAPES.
The agency states that the strata and the finalization of the evaluative process were published in the Sucupira Platform in 2022, as well as documents with the history of the development of Qualis Reference and details of its methodology.
Indicators do drive actions
Publishing outside of your area is not penalized anymore, so the new Qualis may induce faculty and students to produce in areas besides their own. Because journals can only have one Qualis, if the journal is considered good in its main area, it will be considered good in the researcher’s area.
However, the new Qualis lost its qualitative aspect. Now, despite the specific adjustments made by the evaluation Committees, it exclusively bases itself on bibliometric indicators of citations – says Fabio Gouveia, researcher in Bibliometrics and Scientometrics and Technologist in Public Health at Fiocruz.
“First of all, it is important to reinforce that every time we define one indicator in an evaluative process, we induce the actions of those being evaluated.”
Fábio Gouveia
Researchers Washington Segundo and Fábio Gouveia agree that while there was an induction gain in interdisciplinarity, the classifications of a determined area influence the strata too much. They also agree that important issues regarding open access, sustainability, periodicity of the journals, among other factors, were lost in this reform.
Regarding the evaluation reports, there is the implication of only accepting double-blind pairs. For example, it is said on Article 9, norm No. 145, September 10, 2021: “It is considered a violation of good practices to adopt any of the following conducts, in accordance with the stipulated conditions from the Qualis Journals report of the respective Areas: III – non adoption of the double-blind peer review, duly auditable, or the violation of the corresponding confidentiality”.
Another recurrent and controversial aspect is the payment for publication. Free traditional journals might demand more time between submission and publication, while the paid (in high amounts) ones, of Qualis A, publish in a span of a few months.
“This has been a limitation in the tough economic times we live in,” points out the coordinator of the postgraduate program in Veterinary Sciences, Marcelo Bertolini. He says many researchers continue to focus their submissions on classic, traditional journals that are still free. “However, there is a feeling that it is a risk, in case these journals are downgraded in the future”.
This creates consequences, because Qualis is a tool of evaluation for graduate programs and has a direct impact on the submission of articles to journals, especially national ones. That is why specialists say that the greater goal of Qualis should be to evaluate practices of national journals to define access to funding for the maintenance and development of national publishers.
“Qualis can be a tool to search for the graduate program we wish for, but today it seems to confuse more than to contribute to good practices of academic and editorial productions.”
Fábio Gouveia
Practical consequences
Among the repercussions of the changes, there is a global tendency of international scientific journals rising through the rankings, with some going from B1 to A1; and others, with no Qualis, going up to strata A. Most of the national journals, however, were downgraded from strata A to B, with the same downgrade within strata B.
“Though we consider the old classification of national publications ‘artificial’, the continuity of our journals in higher strata (A1, A2, B1) has kept them in the scientifical scenario. Now, it can cause damage and even terminate some of them over time.”
Marcelo Bertolini
This is because, according to him, the level of science and research made can be measured within the scope of the journal and vice versa. “It is alarming, from the perspective of dissemination of knowledge, since the degradation of national journals will damage the local scenario, regional and national,” adds Bertolini.
As registered in the Sucupira platform, during the quadrennial 2013-2016, the area of Linguistics and Literature had 107 A1 publications. In 2017-2020, they were 806. In Veterinary Sciences, the number of A1 publications went from 274 to 855. The significant increase of A1 publications was mostly due to international editions. The Journal of Veterinary Internal Medicine, for example, went up from B1 to A1.
The Research Journal of Brazilian Veterinary Medicine, however, went down from A2 to A4.
To Bertolini, this classification is not balanced nor fair within the Veterinary Medicine areas. The area of surgery, for instance, will have many difficulties producing and publishing in higher strata. With the new Qualis, the best surgery journals in the world are classified as A3 – and publishing in them is not easy.
“Such areas will be damaged if compared with the ones from basic science. It is something that we always experienced and that could cause changes in multiple programs, such as ours, because when one area and line of research advances, others might shrink or even disappear. It is a natural process, but compromises the formation of qualified people, knowledge dissemination and scientific competitiveness in ample scale,” ponders the professor.
However, the Language and Literature graduate program’s supervisor, Simone Sarmento, brings another perspective. She says the program publishes a lot in journals of other areas, such as Education, Philosophy, Communication, Anthropology, Psychology, Computer Sciences and Phonoaudiology. “Before, it was not unusual for us to publish in journals with a higher classification in their original area and a lower one in Language and Literature. Since our area is interdisciplinary, I see this change as positive,” Sarmento explains.
Diversity in the production of knowledge
Beyond the debate in the academic community and the repercussion in the press, the new Qualis has also prompted debate on social media. Luis Felipe Miguel, professor in the Institute of Political Science of the University of Brasília (UnB) and researcher at the National Council for Scientific and Technological Development (CNPq), addressed the subject on his social media profile in January 15.
Praising the election of Professor Mercedes Bustamante as the new president of CAPES, he stressed that the scientific publication’s ranking has an important impact on the classification, as well as the funding, of graduate programs and future researchers. “They keep saying that it is a ranking of the journals, not the articles nor the authors, but, in reality, what really matters is publishing in top-rated journals,” Miguel wrote.
He also said that odd cases appear when you enter the issue of criteria: “The higher category in Political Science shows journals specialized in clay (Applied Clay Science), coronary diseases (Atherosclerosis), air pollution (Atmospheric Environment) and space engineering (Aerospace Science and Technology)”. In an interview with JU, the political scientist delved into other topics:
“I think this update is not going to bring major changes because it only replicates several problems from previous rankings. We should have been asked what we want from this, what is being ranked, instead of searching for a numerical indicator with a series of assumptions that leads the areas to weird discussions.”
Luis Felipe Miguel
He points out the importance of having a postgraduate policy that is not on an ivory tower, that does not include or converse with society, and that only rewards the weight of papers in specialized vehicles. “We are producing and disseminating knowledge in a deeply unequal society. It is not about the university being exclusively responsible for doing that, but it needs to receive and understand actors and social movements so it can actively contribute to the process of knowledge production and dissemination, establishing an exchange relationship. This is true for everyone, but especially for Humanities. Otherwise, we end up discouraging many other ways of producing knowledge,” he explains.
For an effective and extensive change to happen, the UnB professor says funding agencies, such as CAPES, would have to play a more supportive role in assisting graduate programs, instead of making monocratic decisions and establishing unilateral criteria of classification independent of the program’s conditions, as it is done today. “You cannot put core and peripheral programs in the same box and evaluate them with the same metrics. Thus, it is necessary to have an open discussion within the postgraduate scope, and not just reproduce what has been done so far, but to calibrate a new way for the agencies to function,” says Luis Felipe Miguel.
Washington agrees with the need to review agencies’ policies. “If we want to get a better distribution of the scarce funding resources, in favor of better organizing and developing the infrastructure of Brazilian Science, Technology and Innovation, we must rethink Qualis,” he concludes.
Translated into English by Luísa de Figueiredo Dias, undergraduate student enrolled in the course “Supervised Translation Training I (English)” of the Undergraduate Program in Language and Literature, under the supervision and translation revision of Professor Elizamari R. Becker (P.h.D.) – IL/UFRGS.
:: Read in portuguese
Novos procedimentos no modelo de avaliação Qualis Periódicos da Capes provocam debate na comunidade acadêmica