Scimago Lab Blog

La medición holística de las instituciones en el mundo es un deseo que se puede llegar a cruzar por la mente de las organizaciones que generan indicadores y rankings. Sin embargo, es importante mencionar que este deseo no pasa de ser eso, un deseo. Existen en las instituciones muchas variables que no se pueden medir extermante y, mucho menos, desde una política pública. Por tanto, sería sensato admitir que la visión y alcance de los rankings esté focalizada en un conjunto de variables que se puedan observar y analizar.

El SCImago Institutions Rankings en su metodología plantea tres conjuntos de variables, llamadas factores, que son Research, Innovation y Societal. Los tres factores tienen vínculos muy fuertes entre sí y, a su vez, permiten establecer el grado de articulación institucional entre la investigación y la innovación con su contexto, mediante el factor Societal.

Una sencilla comparación de los valores obtenidos por cada institución en el ranking 2020 puede brindar algunas luces sobre la relación entre los factores y los enfoques de las regiones. Las gráficas siguientes muestran los factores Innovation y Societal en los ejes, el factor Research como tamaño de cada circulo (que representa una institución) y el color de cada círculo como indicador del sector al que pertenece la institución (Government, Universities, Health, Private, Other) en las regiones Asia, Norteamérica, Europa Occidental, Europa Oriental, Latinoamérica y el Pacífico.

A la vista de las gráficas anteriores, se pueden detectar algunos comportamientos:

1. Hay una gran disparidad entre los valores promedio de cada factor, siendo el factor Societal el menos representativo en las instituciones.
2. Las universidades tienden a tener mejor proporción entre los factores Innovation y Societal.
3. Latinoamérica es la región que menos evidencia presenta en el factor Innovation.
4. Las instituciones del sector Gobierno son el principal promotor de innovación en Europa y Asia, no así en Norteamérica y en el Pacífico, donde son las instituciones del sector Salud las que llevan la bandera de la innovación.
5. Se destaca un número importante de instituciones en todas las regiones con altos valores en el factor Innovation y a su vez, muy poco valor en el factor Societal, lo cual da cuenta de la falta de vínculo entre la innovación realizada y el impacto social que obtiene la institución por su presencia web y en Redes Sociales.
6. La región pacífica presenta indicadores muy similares a Norteamérica en los factores Research y Societal, y acercándose cada día más el factor Innovation, teniendo presente que el número de instituciones es menor.

Este sencillo análisis, pone de manifiesto que debe existir entre las instituciones y con una visión global de región, una apuesta sólida por la innovación, fruto de los resultados de investigación y del relacionamiento con el sector industrial y empresarial y, con la misma intensidad de su producción, sea difundida adecuadamente a la sociedad. Entendiéndose como adecuadamente, en los canales y los medios más pertinentes para que los diferentes actores de la sociedad identifiquen y reconozcan el aporte de las instituciones al desarrollo de ésta, ya sea en lo científico, lo económico, lo empresarial, lo social, etc.

Latinoamérica aún tiene un gran margen de desarrollo, ha demostrado un crecimiento en sus productos de investigación y destellos de innovación y presencia social, solo es necesario que los investigadores y los directivos reconozcan su propia capacidad y busquen aumentarla y consolidarla mediante la formación y la colaboración internacional, para sacar así, el mejor provecho a los grandes esfuerzos en investigación que se realizan.

Data source: Scimago Institutions Rankings

We, at Scimago Lab, include the Normalized Impact indicator in every ranking SIR we build, both in world and iberoamerican series. This indicator can be seen as a substitute to the overused Impact Factor as it conveniently captures the scientific impact of institutions and corrects the biases generated when heterogeneous outputs –such as those coming from big universities– are analyzed. This post is the first of a series of two in which we describe its main features and computation methods.

When the scientific output of an university is analyzed to evaluate its impact and compare it to other universities’, we must have in mind that there exists many factors influencing the amount of citations achieved by the university such as the research thematic profile, the document types where its output is published, the “age” of the papers under analysis, etc. These biases make a direct comparison on the quantity of citations obtained by universities unfeasible, therefore rankings built over raw citation counts should be avoided. It should happen the same to journal rankings (as IF based journal rankings), but that matter will not be discussed here.

At the level of Universities and large Research Centres (national agencies, big companies, etc), the Normalized Impact indicator is more efficient to reveal the scientific impact (or visibility) of institutions outputs because it includes some tools to correct the topic-type-age biases. The Normalized Impact amends the citation value achieved by every paper by comparing it to an international standard, thus in this way it normalizes the disparity of citation scores obtained by papers in different domains, with different ages and of different types.

This citedeness bias is due to the different  citation dynamics:

Between Thematic Domains. There exists different citation behaviours in different areas of Science, so it cannot be directly compared the citedeness of papers in Medicine with works on Social Sciences, Engineering, Life Sciences nor Health Life. Every thematic domain has its own citation patterns.

The following bubble chart shows Germany’s 30 Top Subject Categories by number of documents published in the period 2009-2010. Bubble size represents Citations per Document in the area. The big light green bubble correspond to Cell Biology –located between 20,000 and 25,000 Cites in the vertical axis– (11.13 citations per document) while the wine-red little one down in the middle correspond to Electrical and Electronic Engineering (2.23 citations per document). So how would influence this parameter to a university output that had a strong Engineering focus? It seems clear that using Impact Factor (or any other non-standardized methodology) to build the rank would push it down, no matter how good it was in its research.

Germany’s 30 Top Subject Categories by number of documents published in the period 2009-2010
Source: SCImago Journal and Country Rankg. Period: 2009-2010. Link

Between different document types. There also exists different citation patterns among the various types of papers/articles/reviews/etc. traditionally included in scholarly journals. So reviews should not be directly compared with papers (original article), conference papers, letters, etc.

Among different papers’ “ages”. Usually, when a ranking is built, the output of universities to be compared includes sets of papers with different ages, that is, papers that have been published in several years. But a paper that has been published three years before another one has more chances to achieve citations because it has had more time to be known by the scientific community, so the ages of the papers in the set influence the number of expected citations.

The following chart shows year by year citedeness achieved by Canada’s output, both the total number of citations and self-citations (that is, canadian papers citing ohter canadian papers). It is clear how the line declines at the end, when it approximates to the current date. So should 2006 output be evenly compared to 2010’s when less citations are “being played”? It must be taken into account that when universities are compared to build rankings, papers from differente ages a measured with this evident age bias.

Annual Series of Citations achieved by Canada’s Output
Source: SCImago Journal and Country Rankg. Period: 2009-2010. Link

How international standards are built

What NI makes is to build citation standards that are used to normalize the citation achieved by an institution. The normalization is then carried out paper by paper o just in one time by summing up all the university citations (i.e. citations received by the whole output of and institution) and normalizing later.

The citation value of a “standard paper” is computed for every topic-type-age  relationship. For instance, to build a “standard 2008 review in Oncology” the citations to every Review in Oncolgy published in 2008 are counted and then divided by the number of reviews or to build a “standard 2009 original article in Applied Mathematics” the mean of citations achieved by all the Original Articles in Applied Mathematics published in 2009 is computed. This is done for each one of these triads so at the end of this process we have a bunch of standard paper values ready to be used to normalize the output of institutions.

How to interpret NI scores

NI scores are standardized to the world average which is asigned a value of 1, then a score of 0.8 means the institution is cited 20% below world average and 1.3 means the institution is cited 30% above world average.

The following chart shows the Top Ten universities by output in China and France ordering them by NI scores. As can be seen, just two Chinese universities equal or exceed the world average barrier, to the rest their scientific impact is lower than the international standard, in some a 40% lower (those that have 0.6 score) On the other hand, french largest universities present scientific outputs with significant impact, all of the over 40% of the world average (1.4 to 1.7)

Source: SCImago Insitutions Rankings. Period: 2006-2010

In the following post of this series we will talk about the two main methods to compute the Normalized Impact indicator: the so called CROWN indicator by the University of Leiden and the “Field Normalized Citation Score” set by the Karolinska Instituet. We will also compare NI with other indicators used in SIR Rankings and describe its features.

To date we have been posting brief scientometric tables of universities and national research agencies. to expand our analysis, the current post is devoted to Private Companies as scientific agents, so we´ll lead the spotlight to the World’s Major Companies and their research performance outputs. We are going to consider the Top 100 scientific producers and show their indicators related to scientific impact, international collaboration, ability to put their papers in top journals, specialization rate and excellence output.

The research activity carried out at big companies is concentrated in Northen America and Western Europe –44 in Northern America and 36 in Western Europe among the Top 100–. The list shows a prominent bias in the Activity Sector as well: most of the selected companies belong to ICT(46), PHARMACEUTICAL (26) or ENERGETIC (14)

The following table contains some key scientometric indicators of these companies based on SCOPUS data during the period 2003-2010:

Download Table

The indicators included in the table are the following:

Output

The number of scientific papers published in scholarly journals reveals the ability of an institution to create scientific knowledge.  In co-authored publications, a score is assigned to each contributing institution through the author’s institutional address.

International Collaboration

It shows the ability of institutions to create international research links through the output ratio that has been produced in collaboration with foreign institutions. The values are computed by analyzing the institution’s output whose affiliations include more than one country address.

Normalized Impact

Normalized Impact scores indicate the scientific impact that institutions have over the scientific community. In order to obtain a fair measurement of impact, the calculation removes the influence due to institutions’ size and research profile making it ideal for comparing research performance. Normalized Impact values show the ratio between the average scientific impact of an institution and the world average impact of publications of the same time frame, document type and subject category. The values are expressed in percentages and show the relationship of the institution’s average impact to the world average, which is 1, –i.e. a score of 0.8 means the institution is cited 20% below world average and 1.3 means the institution is cited 30% above world average. Normalized Impact is computed using the methodology established by the Karolinska Intitutet in Sweden where it is named “Item oriented field normalized citation score average”. The long name used is because the normalization of the citation values is done on an individual article level. Further information on the methodology at Bibliometric Handbook for Karolinska Institutet .

High Quality Publications Q1

Ratio of publications that an institution publishes in the world most influential scholarly journals. Journals considered for this indicator are those ranked in the first quartile (25%) in their categories as ordered by SCImago Journal Rank SJR indicator.

Specialization Rate

The Specialization Rate indicates the extent of thematic concentration / dispersion of an institution’s scientific output. Values range between 0 to 1, indicating generalistic vs. specialized institutions respectively. This indicator is computed according to the Gini Index used in Economy.

Excellence

The Excellence indicates the number of papers an institution included into the set formed by the 10% of the most cited papers in their respective scientific fields. It is a measure of high quality output of research institutions.

Leadership

Number of research papers where the author of the company is the main contributor, selected as paper’s corresponding author which is expected to be in most cases the leader resarcher.

Download the report

Last year we published a series of posts characterizing, through scientometrics methods, the research activity carried outin some of the major National Bodies for Reserach from Europe. Specifically, the series comprised the Cosiglio Nazionale delle Ricerche CNR (Italy), the Max Planck Society (Germany), and the Spanish Consejo Superior de Investigaciones Científicas. With the goal of highlighting some global characteristics involving the research outcomes of national organizations, we are releasing in this post a informative table containing a selection of performance indicators belonging to the six largest National Bodies for Research (in number of scientific publications):

• Centre National de la Recherche Scientifique,
• Chinese Academy of Sciences,
• Russian Academy of Sciences,
• Max Planck Gesellschaft,
• Consejo Superior de Investigaciones Cientificas,
• Consiglio Nazionale delle Ricerche

Félix de Moya Anegón is Research Professor at the Institute of Public Goods and Policies (IPP) from the Spanish National Research Council (CSIC), his academic interests include scientometrics, bibliometrics, research evaluation and science policy; he has published around 100 papers in these fields. He is SCImago Research Group‘s main researcher, where he has led renowned bibliometic projects including Scimago Journal & Country Rank, Scimago Institution Rankings and The Atlas of Science. Prof. De Moya is also advisor for Science Policy issues for national organizations of science and technology and research institutions around the world. 

For this post, we have elaborated rankings of universities based on the output they have in each of the major scientific fields. Then we took the Top 100 in each field and saw them by countries. The result is the following table where each row represents a country and each cell shows how many institutions that country has within the Top 100 in the corresponding field. The rank is ordered by the average of institutions that countries have in Top 100. We have used Scopus as the data source, after a disambiguation process of institution names.

Download the table.

Download the full list of universities.

 

Félix de Moya Anegón is Research Professor at the Institute of Public Goods and Policies (IPP) from the Spanish National Research Council (CSIC), his academic interests include scientometrics, bibliometrics, research evaluation and science policy; he has published around 100 papers in these fields. He is SCImago Research Group‘s main researcher, where he has led renowned bibliometic projects including Scimago Journal & Country Rank, Scimago Institution Rankings and The Atlas of Science. Prof. De Moya is also advisor for Science Policy issues for national organizations of science and technology and research institutions around the world.