Performance Indicators of Spanish CSIC Research Institutes

June 20, 2011

Felix de Moya Anegon. The major European national research councils –Max Plank Society from Germany, Centre National de la Recherche Scientifique (CNRS) from France, Italian’s Consiglio Nazionale delle Ricerche (CNR) or Spanish’s Consejo Superior de Investigaciones Científicas (CSIC)– and science academies from Eastern Europe countries and Asia including China, Russia (Russian only) or Ucrania have highly complex institutional structures consisting of tens or hundreds of research institutes with diverse scientific missions. From a scientometric point of view, the characterization of such institutional complexity demands the use of performance indicators that go beyond global structures and put the focus on each institute, research center and research laboratory that make up these scientific organizations.

Performance Indicators of CSIC Research Institutes
Download the report [English]
Descargar el informe [Español]

With this in mind, this post starts up a series of posts devoted to characterizing the research activity carried out at national research councils and science academies all over the world, through scientometric methods and from an agglutinative perspective. The following table (in PDF format) shows scientometric indicators of research output, impact and collaboration related to Spanish’s CSIC research institutes, based on Elsevier’s Scopus database. Its main goal does not consist in making a ranking table of CSIC centers, but to highlight some of the differential characteristics involving the research outcomes achieved by these institutes.

In order to show value trends, the table includes three consecutive 5-year periods (2003-2007, 2004-2008 and 2005-2009). Ranking criteria is the output of institutions during the period 2005-2009. Also, the institutes are color-marked to indicate which ones have Normalized Impact values higher than CSIC NI average, which overpass Spanish NI average and which ones fall below.

Beforehand, to help put the information in the table into context, we include here the same indicators referred to CSIC entire institution and to Spain. CSIC represents 15.3% of the total Spanish research output:

CSIC Output % IC NI % Q1
2003-2007 35,429 47.8 1.4 69.9
2004-2008 38,665 48.6 1.4 69.5
2005-2009 41,929 49.4 1.4 68.7
SPAIN Output % IC NI % Q1
2003-2007 230,731 33.3 1.1 49.3
2004-2008 252,422 33.9 1.1 48.4
2005-2009 273,482 34.8 1.1 47.8

The indicators exposed are the following:

Output

The output or number of scientific papers published in scholarly journals reveals the ability of an institution to produce scientific knowledge. Output values are affected by institution sizes and research profiles, among others factors. The Output indicator forms the basis for more complex metrics. At co-authored publications a score is assigned to each contributing institution through the author’s institutional address.

International Collaboration IC(%)

This indicator shows the ability of institutions to create international research links through the output ratio that has been produced in collaboration with foreign institutions. The values are computed by analyzing the institution’s output whose affiliations include more than one country address.

Normalized Impact NI

Normalized Impact scores indicate the scientific impact that institutions have over the scientific community. In order to obtain a fair measurement of such impact, its calculation removes the influence due to institutions’ size and research profile making it ideal for comparing research performance. Normalized Impact values show the ratio between the average scientific impact of an institution and the world average impact of publications of the same time frame, document type and subject category. The values are expressed in percentages and show the relationship of the institution’s average impact to the world average, which is 1, –i.e. a score of 0.8 means the institution is cited 20% below world average and 1.3 means the institution is cited 30% above world average. Normalized Impact is computed using the methodology established by the Karolinska Intitutet in Sweden where it is named “Item oriented field normalized citation score average”. The long name used is because the normalization of the citation values is done on an individual article level. Further information on the methodology at Bibliometric Handbook for Karolinska Institutet .

High Quality Publications Q1(%)

Ratio of publications an institution publishes in the world most influential scholarly journals. Journals considered for this indicator are those ranked in the first quartile (25%) in their categories as ordered by SCImago Journal Rank SJR indicator.

photo

Félix de Moya Anegón is Research Professor at the Institute of Public Goods and Policies (IPP) from the Spanish National Research Council (CSIC), his academic interests include scientometrics, bibliometrics, research evaluation and science policy; he has published around 100 papers in these fields. He is SCImago Research Group‘s main researcher, where he has led renowned bibliometic projects including Scimago Journal & Country Rank, Scimago Institution Rankings and The Atlas of Science. Prof. De Moya is also advisor for Science Policy issues for national organizations of science and technology and research institutions around the world.

6 Responses

  1. CSIC researcher says:

    Bibliometric studies always present the same problem: they are one-dimensional. They only observe the publications activity. It is an important indicator to measure the research activity but it is not the only one. For instance, what is the matter with patents? CSIC is the first patenting institution in Spain, it has a strong applied science activity. The Instituto de Tecnología Química (ITQ) is the most important institue of the CSIC in patents and transfer technology as IQAC and IETCC. What are their ranks? This is the same if we talk about books, competitive funds, thesis, etc. Remember, all these indicators are the real “performance indicators” and the bibliometrics ones are just a representative part of these indicators.
    Talking about rankings, you say “Output values are affected by institution sizes and research profiles, among others factors” so Is there any normalization to solve this problem? No, the largest institutes are in the top of the rank, just like that. Is there any normalization between disciplines and areas? No, the Physics institutes (IFIC, IAA, IEM) are in the top because the colaborate more than other research areas, so they produce more documents. In other words, to rank pears with apples is just a nonsense and I hope that the research evaluation never trust in these fast and simple rankings.

  2. Scimago Lab says:

    CSIC researcher, we do not make all those claimings you are attributing to us. Scimago Lab is actually a bibliometric blog, so you should understand the expression “preformance indicators” in this context. We do not state anywhere that patenting activity as well as books, research projects, thesis, etc. should not be taken into account when evaluating (i.e. an institution), in fact if you need to evaluate all these dimensions of CSIC Institutes, this report is a great resource to extract bibliometric data.
    If you have read the post carefully, you will have noticed the part where we claim the following about the ranking “its main goal does not consist in making a ranking table of CSIC centers, but to highlight some of the differential characteristics involving the research outcomes achieved by these institutes”. Even the rank numbers are not included. Is it the ranking or you who is giving so much attention to the output ranks? Maybe the problem is to consider the ranking as a league table, which is it not, instead of a valuable bibliometric information resource to the institutions, policymakers and research managers so they are able to analyze, evaluate and improve their research results.
    We hope have clarified our goals a bit, since we are releasing a full series of reports on national council and academies of sciences from Germany, France, Italy, Russia, China, Ukraine, and possibly more (in addition to the actual Spainsh CSIC).

  3. Ignacio Torres Aleman says:

    Many thanks for making available this very useful information on our work.
    Regards,
    Ignacio Torres Aleman

  4. Jordi Isern says:

    Podeis publicar el valor absoluto de los indices medios de impacto de cada una de las areas?
    Institut de Ciencias del Espacio

  5. Dear friends,

    I think that you have done a great job. I see that your work has raised some criticism among directors of institutes which did not rank high. Perhaps it would be good to establish a period of consultation of the specific records in support of the final output and of fixing possible mistakes or missing data.

    However, this should not hidden that this work was needed and you did it well.

    All the best,

    Joan O. Grimalt

Powered by Wordpress