The availability of large open bibliographic databases has changed the way we think about access to scholarly information. Several databases such as OpenAlex, OpenAIRE and The Lens now contain hundreds of millions of records, providing researchers with access to a volume of information that would have been difficult to imagine only a few years ago, but the question is whether having more records necessarily mean having better data for bibliometric analysis?

To explore this question, we compared three major open databases with Scopus, looking at more than 650 million records published between 1996 and 2024. We looked not only at the number of records these databases cover, but also at some of the basic metadata researchers need to use these records for research evaluation, including DOIs, ISSNs, document types, affiliations, citations and references.

Results

The results show an interesting trade-off. While open databases provide much broader coverage, the same level of metadata completeness does not always accompany the additional records. For example, more than 55% of records in the open databases are missing affiliation information. DOI and ISSN coverage also varies, and these databases do not always consistently classify document types.

This matters because a bibliographic record is more than a title and a citation. If we want to evaluate the research output of an institution, we need reliable affiliation information. The same principle applies also to journals whereas if we want to compare journals, we need to know which source a publication belongs to, and if we want to calculate citation indicators, we need reliable citation and reference data.

The current state of open data sources indicates that they prioritize coverage, bringing together information from a very large number of sources, opposed to curated databases such as Scopus which place greater emphasis on selectivity, standardization and consistency. Neither approach is necessarily better for every purpose, they merely make different choices about what is most important.

This distinction becomes particularly important as institutions and policymakers consider moving toward more open research information. Access to a large dataset is an important first step, but it does not eliminate the need to understand what is actually in the data, how consistent and accurate it is and whether we can safely use it for evaluation.

Therefore, we should think about how much of the data do we need, and how reliable is it for the question we are trying to answer?

Read the full research paper and explore our detailed analysis here: A Comparative Analysis of Open and Commercial Bibliographic Infrastructures: Scale, Metadata Standardization, and Implications for Bibliometric Evaluation