The UK Reproducibility Network (UKRN) has published a new report summarising the results of six pilot projects exploring whether UK universities could one day monitor open research practices using robust, automated indicators. The pilots, delivered as part of the UKRN Open Research Programme (ORP), tested the feasibility of using institutional and third-party datasets to track behaviours such as data sharing, pre-registration, and contributor roles at scale.
Led voluntarily by academic and professional staff across the UK, and supported by international data service providers, the pilots were designed as exploratory groundwork rather than attempts to set sector-wide benchmarks. Their purpose: to understand what might be possible, what gaps remain, and what would be needed to build future systems for monitoring responsible and open research.
Six areas of open research under scrutiny
The projects investigated six key areas:
- FAIR data
- Open data
- Data availability statements (DAS)
- Downstream effects of sharing data and other outputs
- Pre-registration
- Use of the CRediT taxonomy
Across all themes, teams assessed data quality, trialled automated extraction tools, and compared machine-generated results with human validation.
Where automation shows promise
The report finds clear potential for building automated indicators in several areas. Pre-registration emerged as one of the most promising, with relatively good-quality metadata available. Elements of FAIRness—such as detecting persistent identifiers and metadata completeness—also produced encouraging results. The automated extraction of CRediT contributor roles appears feasible with further refinement and standardisation.
…but we aren’t quite ready
Despite these successes, most pilots concluded that current tools, metadata standards, and research workflows are not yet mature enough to support reliable automated monitoring at institutional level. Manual checking remains essential both for ensuring accuracy and for validating emerging automated methods.
A recurring challenge was the lack of consistency across the research ecosystem. Terminology, metadata fields, DAS formats, repository practices, and even basic article type definitions vary widely between publishers, institutions, and platforms. These inconsistencies make automation difficult and undermine comparability across data providers.
A call for shared standards and better infrastructure
The report highlights an urgent need for coordinated development of shared definitions, taxonomies, and data standards. A trusted international body—such as the Open Science Monitoring Initiative (OSMI)—could play a key role in this work.
Improving interoperability and metadata quality across repositories, publication platforms, and submission systems will also be essential. Better use of persistent identifiers, structured DAS templates, and aligned workflows would benefit both researchers and institutions.
Lessons for future large-scale initiatives
While the pilot teams demonstrated strong commitment to transparency and sector leadership, the voluntary nature of the work and differing project management approaches created challenges. The report recommends that future initiatives of this scale be properly resourced, centrally coordinated, and sequenced to support interdependent work packages.
Looking ahead
The overall message is one of cautious optimism: some degree of automated monitoring of these specific open research practices is achievable, but progress will require collective action. Shared standards, improved infrastructure, and responsible approaches to metrics will be vital, alongside continued manual validation during the transition.
The pilots offer an important first step towards a more coherent, evidence-based understanding of some open research practices and lay the groundwork for more mature monitoring systems in the years to come.
Note
You may be wondering why this is UKRN Working Paper #9, released before Working Paper #8. Working Paper #8 will be the report of the Open Research Programme survey, which will be released very soon.