Table 1 Methods for expert stakeholder engagement in ontology development
| Method | Description | Examples |
|---|---|---|
| Advisory Board and Working Groups |
Small and select group of domain experts evaluate progress of ontology development and providing unbiased strategic and scientific recommendations. Delivered via in-person or online meetings, or by email |
Exposure Ontology (EXO) [35] Human Phenotype Ontology (HPO) [36] Scientific Advisory Board of the Human Behaviour-Change Project [23] |
| Expert Panels | ||
| Feedback tasks |
Experts asked to provide feedback on the whole ontology or specified elements of it Delivered in-person or using online survey tools (e.g. Qualtrics, Limesurvey, Google Forms). |
Development of the Behaviour Change Intervention ontology (BCIO) within the Human Behaviour-Change Project [29, 30, 37] Related example from social and behavioural sciences: Development of COVID-19 research priorities in psychology [38] Development of the Theories and Techniques Tool [39] Self-Determination Taxonomy [40] |
| Consensus exercises |
Consists of two or more sequential rounds of questions and feedback to experts, aiming to achieve consensus. Delivered using online consensus (e.g. Loomio, DelphiManager) or survey tools (e.g. Qualtrics) |
Related examples from social and behavioural sciences: Links between Behaviour Change Techniques and Mechanisms of Action [41] Delphi consensus to identify priorities for methodological research in behavioural trials in health research [42] Behaviour Change Techniques Taxonomy v1 [21] |
| Discussions |
Experts are invited to engage in online discussions with other experts on a specific topic, e.g. relationships between certain ontology entities. Delivered using project websites or online consensus tools (e.g. Loomio, DelphiManager). It can be synchronous or asynchronous. |
Cognitive Atlas [43] Related examples from social and behavioural sciences: Links between Behaviour Change Techniques and Mechanisms of Action [41] |
| Workshops |
Small-group discussions and presentation of specific proposals related to ontology development. Decisions are taken by the group during the workshop. Delivered via in-person or online meetings |
Computable Exposures Workshop [44] NCI Semantic Competency Query Review [45] Phenotype Ontologies Traversing All The Organisms (POTATO) workshop [46] Workshops at International Conference of Biomedical Ontologies (ICBO) Workshops at bi-annual Formal Ontologies in Information Systems Related examples from social and behavioural sciences: Current development of the E-cigarette ontology [47] |