Our colleague Catia Pesquita from Lisbon University is presenting a paper, co-authored by Valentina Ivanova and Patrick Lambrix of our group and Steffen Lohmann from Fraunhofer, at EKAW 2018 – 21st International Conference on Knowledge Engineering and Knowledge Management, on a framework to conduct and report on empirical user studies in Semantic Web contexts. As organizers of the VOILA workshops on Visualization and Interaction for Ontologies and Linked Data at the recent ISWC conferences, we noticed that both the assessment of interactive Semantic Web approaches as well as the reporting on conducted user studies still suffer from weaknesses.
Empirical evaluations are essential to compare different approaches, demonstrate their benefits and reveal their drawbacks, and thus to facilitate further adoption of Semantic Web technologies. In our paper, we review empirical user studies of user interfaces, visualizations and interaction techniques recently published at several Semantic Web venues, assessing both the user studies themselves and their reporting. We then chart the design space of available methods for user studies in Semantic Web contexts. Finally, we propose a framework for their comprehensive reporting, taking into consideration user expertise, experimental setup, task design, experimental procedures and results analysis.
The original publication is at doi. A self-archived postprint version of the article is available at Linköping University Institutional Repository (DiVA).