Chapter 1: Studying users in digital humanities
There is little point in creating digital resources in either sector if they are not used, however. We know that people use digital resources if they fit their needs, yet many digital humanities resources are still designed without reference to user requirements. This often means that expensive digital resources remain unused or unappreciated by their intended audience. In Chapter 1, Claire Warwick demonstrates why it is important to seek to understand the behaviours of humanities researchers and the users of cultural heritage resources, in the context of digital tools and resources, so that they may be designed to be more usable and sustainable in the future. She discusses different methods of carrying out such studies and shows why they should be introduced at the beginning of the project and not simply in its later stages. She presents recommendations, based on UCLDH research, for good practice in the design of digital resources, such that they are as appropriate as possible for their users.
Case Study: The LAIRAH Project: log analysis of internet resources in the arts and humanities
Claire Warwick, Melissa Terras, Paul Huntington, Nicoleta Pappa, and Isabel Galina, UCL Department of Information Studies
The aim of the LAIRAH survey is to discover what influences the long-term sustainability and use of digital resources in the humanities through the analysis and evaluation of real-time use. We utilized deep log analysis techniques to provide comprehensive, qualitative and robust indicators of digital resource effectiveness. Our aims were to increase understanding of usage patterns of digital humanities resources, aid in the selection of projects for future funding and enable us to develop evaluation measures for new projects.
Until we carried out our research, evidence of actual use of projects was anecdotal; no systematic survey had been undertaken, and the characteristics of a project that might predispose it for sustained use had never been studied.
Phase 1: log analysis
The first phase of the project was deep log analysis: we were the first team ever to analyse web transaction logs to measure user behaviour within digital humanities resources. Transaction and search log files were provided by three online archives that were supported by the Arts and Humanities Research Board (AHRB) (now the Arts and Humanities Research Council (AHRC)): the Arts and Humanities Data Service (AHDS) Arts and Humanities Collection, Humbul Humanities Hub and the Artefact Database for the Creative and Performing Arts. These provided rich data for comparing metrics between subject and resource type. The search logs showed which resources users were interested in and which ones users subsequently visited.
We analysed at least a year’s worth of transaction log data (a record of webpage use automatically collected by servers) from each resource. This data provided a relatively accurate picture of actual usage, providing: information on the words searched (search logs), the pages viewed (user logs), the website that the user has come from (referrer logs) and basic, but anonymous, user identification tags, time and date stamps.
Phase 2: case studies
We selected a sample of 21 projects that the log analysis indicated to have varying levels of use – chosen to give us coverage of different subject disciplines – to be studied in greater depth. We classified projects as ‘well used’ if the server log data from the AHDS and Humbul portals showed that they had been repeatedly and frequently accessed by a variety of users. We also mounted a questionnaire on these sites and asked which digital resources respondents found most useful. Although most users nominated information resources, such as libraries, archives and reference collections, such as the eDNB, three publicly funded UK research resources were mentioned, and, thus, we added them to the study. We also asked representatives of each AHDS centre to name which resources in their collections they believed were most used. In the case of Sheffield University, the logs showed that a large number of digital projects accessed were based at the Humanities Research Institute (HRI). We therefore conducted interviews about the HRI and its role in fostering the creation of digital humanities resources.
The projects were studied in detail, including any documentation and reports that could be found on the project’s website, and a representative of each project was interviewed about project development, aims, objectives and their knowledge of subsequent usage. We analysed each project’s content, structure and design. We asked whether it undertook any outreach or user surveys and how the results of surveys were integrated into project design. We also asked what kind of technical advice the project received, whether from institutional support people, from humanities computing centres or from central bodies, like the AHDS. All these measures are intended to determine whether there are any characteristics shared between ‘well used’ projects.
We also studied projects that appeared to be neglected or underused. A small group of humanities users were asked to investigate a sample of digital resources: half were well used and the others neglected, but their status was not initially revealed. A hands-on investigation was followed by a discussion of factors that might encourage or deter future use of such resources. We aimed to find out whether their lack of use was because users had not heard of a given resource or whether there were more fundamental problems of design or content that would make the resource unsuitable for academic work.
We found that roughly one-third of all projects appeared to be unused. When asked to evaluate unused resources, users were able to identify several problems with design and content. They were deterred from use because of unintuitive interfaces, the need to download data for use in another application, confusion as to what the content might be used for and even a confusing name. They also needed more information about the content of resources, how and why it had been selected and the expertise of the project team.
Well-used projects did share common features that predisposed them to success. The effect of institutional and disciplinary culture in the construction of digital humanities projects was significant. We found that critical mass was vital, as was prestige within a university or the acceptance of digital methods in a subject field. The importance of good project staff and the availability of technical support also proved vital. If a project is to be well used, it was also essential that information about it should be disseminated as widely as possible. The single most common factor in use of a project was a good dissemination strategy. Even amongst well used projects, however, we found areas that might be improved: these included organized user testing, the provision of, and easy access to, documentation, and the lack of updating and maintenance of many resources.
Digital humanities projects should undertake the following actions:
1. Keep documentation and make it available from the project website, making clear the extent, provenance and selection methods of materials for the resource.
2. Have a clear idea of whom the expected users might be; consult them as soon as possible and maintain contact through the project, via a dedicated e-mail list or website feedback.
3. Carry out formal user surveys and software and interface tests and integrate the results into project design.
4. Have access to good technical support, ideally from a centre of excellence in digital humanities.
5. Recruit staff who have both subject expertise and knowledge of digital humanities techniques, then train them in other specialist techniques as necessary.
6. Maintain and actively update the interface, content and functionality of the resource, and do not simply archive it.
7. Disseminate information widely, both within specialist subject domains and in digital humanities.
Case Study: The VERA project
Claire Fisher, British Museum, Melissa Terras, UCLDH, and Claire Warwick, UCLDH
The Virtual Environments for Research in Archaeology (VERA) project was funded as part of the Joint Information Systems Committee (JISC) and involved the Department of Archaeology and the School of Systems Engineering at the University of Reading, the York Archaeological Trust and the School of Library, Archive and Information Studies at UCL. The project was based around the University of Reading’s well established excavation at the Roman site of Silchester. The Silchester Insula IX project provided the ideal test-bed for a Virtual Research Environment (VRE) project, because key to the smooth running of this large, urban excavation was the Integrated Archaeological Database (IADB). Used for recording, analysis, archiving and online publication of archaeological finds, contexts and plans, the IADB allows integrated access to all aspects of the excavation record. It was used to populate the VRE with data generated by a complex urban excavation.
The VERA project set out to:
1. Investigate how digital input could be used to enhance the process of documenting, utilizing and archiving excavation data.
2. Create a suitable Web portal to provide an enhanced user experience.
3. Develop tools that could be integrated with existing practices of research archaeologists unfamiliar with VREs.
4. Test the tools in a real world setting.
UCL’s role was to ensure that the needs of the archaeologists and associated specialists remained at the heart of developments.
The VERA IADB usability study was carried out at the 2007 VERA winter workshop at Reading. The development of the IADB has always been driven by its users and has developed alongside their working practices. However, this was the first time that user reactions to the IADB had been formally documented. Participants at the workshop were divided into two groups:
• those with no (or little) experience of using the IADB, designated ‘novice users’
• those who have experience of using the IADB in their work, designated ‘experienced users’.
The usability study provided the team with useful information about user perceptions, plus details of the typical tasks carried out by archaeologists and associated specialists. The novice users felt that they could quite quickly get to grips with the system; the experienced users carried out a wide range of tasks using the IADB and used it at (almost) all stages of various projects.
The Silchester project utilizes the skills of a large and geographically disperse group of specialists. Each specialist uses the IADB for varying purposes, and one of the aims of the VERA project was to enhance the ways in which each researcher uses it. Interviews were carried out to explore how the existing users organize their work; to discuss their experiences of working with the IADB; to find out to what extent the IADB met their needs; and if any changes might make their work easier. The results from these interviews were used and fed into IADB development.
Excavation data has traditionally been entered into the IADB through manual digitization, usually once the excavation season is over. A key aim of the VERA project was to investigate the use of information technology (IT) within the context of a field excavation and to ascertain whether it may be appropriated to speed up the process of data recording, entry and access. From 2005 onwards, a number of field trials had been carried out at the Silchester excavation, using a variety of digital recording methods, including digital pens and notebooks and hand-held internet browsers. The 2008 field season focused on the use of digital pens for direct digital data gathering. We used fieldwork observations, user needs discussions, a diary study and an end-of-season questionnaire to analyse user reactions to the digital pens.
We aimed to observe how well the digital pens fitted in to the workflow of the site and to record user feedback. The discussions provided the framework for creating the end-of-season review for the digital pens. A diary study was used to gather information about different archaeological roles and the way that they are supported by both digital and analogue technologies. These studies allowed the VERA team to explore how the implementation of new technology affected the workflow of the site. Lessons learnt included the central role of the traditional context-recording sheet and the need for any new technology to integrate with existing workflows.