Chapter 4: Image processing in the digital humanities
Chapter 4 discusses the ways in which images, once digitized, can be manipulated, studied and processed. Melissa Terras shows how image-processing techniques may be used to reconstruct ancient Theran wall paintings or to help us to read ancient documents, such as the Vindolanda Tablets from Hadrian’s Wall. She discusses different processing techniques and research methods and the new discoveries that these have made possible. In addition to this, Terras explores why image processing is not a commonly used procedure in digital humanities and advises on how an individual may undertake research in this area.
Case Study: The Griphos project: reassembling ancient frescoes
Benedict Brown, Department of Computer and Information Science, University of Pennsylvania, Antonio García Castañeda, Department of Computer Science, University College London, David Dobkin, Department of Computer Science, Princeton University, Thomas Funkhouser, Department of Computer Science, Princeton University, Lara Laken, Auxilia, Radboud University Nijmegen, Szymon Rusinkiewicz, Department of Computer Science, Princeton University, and Tim Weyrich, Department of Computer Science, University College London
The reconstruction of fragmented objects is of great interest in archaeology, where artefacts are often found in a fractured state. Vast quantities of fresco material, commonly excavated from archaeological sites in thousands of pieces, remain in storage indefinitely, unexamined and unstudied, as resources are simply not available to complete time-consuming manual searches for any fragments that match. Frescoes are usually of unknown design, with many pieces of the same colour, with few clear distinguishing features and often with missing sections. Fragments are generally fragile, and excessive handling is undesirable, at best, and highly damaging, at worst.
The Griphos Project (in Greek, griphos means ‘puzzle’ or ‘riddle’) was established to develop computational methods to aid in the reconstruction of fresco fragments. Griphos has been utilized, so far, to help piece together Bronze Age wall paintings from the site of Akrotiri on the volcanic island of Thera (modern-day Santorini, Greece) and Roman frescoes from a luxurious mansion in the Roman city of Municipium Tungrorum (modern-day Tongeren, Belgium). The computer scientists developing the system work closely with archaeologists and conservators, and the project team incorporates partners from University College London, Princeton University, the Katholieke Universiteit Leuven, the University of Athens, the University of Ioannina and the Vlaams Instuut voor het Onroerend Ergoed.
The first step in aiding the sorting of fresco fragments is digitization. Each fragment is placed face-down on a turntable, scanned via a 3D scanning system (see Figure below), turned face-up and scanned again. Alignment is performed automatically, with the operator verifying the results and correcting any errors. A flatbed scanner is used to capture high resolution colour images and texture information of the front surface of each fragment, with the back surface being scanned once for documentation purposes. This yields a throughput of approximately ten fragments per hour, made possible through the bespoke, end-to-end design of the acquisition pipeline.
The digital surrogates of the fresco pieces are utilized in the Griphos software, which functions as a virtual table top, on which the user places fragments and arranges them for assembly, documentation or further inspection. Griphos maintains a high level of realism in an interactive viewer, which combines full 3D models and layered 2D representations. The user can have many different virtual table tops, each containing an arbitrary number of fragments, to allow multiple, competing reconstructions alongside the current physical arrangement (see Figure below).
Griphos provides a variety of tools to study the relationship between fragments. Related pieces may be grouped together manually or snapped to each other automatically, based on computationally matched proposals, which can then be evaluated and interactively explored. An initial placement may be refined using algorithms, which provide a convenient visualization of the alignment quality, showing a colour-coded plot of where fragments touch, intersect or contain gaps. A grid is provided to aid in reconstructing hypothetical layouts and in generating overviews of fragment combinations, which saves time, as these were previously prepared by hand.
There are currently 1516 fresco fragments from Tongeren, which, statistically, means there are millions of possible match suggestions. Griphos proposed 302 matches, of which 170 were previously known, helping establish the most likely combinations for further examination. Confirming them took an expert four weeks of digital sifting, before verifying the matches against the original material. Based on discussions with expert archaeologists, we can state, with confidence, that our combination of acquisition- and match-browsing provides a dramatic increase in efficiency and lowers the cost of matching, compared to traditional methods. (It is difficult to quantify this exactly, as experts have been trying to piece together the fragments for years, and there is little documentation available regarding this process and the physical state of the fragment corpus).
In addition to its value for identifying combinations of fresco pieces, the application of our system to an entire excavation has confirmed several other major advantages over traditional matching, including demonstrating that digital matching can be undertaken by unskilled workers with minimal archaeological training and that matching can be safely done with minimal handling of fragile primary source material. Moreover, physical access to the material is only needed for acquisition and final verification and assembly; the software can be used to search for fragment matching remotely, at any time. This opens up possibilities for future crowdsourcing of the matching process, and our team is currently exploring viable options. The virtual table tops are quicker to produce, more accurate, easier to read and easier to update than traditional outline drawings of fresco fragments. A by-product of the Griphos system is high quality digital surrogates of each fragment piece (1/4mm resolution 3D models and 600dpi images) organized in a database that can be used for further study, as well as for producing reproductions for catalogues and publications.
In addition to helping to piece together wall paintings from these two sites, the databases of digitized fragments are large enough to evaluate the performance of the system under development and the appropriateness of the computational algorithms which have been appropriated for this task. The proposed matches have proved important for analysing our ranking results and finding ways to improve them. While much of our previous work has focused on improving the quality of our matching algorithms, by improving the ranking of proposals (Funkhouser et al., 2011), based on edge shape, colour, surface variations and other cues (Toler-Franklin et al., 2010), current efforts move on to the search for clusters and globally consistent assemblies of fragments (García Castañeda et al., 2011), as well as the analysis of fracture processes and characteristic fracture patterns (Shin et al., 2011).
Our consortium is currently trying to raise funding for the acquisition and assembly of a new fresco at Akrotiri. This would provide controlled conditions to evaluate the success of the system in piecing together a ‘virgin’ fresco. Further information about the Griphos project can be found at www.cs.ucl.ac.uk/staff/t.weyrich/griphos/.
Case Study: Developing Image capture and processing techniques to enhance the legibility of incised texts
Ségolène Tarte, Oxford e-Research Centre, University of Oxford, Mike Brady, Department of Engineering Science, University of Oxford, Alan Bowman, Centre for the Study of Ancient Documents, Classics Faculty, University of Oxford, and Melissa Terras, Department of Information Studies, University College London
Incised texts are often very difficult to read. Deciphering such texts draws on a wide range of expertise that papyrologists develop throughout their career. These skills are as much related to visual perception of the artefact and its text as they are to palaeographical, linguistic and historical knowledge. Computational tools can be used to aid papyrologists in their complex task, by providing alternative visual clues and evidence regarding textual incisions on documentary material. This case study presents methodologies that have been adopted to enhance the legibility of the Roman wooden stylus tablets from Vindolanda, on Hadrian’s Wall, which consists of incisions left in wood through a now-perished coat of wax. The first step in deciphering these three-dimensional texts is by using advanced digitization techniques, called Shadow Stereo, and reflectance transformation imaging to capture and encode multiple images of the text under varying illumination conditions for further processing and visualization. In addition, image processing algorithms were developed to isolate the features of the text and aid in their interpretation.
Digitization of such artefacts is the first step in the development of an interpretation of the document. Observing the experts trying to interpret the stylus texts informed us as to how the tablets could be digitized, whilst retaining, in particular, one major piece of information that papyrologists exploit when deciphering it – namely, the volumetric nature of the text. Experts who have access to the actual tablet they intend to transcribe have developed a very specific and intuitive strategy to enhance the visibility of the incisions that the stylus left in the wood. They lay the tablet flat on their hand, lift it up at eye level against a light source and apply pitch-and-yaw motions to the artefact. This effectively enhances the visibility of the incisions, by accentuating the highlights and shadows that the raking light generates; the lower the light, the longer the shadows projected by the text in (inverted) relief, created by the incisions carved on the surface of the tablet. The principle that is put into application here is the ‘shadow-stereo’ principle, by which concave shapes are revealed from shading, and the motion of the shadows exposes the location of the incisions (Brady et al., 2005). This process can be digitally imitated: a digital camera is affixed above a tablet, and a high resolution digital picture is taken for each one of a set of pre-established positions of a light source around the tablet. Each light position is described by an elevation angle and an azimuth angle, where the azimuth angle corresponds to an angle deviation from the horizontal in the plane of the tablet and the elevation angle describes the height of the light, with respect to the plane of the tablet. Both angles are measured from the centre of the tablet.
The collected data for each tablet was further used to digitally recreate the shadows and their motion. By adopting an appropriate model of image formation, not only can one store the set of images efficiently, without having to store each image, one can also interpolate between light positions, thus simulating lighting conditions, where a picture was not actually captured. The image model that is adopted relies on this technique of ‘reflectance transformation’ (Malzbender, Gelb and Wolters, 2001; Goskar and Earl, 2010). This digital model can be remotely used by experts who do not have access to the original text, allowing them to digitally pitch-and-yaw this avatar of the artefact to aid them in interpreting the text: by exploiting the play of light with the 3D nature of the textual representation, we are able to enhance the text’s legibility.
Image processing algorithms were developed to isolate textual features. Noise removal homogenizes the background of images by removing patterns or behaviours that are irrelevant; we can increase legibility, by neutralizing some of the distracting noise present in the images, through homomorphic filtering and wood-grain removal. Image segmentation algorithms were then used to extract the meaningful areas or features of the image and, thereby, identify where the text is located on the complex, abraded surface of the texts. There is no room to provide adequate explanation of these techniques in this case study, but to summarize: background correction is first performed; then ways to achieve text feature extraction. Phase congruency, which exploits the fact that visual features are detected for some properties of the local phase of the image in the Fourier Domain, and Markov Random Fields, which take a statistical approach to region labelling for image segmentation, were employed. Explanation of these techniques can be found in Tarte et al. (2010), for those interested in further detail.
Most techniques utilized in this project were inspired by approaches adopted by medical image processing. However, these methods had to be largely adapted to our specific application; in general, the images of artefacts and their features of interest are not only different from, and noisier than, medical images, but, also, the type of visual expertise required to detect the textual features differs greatly from that of radiologists.
The challenge of image processing and capture for ancient incised documents is multiple and specific to both the imaging method and the artefact. Our research suggests that by observing and understanding the nature of the classicist’s visual expertise, we will be able to integrate prior knowledge into a model of visual perception adapted to the classicists needs, hence, supporting them in building meaning out of a pure signal – in building an interpretation of an artefact. Digital imaging and image processing can create novel, alternative visual clues as to writing contained within incised texts, assisting papyrologists in reading damaged, ancient texts.
This work was undertaken as part of the Image, Text, Interpretation: e-science, technology and documents project (also known as eSAD: e-science and ancient documents) at the University of Oxford, with input from University College London, which was funded under the AHRC-EPSRC-JISC Arts and Humanities e-Science Initiative. Further resources can be found at http://esad.classics.ox.ac.uk/.
• Brady, M., Pan, X., Schenck, V., Terras, M., Robertson, P. and Molton, N. (2005) Shadow Stereo, Image Fltering and Constraint Propagation. In Bowman, A. K. and Brady, M. (eds), Images and Artefacts of the Ancient World Vol. 4, British Academy Occasional Paper, Oxford University Press/British Academy.
• Funkhouser, T., Shin, H., Toler-Franklin, C., Castaneda, A. G., Brown, B., Dobkin, D., Rusinkiewicz, S. and Weyrich, T. (2011) Learning How to Match Fresco Fragments, Eurographics 2011 Special Area Track on Cultural Heritage, April , http://doi.acm.org/10.1145/2037820.2037824.
• García Castañeda, A and Brown, B and Rusinkiewicz, S and Funkhouser, T and Weyrich, T (2011) Global consistency in the automatic assembly of fragmented artefacts. In: Dellepiane, M and Niccolucci, F and Pena Serna, S and Rushmeier, H and Van Gool, L, (eds.) 12th international symposium on virtual reality, archaeology, and cultural heritage. (pp. 73 – 80). Eurographics Association: Goslar, Germany.
• Goskar, T. A. and Earl, G. P. (2010) Polynomial Texture Mapping for Archaeologists, British Archaeology, 111 (111), 28–31.
• Malzbender, T., Gelb, D. and Wolters, H. (2001) Polynomial Texture Maps. In Pocock, L. (ed.) ACM SIGGRAP’01: proceedings of the 28th annual conference on computer graphics and interactive techniques held on 12-17 August 2001, Los Angeles, CA, USA. organized by ACM.
• Shin, H., Doumas, C., Funkhouser, T., Rusinkiewicz, S., Steglitz, K., Vlachopoulos, A. and Weyrich, T. (2011, forthcoming) Analyzing and Simulating Fracture Patterns of Theran Wall Paintings, ACM Journal of Computing and Cultural Heritage (JOCCH).
• Tarte, S., Bowman, A., Brady, M. and Terras, M. (2010) Image Capture and Processing for Enhancing the Legibility of Incised Texts. In Holappa, M. (ed.O , Eikonopoiia: Digital imaging of ancient textual heritage: technological challenges and solutions held on 28–29 October 2010, held on 28–29 October 2010, organized by the Centre of Excellence (Academy of Finland)Helsinki , www.eikonopoiia.org/files/Eikonopoiia-2010-Proceedings.pdf.
• Toler-Franklin, C., Brown, B., Weyrich, T., Funkhouser, T. and Rusinkiewicz, S. (2010) Multi-Feature Matching of Fresco Fragments, ACM Transactions on Graphics (Proc. SIGGRAPH Asia 2010), 29 (5), 185: 1–185: 11 , http://dx.doi.org/10.1145/1882261.1866207.