X Close

Transcribe Bentham


A Participatory Initiative


Archive for the 'READ project' Category

Project Update – So long to Transcribe Bentham

uczwlse23 April 2019

Hello! I’m checking in with a final blog post before I move on to my new job at The National Archives. My last day in the office will be 26th April.  If you need any assistance after I’ve left, please contact transcribe.bentham@ucl.ac.uk and Bentham Project staff will pick up your message.

Working at the Bentham Project has been an incredibly rewarding experience and one which has certainly caused me more pleasure than pain over the past four years. I hope that in my own small way I have contributed to an improved understanding of Bentham’s life and philosophy.

I knew that transcription would be a large part of my job when I was asked to transcribe a Bentham manuscript during my job interview! Hopefully my eye for Bentham’s handwriting has improved somewhat since then but I still feel a sense of trepidation when I encounter a page from the 1830s (when Bentham’s handwriting was at its worst!).

After initially supporting my Bentham Project colleagues with the editing and researching of Bentham’s papers, I became the coordinator of the Transcribe Bentham crowdsourcing initiative in January 2016. It has been a real privilege to work alongside such a dedicated group of volunteers, many of whom have been with the project for several years. They have blossomed into extraordinarily skilled transcribers who bravely tackle Bentham’s writings, which can at times be chaotic and convoluted. Our volunteers play an invaluable role in helping the Bentham Project to complete the mammoth task of transcribing thousands upon thousands of pages left behind by Bentham. We endeavour to ensure that our volunteers feel part of our community, sending them a monthly newsletter and encouraging them to take part in events like the Bentham Hackathon. I was delighted to be able to support volunteers to transcribe 20,000 pages of Bentham’s writings, an important milestone which was reached in April 2018. Our transcribers add to this total daily and I will enjoy watching their progress continue from afar. I would like to give huge thanks to all of our volunteers for their patience, hard work and company over the past few years. They are living proof that amazing things can be achieved through collaboration between researchers and members of the public.

Transcribe Bentham volunteers Annette Brindle, Simon Croft and Gill Hague celebrating the complete digitisation of Bentham’s papers in June 2018.

I’ve also had the chance to experiment with new ways of transcribing documents thanks to the Bentham Project’s role in the EU-funded READ project. Handwritten Text Recognition (HTR) technology is progressing rapidly and computers are getting better at tackling difficult manuscripts like those written by Bentham. As reported here on the blog, we’re now able to use the Transkribus platform to recognise Bentham’s hand with an average Character Error Rate of just 9% (meaning that 91% of characters are transcribed correctly by the machine). We also have a Keyword Spotting interface where users can work with HTR to search the entirety of the Bentham papers. I’ve been lucky enough to present our results internationally, from the Nordic Countries to the Balkans, and have taught hundreds of academics and archivists in regular Transkribus workshops. I’ve enjoyed spreading the word about Transkribus far and wide through writing the READ project’s blog, newsletter and social media and organising the annual Transkribus User Conference. I have high hopes for the future of HTR and look forward to seeing volunteer skills and machine recognition integrated in a future version of Transcribe Bentham.

On the judging panel at the 2017 Bentham Hackathon.

Of course I need to thank all of my colleagues, in the Bentham Project, in UCL Laws and in the READ project, for the inspiration, support and fun. I have learnt a lot about Bentham, historical research, archives, scholarly editing, digital humanities, public engagement, crowdsourcing and much more, which will all stand me in good stead for my new role at The National Archives.

Project update – master Bentham’s handwriting with Transkribus Learn

uczwlse18 January 2019

Over the years, our volunteers have developed an enviable expertise in deciphering Bentham’s decidedly difficult handwriting.  They can even transcribe pages like this! By contrast, many newcomers are understandably daunted by Bentham’s scrawl – they may start transcribing a page one day but then never return.

A new e-learning website, produced by the University of Innsbruck as part of the READ project, promises to help anyone and everyone get to grips with all kinds of historical handwriting. Transkribus Learn does not replace systematic paleography training but it allows users to practice reading and transcribing individual words, learning as they go.

To try it out:

Transkribus Learn has two transcription modes – ‘Study’ and ‘Test’.

In the former, users can guess and then reveal the transcription of individual words in a manuscript.  In the latter, users will be prompted to transcribe the missing word in a series of examples. At the end, you receive your score and a list of correct and incorrect answers. You can keep studying and testing yourself, as often as you like.

There are two Bentham collections on the site – categorised as ‘easier Bentham (containing writing by Bentham and his secretaries) and ‘difficult Bentham’. Both collections are an ideal training ground for new volunteers, offering the opportunity to practice transcribing different words in rapid succession.

We recommend that new volunteers start with the ‘easier Bentham’ and move onto the more difficult pages once they feel ready.  I hope some of our long-standing volunteers might also have a play and challenge themselves to ready some of Bentham’s nastiest handwriting!

What does that say?? Practice transcribing Bentham and more with Transkribus Learn

Access the Bentham material in Transkribus Learn:

As one of the partners in the READ project, the Bentham Project helped to develop Transkribus Learn. But there’s so much more than Bentham to discover on the site. The site currently contains scripts from the 12th to the 19th centuries in a range of languages. Users can also upload their own documents to the platform as a training exercise for students or volunteers.

The Transkribus team look forward to helping a broad range of people learn valuable new transcription skills!  They welcome any feedback or questions (learn@transkribus.eu).

Project Update – Improving the Automated Recognition of Bentham’s handwriting

uczwlse28 November 2018

As our volunteer transcribers know, getting to grips with Bentham’s handwriting can be a steep learning curve.  Bentham never wrote particularly neatly and his scrawl became increasingly difficult to comprehend as he grew older.  Since 2013, the Bentham Project has been experimenting with advanced machine learning technology via the Transkribus platform in an attempt to train algorithms to automatically decipher Bentham’s handwriting.  And we have lately seen vastly improved results!

Read about our progress with Handwritten Text Recognition (HTR) technology and the Transkribus platform in blog posts from June 2017, February 2018 and October 2018.

HTR technology is open to anyone around the world thanks to Transkribus and the READ project. Once users have installed the platform, they can set about processing images and transcripts as training data for automated text recognition.  The software uses computational models for machine learning called neural networks.  These networks are trained to recognise a style of writing by being shown images and transcripts of that writing.  Anyone can start a test project in Transkribus by uploading around 75 pages of digitised images to the platform and transcribing each page as fully as possible.  The software learns from everything it is shown and so the more pages of training data, the better!  Find out more about getting started with Transkribus in the Transkribus How to Guides.

When we started working with HTR, it is fair to say that we were somewhat uncertain about the capabilities of the technology.  So we decided to focus on training a model to recognise some of the easier papers in the Bentham collection – those written by Bentham’s secretaries who tend to have neat handwriting.  Using around 900 pages of images and transcripts, we trained a model that is now publicly available to all Transkribus users under the name ‘English Writing M1’.  This model can produce transcripts of pages from the Bentham collection with a Character Error Rate (CER) of between 5-20%.  It produces good transcripts of pages written by Bentham’s secretaries but struggles to decipher Bentham’s own hand.

So our next challenge was to improve the recognition of Bentham’s most difficult handwriting.  For the past 18 months we have been continually creating training data in Transkribus based on very complex pages from the Bentham collection, periodically retraining HTR models and then assessing the results.  Until recently, our best result was a model trained on 81,000 transcribed words (around 340 pages) which used the ‘English Writing M1’ model as a base model.  By using a base model, Transkribus users can give the system a boost and ensure that it builds directly on what it has already learnt from the creation of an earlier model.  In this case, our resulting model could produce transcripts with an average CER of 17.75%.

The great thing about working with Transkribus is that the technology is improving all the time, thanks to the efforts of the computer scientists who work on the READ project.  The latest innovation is HTR+, a new form of Handwritten Text Recognition technology formulated by the CITlab team at the University of Rostock.  HTR+ is based on TensorFlow, a software library developed by Google.  It is works similarly to the existing HTR but processes data much faster, meaning that the algorithms can learn more quickly and so produce better results.  We used HTR+ to train a model on 140,000 transcribed words (or 535 pages) of Bentham’s most difficult handwriting.  This model can generate transcripts with a CER of around 9%.

An automated transcript of very difficult handwriting, using our latest HTR+ model. Image courtesy of UCL Special Collections.

HTR+ is not yet available to all Transkribus users – but users can request access by sending an email to the Transkribus team (email@transkribus.eu)

We are getting closer to the reliable recognition of Bentham’s handwriting and this is very exciting!  As a scholarly editing project dedicated to producing Bentham’s Collected Works, we require highly accurate transcripts as a basis for our work.  The experience of other Transkribus users suggests that transcripts which have a CER of around 5% can be corrected rapidly and easily.  So our next priority is to conduct some tests to see how easy Bentham Project researchers find it to correct and edit transcripts generated by this model where the CER is 9%.

We will also continue creating new pages of training data in Transkribus using images and transcripts of Bentham’s most difficult handwriting.  As well as retraining our current model with additional pages of data, we want to create smaller models focused on specific hands and languages in the Bentham collection.  This new training data could also be used to improve our Keyword Spotting tool, which was set up by the PRHLT research center at the Universitat Politècnica de València.

We are also preparing a large-scale experiment with Text2Img matching technology devised by the CITlab team.  This technology allows users to use existing transcripts as training data for HTR, rather than creating transcripts afresh in Transkribus.  We hope that this technology will allow us to create a new model based on several thousand pages from the Bentham collection – watch this space!

And of course, we can’t forget Transcribe Bentham.  We still plan to be able to integrate HTR technology directly into our crowdsourcing platform over the next few years.  The idea is that users will be able to check and correct automated transcripts or simply transcribe as normal and receive computer-generated suggestions of words that are difficult to decipher.  We believe that new users, who tend to be daunted by the complexity of Bentham’s handwriting, are likely to find these transcription options more attractive.  Experienced users may also appreciate word suggestions to assist their transcription work.

The Bentham Project is at the cutting-edge of this transformational technology and we hope that these advances will ultimately bring us closer to the complete transcription and publication of Bentham’s Collected Works.

My thanks go to Chris Riley, Transcription Assistant at the Bentham Project for his assistance with the preparation of training data in Transkribus.

Project Update – Searching Bentham’s manuscripts with Keyword Spotting!

uczwlse15 October 2018

The Bentham Project has been experimenting with the Handwritten Text Recognition (HTR) of Bentham’s manuscripts for the past five years, first as a partner in the tranScriptorium project and now as part of READ.

Read about our progress with HTR and the Transkribus platform in blog posts from June 2017 and  February 2018.

Keyword Spotting

Our results have thus far been impressive, especially considering the immense difficulty of Bentham’s own handwriting.  But automated transcription is not yet at a point where it is sufficiently accurate to be used by Bentham Project researchers as a basis for scholarly editing.  It would be too time-consuming (and probably too irritating!) for us to correct the errors in the computer-generated transcripts of papers written in Bentham’s hand.

However, the current state of the technology is strong enough for keyword searching!  And thanks to a collaboration with the PRHLT research center at the Universitat Politècnica de València (another partner in the READ project) we have some exciting new results to report.  It is now possible to search over 90,000 digital images of the central collections of Bentham’s manuscripts, which are held at Special Collections University College London and The British Library.

A Keyword Spotting search for the word ‘pleasure’


Appeal for volunteers!

I have prepared a Google sheet with some suggested search terms in 5 different spreadsheet tabs (Bentham’s neologisms, concepts, people, places and other).

It would be fantastic if people filled in the spreadsheet to record some of their searches, using my suggested search terms and some of their own.  Transcribers could search for subjects they are interested in and then cross-reference to material on the Transcription Desk that they might like to transcribe.

Who knows what we might find??  I hope to share some of these results in my upcoming presentation at the Transkribus User Conference in November 2018.  Thanks in advance for your participation.


The PRHLT team have processed the Bentham papers with cutting-edge HTR and probabilistic word indexing technologies.  This sophisticated form of searching is often called Keyword Spotting.  It is more powerful than a conventional full-text search because it uses statistical models trained for text recognition to search through probability values assigned to character sequences (words), considering most possible readings of each word on a page.

We delivered thousands of images and transcripts to the team in Valencia and gave them access to the data we had already used to train HTR models in Transkribus.  After cleaning our data and using Transkribus technology to divide the images into lines, the team in Valencia trained neural network algorithims to recognise and index the collection.

The result is that this vast collection of Bentham’s papers can be efficiently searched, including those papers that have not yet been transcribed!  The accuracy rates are impressive.  The spots suggest around 84-94% accuracy (6-16% Character Error Rate) when compared with manual transcriptions of Bentham’s manuscripts.  More precisely speaking, laboratory tests show that the word average search precision ranges from 79% to 94%.  This means that, out of 100 average search results, only as few as 6 may fail to actually be the words searched for. The accuracy of spotted words depends on the difficulty of Bentham’s handwriting – although it is possible to find useful results in Bentham’s scrawl!  There could be as many as 25 million words waiting to be found.

Use cases

This fantastic site will be invaluable to anyone interested in Bentham’s philosophy.  It will help Bentham Project researchers to find previously unknown references in pages that have not yet been transcribed.  It will allow researchers to quickly investigate Bentham’s concepts and correspondents.  I hope that it will also help volunteer transcribers to find interesting material.

This interface is a prototype beta version.  In the future we want to increase the power of this research tool by connecting it to other digital resources, allowing users to quickly search the manuscripts at the UCL library repository, the Bentham papers database and the Transcription Desk and linking these images to our rich existing metadata.

Similar Keyword Spotting technology (based  on research by the CITlab team at the University of Rostock, another one of the READ project partners) is currently available to all users of the Transkribus platform.  Find out more at the READ project website.

I welcome any feedback on our new search functionality at: transcribe.bentham@ucl.ac.uk

My thanks go to the PRHLT research center, the University of Innsbruck and Chris Riley, Transcription Assistant at the Bentham Project for their support and assistance.

Project Update – Bentham vs the computer

uczwlse23 February 2018

Throughout it’s long history, the Bentham Project has always been interested in the way in which technological advances could be integrated into its work on the scholarly edition of Bentham’s Collected Works.  Transcribe Bentham is currently a proud partner in an international collaboration focused on using innovative computer science techniques to process historical manuscripts.  The mission of the READ (Recognition and Enrichment of Archival Documents) project is to make archival collections more accessible through the development and dissemination of Handwritten Text Recognition (HTR) technology.

This technology is freely available through the Transkribus platform.  Using algorithms of machine learning, it is possible to teach a computer to read a particular kind of handwriting – even Bentham’s!  The technology is trained by being shown images of documents and their accurate transcriptions.  Thanks to the hard work of the Transcribe Bentham volunteers, we are lucky to have a sizeable collection of transcripts that can be used as training data for automated text recognition.

We last summarised our experiments with HTR in a blog post from June 2017.   At that point, we had used technology from the Computational Intelligence Technology Lab (CITlab) at the University of Rostock to produce a model capable of processing the easier papers from the Bentham collection, largely those written by Bentham’s secretaries.  This model can automatically produce transcripts with a Character Error Rate of between 5 and 10%, meaning that 90-95% of characters in the transcript are correct.  The Bentham model is now publicly available in Transkribus under the title ‘English Writing M1’ and has been applied to other collections of eighteenth- and nineteenth-century English handwriting with some success.

Screenshot from Transkribus with automatically generated transcript. Box Add 3350, fo. 158, The British Library (Click to enlarge image)

Although this model copes well with documents where the handwriting and layout are relatively clear, it struggles to recognise the more difficult examples of writing from Bentham’s own hand.  So we decided to take on the challenge of teaching a computer to read some of the very worst examples of Bentham’s handwriting!

We used the Transkribus platform to create training data based on Boxes 30 and 31 of the Bentham Papers held in UCL Special Collections.  These manuscripts were part of Bentham’s lifelong obsession with critiquing the work of William Blackstone, the English jurist who was most famous for his Commentaries on the Laws of England (1765-9).  To create the training data, we uploaded around 200 digital images to Transkribus, segmented each image into lines and then copied over existing transcripts to match each image.

The resulting model generates transcripts with an average Character Error Rate of 26%.  This error rate is unfortunately too high to automatically produce transcripts suitable for scholarly editing.  Nevertheless, it does have the potential to facilitate the full-text search of the Bentham Papers.  Transkribus now includes sophisticated Keyword Spotting technology, which is capable of finding words and phrases in documents, even if they have been mistranscribed by the computer.

Screenshot from Transkribus with automatically generated transcript.  Box 31, fol. 78, UCL Bentham Papers, Special Collections, University College London (Click to enlarge image)

We are working with the Pattern Recognition and Human Language Technology (PRHLT) research centre at the Universitat Politècnica de València and the Digitisation and Digital Preservation group at the University of Innsbruck to present an open-access search functionality for the Bentham Papers.  We are also hoping that volunteers could get involved in this endeavour by checking and correcting the results of significant search queries to ensure their accuracy.

Improving the recognition of Bentham’s handwriting is our other aim and to this end, we will be producing more pages of training data in Transkribus.  The technology moves so fast that the efficiency of this process has already been streamlined thanks to technology from CITlab.  Transkribus can now easily find lines in images (even in documents with complex layouts) and it is also possible to use existing transcripts to automatically train a model, rather than copying them into Transkribus line by line.

If you are interested in following in our footsteps, you are welcome to give Transkribus a try!  You can find more information on the READ website and in the Transkribus How to Guides.

I would like to thank Chris Riley, PhD student and transcription assistant at the Bentham Project, for helping to produce the training data for the latest Bentham model.