... and how this affects researchers and volunteers

The changes we're making at the moment will take FreeCEN into the future. We are no longer a lookup service, which is how we started. We are becoming a dynamic source of data for both the amateur and the professional. 

In this post we will explain the issues driving the changes, and how they will affect you. It's broken down into the following sections:

Changes for Researchers

Changes for Volunteers

• Transcribers
• Proofreaders(formerly 'Checkers')
• Validators
• Coordinators

Future Developments (for Researchers)

Changes for Researchers

A growing number of users are accessing the web through phones and tablets, rather than laptops and larger computers - they expect an interface that looks good and works well on a small screen. We think we are ‘almost there’ on this.

We also want to make the website more accessible. This work is just beginning.

We want to tackle some issues in searching, including:

• Place: The boundaries used by the census are not easy to visualise, often cross historic boundaries (e.g. including parts of two counties) and change name and shape - so even if you know the village or part of a town your ancestor might have resided in, choosing which census district to filter by is not easy, particularly for those unfamiliar with the geography of the 19th century.
• Names: The names used by people to record their place of birth are faithfully transcribed. But this also makes searching difficult, plus the person entering the data might know that their home village was in one county when they were born, and another at the time of the census, and chosen either. So we are looking into more choice of how you search, including a map-based search.
• Too much choice: It can be difficult to know which of the many fields you should fill in, in order to be able to identify the person you are seeking, but not rule them out due to something not being recorded quite as you might have expected (we are looking into Artificial Intelligence to solve this one for you).

We also want to improve the quality of the data we transcribe - in the past, we have had to use abbreviations and compress two fields into one, in order to keep within the spreadsheet restrictions of 20 years ago. We're moving to a flexible spreadsheet (CSV) system which will mean transcribers can type what they see everywhere.

Changes for Volunteers

Volunteers may think "I am happy with what I am doing now." That's fine, you can keep doing what you are doing and we will continue to appreciate your work. It is very valuable. Alternatively, you can come along for the ride. You will find it is going to be a very exciting one!

Transcribers

You will have an option of either transcribing onto a spreadsheet as you have been doing, or transcribing online. The spreadsheet will look and feel very similar to the one you are using now, but there will be some changes.

Columns in the spreadsheet become fields in the FreeCEN database. As the fields in the database have become more flexible we can be more flexible in what you enter on your spreadsheet. For instance, transcribers put a lot of effort into making the Occupation fit the Occupation column. That will no longer be the case. You will be able to enter what you see without worrying about the length and some columns can be amended. Column H in the existing spreadsheet, for instance, contains details of unoccupied houses, people visiting away, buildings in progress etc. but it is also used for the query flag. We are looking at moving the query flag into its own column. 

In the online version, you will be entering data directly into a database. You will, therefore, see field boxes on your screen instead of columns. Instead of your coordinator sending you images, via Dropbox, for instance, the image will be displayed on your screen. When you have transcribed all the rows on an image you then move on to the next image. Each field in the online database will exactly correspond to a column in the spreadsheet.

The online version will also capture the number attached to the image. This will allow FreeCEN the ability to link the image to the transcription and display the images on-screen for the researcher, much the same as commercial sites do now. We still need to work out how we are going to add this information to the almost 36 million records already online (as of November 2018) so the displaying of images is something for the future. Nevertheless, we can start to collect that information with the online version.

One difference between the spreadsheet and the online database is that a transcriber can stop and start wherever they wish. The online version is displayed one page at a time so transcription has to stop at the end of each page, and not in the middle of a page. Another difference is that transcribers can share a census piece. The next available page is displayed to each transcriber (not the same page). Once a page is transcribed it will not be displayed to anybody else.

So, whether you choose to keep transcribing on a spreadsheet or to transcribe online, the changes should make transcription easier.

When will all this happen?

The first test piece was transcribed in August 2018 and has been proofread and validated. We are creating a detailed audit trail for this piece so that the developers can refine the system.

There is a test piece available online if you wish to have a go. This is an 1871 Somerset piece that has already been published online. It has been made available for multiple people to transcribe, a page at a time. Once it has been completed we will perform a quality check to compare the shared transcription with the published version. We would like to know whether our quality drops using this method. You do not need to be a Somerset volunteer to work on this; anybody can give it a go. If you do give it a go, please give us feedback. That way we can continue to improve. 

You can access it at https://csindexing.com/projectinformation.php?p=358 (link opens in a new tab). This version is a prototype version and therefore is not very pretty. Nevertheless, we do need to know of any problems transcribers may experience with it.

Proofreaders

You will be using new software for the proofreading process. (No, we no longer have checkers - the transcribers' work is being proofread. Transcribers are not being checked on!). 

Some proofreaders will remember the old WinCC software, used before we moved to proofreading spreadsheets. WinCC became incompatible with modern operating systems as it is badly in need of an upgrade. It certainly is not compatible with the new database format. 

This upgrade will happen in conjunction with the introduction of the online transcription system. Once the transcription trial is finalised we will know the design of the proofreading software. We will also be able to make the decision whether this will be an online or offline process, or whether the proofreader will have the choice. 

The trial transcription has been converted to a spreadsheet and the Proofreader is using a spreadsheet to proofread.

Validators

You will definitely be using new software. Neither FCTools nor ValdRev is compatible with the new database format. The validation process will not vary a great deal. The biggest change is that a map reference will be automatically attached to every Place of Birth. This will affect the way that the Validator manages the Validator’s Choice Place of Birth where the original entry cannot be found. The development of the map location system is in progress (as of November 2018). Once again we are not sure whether Validation will be online or offline, or whether there will be a choice.

Coordinators

Coordinators have already seen changes in registration of new volunteers. This will continue but you will have two methods of issuing transcription work: by sending images as you do at the moment, and by giving access to online images. The Coordinator will upload the online images to be transcribed from their master copies. The online software will record who has worked on these images, so the Coordinator will only need to keep track of who is working on the spreadsheet images. 

Coordinators will also be able to upload their own completed data to the database. You used to be able to do this but during the changeover to FreeCEN2 this function was centralised to give more control. Uploading will be devolved back to the Coordinators in the foreseeable future.

Future developments (for Researchers)

We're about to give each record (household) a permanent URL. To do this, we recently made the “FreeCEN2 website” available at www.freecen.og.uk, and the existing website freecen1.freecen.org.uk. The old FreeCEN will be phased out in the future. We intend to develop new ways to search the data. 

The addition of map references is one way of doing this, for example. Instead of having to know the county border in the year being researched, (or in the birth year of the person being looked for) a place will be able to be chosen. This place has a map reference. The search can then be done around that map reference. This would be transparent to the end user. 

An example would be a researcher in the future looking for me. I was born in Bristol, census county GLS. However, I was brought up 7 miles away in Somerset (Census county SOM). Whenever I fill in a form I show my Place of Birth as Bristol (that was where the hospital was!). A future researcher would be frustrated searching the Bristol 1951 Census. I am not there. I am in a different county. However, a search of 10 miles around the map reference would discover my entry. A much better result for the researcher and for FreeCEN.

We tend to think of a researcher as somebody building their family tree. However, there are other types of researchers out there. A university student doing a PhD may need to identify the demographics of a specific geographical area in a specific timeframe. Because FreeCEN is an Open Data platform we can provide a data dump of that geographic area for the student to use. Once again this is a win-win for both the student and FreeCEN.