ARCRE British & Commonwealth War Diary Search Engine - no longer online

Oh yeah, I meant to mention that as well. None of the navigation in the menu bar is working just yet. Basically I haven't gotten around to creating the pages. I've been concentrating on the fun stuff of programming the search engine and database.

Next job is to get those pages completed and then on with the programming of the content management system, user authenication and some bells and whistles for the search engine like saving a search, sending an advanced document order directly to the National Archives, requesting documents to be copied, etc.

There are some other useful search engines in the pipeline too, if this one proves a success.

No problems with that, makes complete sense. I am looking forward to the updates.

 

No problems with that, makes complete sense. I am looking forward to the updates.

Mark, it's fair to say the whole website is in the Beta stage at the moment.

But if you think of something you'd like to see incorporated let me know.

There will also be a document library which will include transcripts of interesting documents. (The first ones to go on there will be a report on Rudolf Hess' arrival in Britain and parts of an official ULTRA-classified report on Operation Mincemeat - both of which I think I've posted either here or on psywar.org in the past. That kind of thing.)


Rich, I've fixed the issue with undated WWI diaries popping up in a WWII all theatres search. Thanks for pointing it out to me.

I haven't decided what to do about the ordinal numbers just yet. One thing I'm thinking about is to actually modify the catalogue data by removing ordinal numbers, e.g. turn 2nd into just 2. I might modify the abbreviations as well by changing Coy. into Company, etc. Or perhaps leave both versions in the data like "Coy. (Company)".

I'll probably put in an option for "find all words" or "find any of these words". Although I want to keep the search engine as simple as possible.

 

I haven't decided what to do about the ordinal numbers just yet. One thing I'm thinking about is to actually modify the catalogue data by removing ordinal numbers, e.g. turn 2nd into just 2. I might modify the abbreviations as well by changing Coy. into Company, etc. Or perhaps leave both versions in the data like "Coy. (Company)".

You've noticed. Never any consistency in data! For some of the CWGC fields, I stripped out such things as stray dots and there were quite a few double spaces - which made a string comparison fail. There are still variations in unit descriptions, but far too many records for me to correct.

I'll probably put in an option for "find all words" or "find any of these words". Although I want to keep the search engine as simple as possible.
The AND function on two or more enteries would be quite useful. If there are many false hits (not tried it enough to know yet) you could add an Excluding field. But I understand the problem, keeping it simple reduces user errors.

 

Especially for Rich, numbers will now be automatically searched also using their ordinal suffix.

Therefore a search for "2 Armoured Division" will also find records that include "2nd Armoured Division".

Searching for "2nd Armoured Division", however, will not find plain "2 Armoured Division". So best to leave out the ordinal suffix in a search unless especially needed.

 

You've noticed. Never any consistency in data! For some of the CWGC fields, I stripped out such things as stray dots and there were quite a few double spaces - which made a string comparison fail. There are still variations in unit descriptions, but far too many records for me to correct.

The TNA data is so inconsistent in the way it was originally catalogued. I'm sure it can be clean up quite a lot. The major problem is the use of abbreviations at the piece level, like "225 Fld. Coy." or sometime just "225 Coy." for the diaries for 225 Field Company, Royal Engineers.

The AND function on two or more enteries would be quite useful. If there are many false hits (not tried it enough to know yet) you could add an Excluding field. But I understand the problem, keeping it simple reduces user errors.

Good ideas Geoff, I will almost certainly include some AND, OR, MINUS functions. Although the official TNA catalogue has have some great advanced searching features if you know how to use them.

 

Just as test here's a search comparison between my search engine and the official TNA catalogue...

ARCRE war diary search engine search for 2 Armoured Division (27 results):
WO 166/805 DIVISIONS: 1ST. ARMOURED DIVISION: 2 Tank Transport Company. 1941 Aug.
WO 166/814 DIVISIONS: 2ND. ARMOURED DIVISION: General Staff (GS). 1940 June-Oct.
WO 166/815 DIVISIONS: 2ND. ARMOURED DIVISION: Adjutant and Quartermaster (AQ). 1939 Dec.- 1940 Oct.
WO 166/816 DIVISIONS: 2ND. ARMOURED DIVISION: Support Group. 1940 Feb.-Oct.
WO 166/817 DIVISIONS: 2ND. ARMOURED DIVISION: Commander Royal Engineers (CRE). 1940 Jan., June-Oct.
WO 166/818 DIVISIONS: 2ND. ARMOURED DIVISION: Signals. 1939 Sept.- 1940 Oct.
WO 166/819 DIVISIONS: 2ND. ARMOURED DIVISION: Support Group Field Park. 1940 Nov.
WO 166/820 DIVISIONS: 2ND. ARMOURED DIVISION: Postal Unit. 1940 July-Dec.
WO 169/86 2 Armoured Division: General Staff (GS) 1940 Nov.- Dec.
WO 169/87 2 Armoured Division: Adjutant and Quartermaster (AQ) 1940 Nov.- Dec.
WO 169/88 2 Armoured Division: Commander Royal Engineers (CRE) 1940 Nov.- Dec.
WO 169/89 2 Armoured Division: Signal Company (Sigs) 1940 Sept.- Dec.
WO 169/90 2 Armoured Division: 4 Squadron Signals 1940 Sept., Nov.- Dec.
WO 169/91 2 Armoured Division: Headquarters Royal Army Service Corps (HQ RASC) 1940 Dec.
WO 169/92 2 Armoured Division: 2 Support Group Company Royal Army Service Corps (CRASC) 1940 Oct.- Dec.
WO 169/93 2 Armoured Division: 3 Armoured Brigade Company Royal Army Service Corps (CRASC) 1940 Nov.- Dec.
WO 169/94 2 Armoured Division: 1 Armoured Brigade Ordnance Field Park Section (OFP) 1940 Sept.- Dec.
WO 169/95 2 Armoured Division: Headquarters Support Group 1940 Nov.- Dec.
WO 169/108 7 Armoured Division: 2 Squadron Signals (Sigs) 1940 June
WO 169/1159 2 Sp Gp HQ 1941 Jan - May
WO 169/1160 2 Sp. Gp. R.A.S.C. 1941 Jan.- Feb.
WO 169/1161 2 Sp. Gp. Ord. Fd. Pk. 1941 Jan.- Mar., June- Dec.
WO 169/1162 2 Sp. Gp. Lt Repair Sec. 1941 Jan.- Feb.
WO 169/1163 2 Sp. Gp. Rec. and Lt. Repair Sec. 1941 Jan.- June
WO 169/4073 2nd Sp. Gp. Ord. Fd. Pk. 1942 Jan.- Mar.
WO 169/8684 1 and 2 sec. Tps. Sub. Pk. 1943 Jan.-June
WO 170/7669 Detachment 26 British Liaison Units Attached 2 Polish Armoured Division 1946 Jan.- June

Official TNA Catalogue search for 2 Armoured Division (10 results):
WO 169/86 2 Armoured Division: General Staff (GS) 1940 Nov.- Dec.
WO 169/87 2 Armoured Division: Adjutant and Quartermaster (AQ) 1940 Nov.- Dec.
WO 169/88 2 Armoured Division: Commander Royal Engineers (CRE) 1940 Nov.- Dec.
WO 169/89 2 Armoured Division: Signal Company (Sigs) 1940 Sept.- Dec.
WO 169/90 2 Armoured Division: 4 Squadron Signals 1940 Sept., Nov.- Dec.
WO 169/91 2 Armoured Division: Headquarters Royal Army Service Corps (HQ RASC) 1940 Dec.
WO 169/92 2 Armoured Division: 2 Support Group Company Royal Army Service Corps (CRASC) 1940 Oct.- Dec.
WO 169/93 2 Armoured Division: 3 Armoured Brigade Company Royal Army Service Corps (CRASC) 1940 Nov.- Dec.
WO 169/94 2 Armoured Division: 1 Armoured Brigade Ordnance Field Park Section (OFP) 1940 Sept.- Dec.
WO 169/95 2 Armoured Division: Headquarters Support Group 1940 Nov.- Dec.

 

Do we get multi-colored anoraks?

 

Although the official TNA catalogue has have some great advanced searching features if you know how to use them.

I agree.

Often amused by search engine inputs, some folk seem to think search engines are super-intelligent. For example entering:

DATE + REGIMENT

does not find what they are looking for, so they add:

DATE + REGIMENT + PLACENAME

Not gonna work, is it?

 

Another small modification, by default any words with four on more letters are treated as word stems, e.g. search term "engin" will return "engine", "engineer", "engineers", etc.

 

You can never have too many ' clever buggers'.

Quite right Geoff, because you'll never outnumber us village idiots !

Thanks for the adjustments Lee. I have to confess that I didn't see the 'Theatre' box with 'ALL' appear after I'd entered 'War' Doh !

Fantastic now. All the Div HQ diaries that a chap could possibly wish for !B)

 

I agree.

Often amused by search engine inputs, some folk seem to think search engines are super-intelligent. For example entering:

DATE + REGIMENT

does not find what they are looking for, so they add:

DATE + REGIMENT + PLACENAME

Not gonna work, is it?

Exactly Geoff. Less is more when it comes to search results.


I've now added a "find all words" and "find any words" option. Not sure if that is a big bonus or not.

 

Lee, I've worked out what has caused my over-sized results. It seems that the 'Theatre' box does not reappear after using the back-button and entering 'WWII'. I hadn't realised that I was missing it.

 

Lee

Thank you, very useful.

I've just found an file that I knew existed but could never get TNA's search engine to list.

Nick

 

Rich, the back button will mess up the Javascript on the theatre field. Although there shouldn't be a need to back-up as a new search form loads at the bottom of the search results.

I was thinking of populating the form with the last search details. Would that be better than a blank form?

(If you have backed-up then a workaround is either refreshing the page or moving the war setting from WWII to something else and then back to WWII.)

Nick, that's great, just what I was hoping to hear!

 

Lee, I've worked out what has caused my over-sized results. It seems that the 'Theatre' box does not reappear after using the back-button and entering 'WWII'. I hadn't realised that I was missing it.

Server side programming really is a pain in the @ss. Stuff works much better on a desktop. Looks like we're gonna be stuck with cloud computing soon. I wonder when the last PC will be sold?

 

I've just found an file that I knew existed but could never get TNA's search engine to list.


Excellent!

 

Lee, I probably over-use the back button. It's certainly not insurmountable now that I know what's happening. The last search would suit me as I'm usually looking for the same period at least.

It is a most-useful resource and begs the question why, when half the NA tables are taken up by people with war diaries, the NA IT pros haven't done this themselves (but then I thought that about CWGC too).

Keep up the good work chaps.

 

Rich, I can probably fix the issue with the back button and disappearing theatres option. I'll look into it.

Good question why TNA haven't done something similar themselves. I know they are working on a new, improved search engine but from what of it I've seen, war diary searches will still be problematical. Basically that is a result of way the diaries have been catalogued by using the same hierarchical model of the original paper-based catalogue. I've tried to overcome that problem to a limited extent but ultimately I will need to actually update the catalogue data to something more useful.

 

A quick upadate, the Royal Marine Commando diaries are now included in the search engine. There should be a complete dataset now for all WWII diaries.

There you go Andy, can bookmark it now ;-)


Lee