WO361 - Missing Personnel Files - Database Discussion

Discussion in 'Prisoners of War' started by Drew5233, Sep 30, 2014.

  1. dbf

    dbf Moderatrix MOD


    Some of the Missing Personnel files contain more names of witnesses and what I term 'incidentals', than actual CWGC/POW cases...

    There are correspondence with the wounded, the discharged, the still serving but transferred, report forms by Coy Commanders which list names of those last in contact/on patrol/in Section, as well as a good few Acquittance rolls sometimes for whole Company ... and so it goes on.

    And then there are the files which interested me the most, which contain very little other than lists, re-done lists, up-dated lists, all with the same poor souls, mostly "Category C".
     
  2. bamboo43

    bamboo43 Very Senior Member

    It's a cunning plan, a plan as cunning as a fox who's just been appointed Professor of Cunning at Oxford University.

    Fair enough then Capn' Black Adder, by the way, its rat au van for dinner.
     
  3. ClankyPencil

    ClankyPencil Senior Member

    http://ww2talk.com/forums/topic/54959-ww2-commonwealth-army-war-diaries/

    Further to the posts in the above thread regarding a WO361 database.

    I had a good think about a WO361 Database last night while looking at the 4 walls of my digs (not really i was in the pub next door watching the football).

    This would a huge undertaking taking years (and then maybe only scratch the surface)

    but i came to the view that if your told you need to move a mountain, then there is not much point standing there gawping at it saying it can't be done, you need to start moving the first stones and see where it leads.

    Baring a lot of typing (and time), i see it as a win-win situation for myself and others. I get to read through a lot of files and othersget a searchable database which may help them further their research.

    Therefore i don't see any reason not to start a database, but would be happier if a few ground rules were sorted and a few concerns addressed before i embark down the road.

    Here are a few random thoughts/concerns/preferences of mine regarding a database:-

    Legacy & Ownership & Access

    I would like any database to be 'freely' available and accessible to all.

    Depending on if the database is posted on this site or Lee hosts it on his site, what would happen to it if say this site folded (i know its highly unlikely) or Lee's numbers come up on the Lottery and he retires off into the Sunset? It would be nice to have agreement as to what would happen to the database or who would become its custodian.

    How should i treat any files that are passed to me with regard to divulging information within them? i.e. If someone posts a query up on here about a missing soldier do i just say he is mentioned in the file or could i post relevant pages from the files. If i did this then potentially i could be diddling Lee and Andy out of work.
    If this database grows, human nature being what it is (this is not directed to anyone on here by the way), someone somewhere will see the potential to earn a quick buck off the back of it. I won't be doing the database with any view for any future financial gain, but IF a potential monetary gain rears its head in the future then i would be happier if this were to happen there was an agreement now that any monies would go to a charity like the RBL.

    Format of the database and information covered etc
    will only the names of those on the 'missing' rolls be covered? or as Diane refers to them, will it cover the 'incidentals' like names of those that gave testimony or are just referenced within?
    Personally i would prefer to include the 'incidentals', but this would be time consuming.
    Therefore it may be an idea to just do the rolls first to build up the database quickly to a reasonable size,
    then do a second pass to pick up the 'incidentals' at a later date.

    Which files to cover/ priority list
    Might not be an issue, as it will likely be governed by what files i get sent. Most of the files i've already got are for North Africa & Italy and units specific to my own research.
    But its obvious the files likely of most interest will be from the NWE campaign and the more 'fashionable' units.
    Personally i don't mind which campaigns or unit files i do, but wouldn't want to concentrate on just one campaign or a narrow scope of units.
    My preference would be cover all areas/units proportionally something like as follows:-
    For every 3 NWE ones, i do maybe 2 BEF's, and 1 each from remaining campaign areas (Italy, N. Africa, Far East etc) and something similar with regard to which units i.e. for every 4-5 more well known units i do an obscure of lesser known unit.

    Just a little food for thought for the mo, but would welcome any further suggestions/ concerns

    Will see where the discussion goes and will try to add to it later


    Cheers

    Scott
     
    brithm and bamboo43 like this.
  4. bamboo43

    bamboo43 Very Senior Member

    Hi Scott,

    First and most of all, I congratulate you for taking on this project. I have collated two databases over the last seven years and I know just how much work this entails and the stamina required.

    Ownership and protection of the database is a difficult one. I agree totally with you that it must be freely available to those who need to access it. I had a decision to make back in 2011 when I was deciding whether to write a book about Chindit 1 or create a website. Obviously I chose option two, with the view that although the information posted on the site is vunerable to 'theft' or misuse by others, there was more scope for the families I wanted to attract to find my work. This decision has proved overwhelmingly to be the correct one for my objectives.

    I know that my work has been used 'unoffically' let's say by the medal dealing fraternity. This is not something I am happy with, but comes with the territory of not being able to protect or copyright data on line. It is the price that you have to pay every now and then.

    I think you are right to ask the question about posting actual casualty details or pages on the forum. Some families are so delighted to find out information they are not concerned by this, but I have known some to be most put out by this action too. I think you are right to consider our members who perform a copying/research service for others and also that, some people will want to be pointed in the right direction, but would like the thrill of finding and reading the information for themselves.

    I think it is as well to add all names invovled in the material, whether they are the casualty or the informant or witness. Countless times in my research and I'm sure in yours, small connections occur from the work of just one witness and a good overall picture can emerge when you cross-reference names, dates and places.

    I wouldn't worry about which theatres to concentrate on. As you say this database will build with the input of others and they will send what they have to hand.

    I'm also thinking that the WO361 File Reference Excel that Andy posted on to the forum a couple of years back would be a usefull addition at the begining of this thread as a reference tool.

    Lastly, I will be glad to help if I can with some WO361 rolls from the Burma theatre. I'm positive others will too, so be prepared! :salut:

    Best wishes

    Steve
     
    ClankyPencil and brithm like this.
  5. von Poop

    von Poop Adaministrator Admin

    There is a database facility built into the forum software.
    It's theoretically quite sophisticated, but attempting to set something genuinely complex with lots of sortable fields up with D highlighted an issue with searches on it that might knock it out of contention as a useful tool.
    With that caveat, I'd be happy to knock up a trial setup so people could have a look. It can be wiki style open edit, or restricted to a select few editors, with all amendments/changes stored.

    Another thing I'd have to check is how easy the data is to export from ipb databases. With something big it'd be unwise to put all the eggs in one basket, and I obviously can't 100% guarantee the really long-term survivability of a forum.
    Though I think it exports to spreadsheet files like most DBs.
    (Sorry, just reread your post properly on this 'legacy' aspect - I agree; really important if embarking on a big project. Very little on the internet is carved in stone.)

    Anyway, just a thought. Do not let me tread on any other plans that may be in process - throwing the offer out there just in case.
     
    brithm likes this.
  6. ClankyPencil

    ClankyPencil Senior Member

    Thanks Adam & Steve.

    With regard to the database i don't think believe the data mining/entry side or proof reading etc would cause any real problem for me.

    But the main area of concern would be the initial set up. My excel skills are 'passable', so I would gladly welcome any help available regarding initial formatting and set -up etc.

    Once an agreed 'template' was established i can then pootle along to hearts content.

    Apologies for only a brief reply but everytime i've go to compile a detailed reply i keep getting 'sidetracked' with work.

    Cheers
     
    brithm likes this.
  7. dbf

    dbf Moderatrix MOD

    I agree working out the fields at the start is important... a lot more work to fix afterwards.

    When working out what fields to use, apart from the obvious (name, army no.) you would also have to consider what combos people will be searching with (surname/unit, surname/first name or initials, surname/army no.).
    Then with possible the issues regarding privacy/sensitivity to details given/a desire to discover for oneself, there is the option of just giving the file ref
    or file ref with a page no. would have to be added,
    or image no. (easier option than former)

    Theatre for those who want to narrow down information, but the TNA ref. already contains this info
    Regi-mentalists might want to search mainly by either unit or by bn.
    Do you want to merely list all the men and give them a sep field for reason listed, i.e. casualty, witness, 'incidental' (i.e. from a roll)

    first name / initials
    surname
    army no
    unit (are sep fields really needed for bn, coy, sqn, pl? necessary, or can extra info be placed in a Notes field)
    regt/corps
    Casualty status, if applicable
    Details (if desired, i.e. type of info, whether the person was witness, on Acquittance roll etc, or was POW/CWGC casualty)

    TNA Ref
    page no./or image no. if keeping these archived somewhere
    Source type (report, narrative account, radio, etc )

    Theatre
    Witness (if any)
    Notes (a general field for anything else not covered and not worth having a field for)


    putting combos of unit/reg makes the list of units longer,
    1st Irish Guards
    2nd Irish Guards
    3rd Irish Guards

    adding a sep. field for e.g. Bn will make the spreadsheet more complex


    Checking out what others have included might help...
    http://ww2talk.com/forums/topic/46144-1st-battalion-irish-guards-italy-missing-personnel-file/
    http://ww2talk.com/forums/topic/30283-1st-battalion-irish-guards-north-africa-missing-personnel-file/
    http://ww2talk.com/forums/topic/27626-grenadier-guards-missing-personnel-bef-1940/
    http://ww2talk.com/forums/topic/46426-scots-guards-north-west-europe-missing-personnel-file/
    http://ww2talk.com/forums/topic/46424-scots-guards-north-africa-missing-personnel-file/
    http://ww2talk.com/forums/topic/46425-scots-guards-italy-missing-personnel-file/


    sorry that's all a bit scattered, but having connection problems.
     

    Attached Files:

    ClankyPencil likes this.
  8. von Poop

    von Poop Adaministrator Admin

    There doesn't appear to be a native way of exporting databases from the forum to excel or similar.
    Seems entirely possible with a custom hook or mysql access, but beyond my permissions. So without that safety net, I fear we're probably not the best place to help.
    Sorry about that.
     
    brithm likes this.
  9. papiermache

    papiermache Well-Known Member

    An elaborate index used to be called a calendar at the Essex Record Office. The Imperial War Museum publishes useful summaries of some books and papers it holds: I would call these long lists of subject headings a calendar.

    On a computer a search for a name would find the name in a calendar. If I were doing a calendar I would use "Text Edit" because I use a Mac but some might prefer the Microsoft version ( is it Notepad ? ) or Word. I actually prefer Word Perfect for Windows on my 1992 laptop, 96 Kb basic RAM, which still works. Wonderfully clear and fantastic search facility because no box to open. Word Perfect converts to Text Edit with a few extraneous lines.

    The term "database" seems to imply a spreadsheet, which probably means a Microsoft program called Excel, or it could be the Mac version called "Numbers", which program seems to involve four times as many bytes as Excel. I use Excel for Mac, but I have two versions, 2007 and 2011. They seem to work together, but for how long ?

    I don't like computer spreadsheets but I do use them. However, for a long list of casualties I prefer pieces of A4 paper stuck side by side and I can add dates when I added information much more easily than in a spreadsheet. Note that in an Army service record the date when information was added is extremely useful.

    As for The National Archives Copyright Policy, Takedown and Reclosure Policy, Privacy Policy, Record Copying, and User Participation I could make my way through it given several days but the conclusion I would reach is that as long as you say that the information came from Kew and it is Crown Copyright then....( please read the babble produced by Kew, it's all rather circular.)

    The best of luck to anybody taking on a new spreadsheet.

    That's my five pennies worth.
     
  10. bamboo43

    bamboo43 Very Senior Member

    I use Excel 2011 on my Mac too. I find it very easy to use and to sort data quickly and precisely. I'm sure almost all of us have used this system or something similar when decanting paper lists or other data.

    Here is one of my smaller examples just to illustrate the point:

    13KLR Officers.jpg
     
  11. dbf

    dbf Moderatrix MOD

    Import/upload from spreadsheet would've been more useful in my case. Work for months on a spreadsheet, then have to enter each and every bit of data, all over again...



    Mac Numbers allows export as Excel, pdf and as CSV, which is what i did with the test template in previous post.

    I love spreadsheets, which my other half scoffs at, he being an Oracle DBA.
     
  12. papiermache

    papiermache Well-Known Member

    Thanks, Steve, everybody I know say that spreadsheets and filters are good.

    But just think, press F2, type in a few characters, and if it's there the cursor goes straight there. Magic !

    John
     
  13. papiermache

    papiermache Well-Known Member

    Dear dbf,

    I agree with the Oracle, ( oh for the return of Ceefax and Oracle ! ) but the screenshot will please Mrs. papiermache, who will shout QED !

    John
     
  14. ClankyPencil

    ClankyPencil Senior Member

    I've had another think and took on board various suggestions/ideas posted etc and this is my current way of thinking.

    I think the 2 main functions of a database should be geared at:-

    1. Helping the 'Regimentalist' type researcher compile nominal rolls of battalions etc
    2. Provide people researching individual soldiers or relatives with possible further avenues of research

    I've also decided that i personally should not divulge any detailed information contained in the files or add any to a database. But, think it maybe
    beneficial to give an indication of the type of information contained (i.e. Missing Personnel, provided testimony, listed as potential witness, Incidental reference).
    I also think the format, number of fields, information contained should tried to be kept to a minimum

    My reasoning behind this is as follows:-
    1. Saves time
    2. I don't get judged, reprimanded or upset people by divulging what could be sensitive or unpleasant info (The decision to pursue the info will still lie with the individual).
    3. I am not potentially diddling people like Andy or Lee out of possible future work research. (and it may even put more work their way).
    4. With regard to the 'indication of type of information' it may help people make an informed decision whether to acquire the file or not (either by a trip to Kew or through someones
    services)
    5. Researchers who already have WO361 files might be happy with names, basic info being added to a DB, but not full details of files that they either spent time or money to acquire. Therefore
    they might be more inclined to send me what they have.
    6. Still gives the researcher that 'element' of discovery.
    7. By keeping required info to a minimum, hopefully more people might feel more inclined to chip in and help along the way.
     
  15. ClankyPencil

    ClankyPencil Senior Member

    The Actual Database

    Main DB pages
    Format & information fields

    Based on Diane's previous list i see the minimum fields required for the DB as follows:-

    1. Surname
    2. First Names, Initals etc
    3. Army No.
    4. Rank
    5. Primary Regiment
    6. Battalion or Unit

    All the above as self explanatory
    7. Secondary unit
    My reasoning for this is that i've come across a fair amount of correspondance in the files, between various Army departments, trying to trace the whereabouts of men that
    have later been transferred to other units. If this info is available it may help in compiling unit rolls etc.
    8. Campaign (if required)
    9. WO Ref (i.e. WO 361/809 and only this; no page ref's or image no's etc).
    In essence, all i think is required is to point a researcher in the right direction, and will save time.
    10. Remarks
    Generally for highlighting things like inconsistancies (i.e. the same man being referenced with slightly different Army No.s etc)

    Front page

    Basically a front page listing WO361 files covered by the DB listing following info (which will also help track where i am up to)

    1. Wo Ref
    2. Regiment(s), units
    3. Which battalions/sub-units it covers
    4. Extent of Info covered
    (i.e. All names including incidentals, Just 'missing' roll, file received but not yet transcribed etc)
    5. Date added or updated
    6. Who or where file originated.

    Basically so people can gain accreditation for supplying the file. But, there could be a potential negative side to this, with people constantly getting pm'ed etc for more
    information etc. Therefore anyone supplying a file would need to specify if they would require the accreditation or would require their name details witheld.

    Producing the DB

    Obvious choice is as an excel file xls or similar, mainly because its what i am most comfortable using, and also it can exported/imported/manipulated easily.

    Basically, i am hoping to concentrate on the raw data side or things, and leave the technical side of things to the more tech/database savvy people. Therefore if anyone wants
    to look into things like search forms, data manipulation etc then be my guest.


    So.... for the moment thats my twopenneth worth.

    If anyone has further suggestions, ideas or areas of concern then let me know.

    Also, i wouldn't mind a bit more help, ideas and guidance to the 'Ownership & Protection' side of things as Steve previously mentioned.

    And finally if anyone fancies knocking up a basic excel template based on the above then then feel free

    Cheers
     
  16. dbf

    dbf Moderatrix MOD

    Scott
    One for you to consider.

    I've added a second template sheet which has a list of ranks and of units immediately after the heading.

    These can be deleted after completing the sheet, and will allow standardisation of terms in the rows completed below.

    Happy for you to change or disregard altogether.

    View attachment WO 361.xls
     
    ClankyPencil likes this.
  17. jacksun

    jacksun Senior Member

    Hi everyone, my first questions would be from a long term and volume perspective. Excel spreadsheets can rapidly become bogged down during searches when their data limits are pushed. If it is just a flat spreadsheet - no references, links to other sheets with more detail etc it may support a great deal of data, but it also may not be the best solution for long term use, growth, and sharing.

    One should seriously look at developing a true database backend, with a web based front end. This would allow significant growth, ability to search effectively, capability to add different levels of access (basic info for some, more detailed for others, full details for maintainers etc) using security models, as well as a standardized interface.
    This would also take the load off the local machine, where search time and capability is restricted by local resources and place it on a server where more resources are available. It would also allow for better protection of the data through redundancy and backups provided at the server level. Access to multiple users, admins, and maintainers could also be provided simply. Images could be locally stored and linked from the individual records.

    With a web based front end a publicly available search function could also be provided, themed to WW2Talk, and linked from the forum.

    I would take a bit more to design and setup - but well worth it in the long run IMHO.

    As I've previously said, I can donate a domain name and the necessary hosting for the site as well as daily backup. I would also be willing to be involved in the design of the back and front end.

    Wayne
     
    dbf and ClankyPencil like this.
  18. ClankyPencil

    ClankyPencil Senior Member

    Diane

    Thanks for the xls

    Wayne

    The one glaring weakness for the producing a database would be my lack of knowledge of the technical side regarding databases, interfaces, hosting, search facilities etc. Therefore i will take any help i can get.

    If you are prepared to be involved with the front & back end design, hosting etc then it would be greatly appreciated.

    Cheers
     
  19. bamboo43

    bamboo43 Very Senior Member

    Scott,

    Been looking through my files to see what I have. More than I first thought really. 20 sets of MIA Rolls from various WO361 files, mostly Burma. Some typed, some in good readable order and some handwritten and not so easily read. I'm sure you have seen a fair amount of examples yourself.

    A question for you. Would you be interested in men who after being classified as MIA, were later to be found as POW's. I ask this because of my work on the prisoners of Rangoon Jail during WW2. Here I can decant the info from my own Excel and therefore data may be able to be copied in very straightforwardly.

    Steve
     
  20. ClankyPencil

    ClankyPencil Senior Member

    Steve

    I have been looking through my own WO 361 files and there are loads of men posted MIA that turn up as POW's, along with correspondance from various Army departments, Red Cross etc giving camp names or details.

    My view is if a researcher gets pointed in the right direction and finds these additional details it will help fill in the gaps and give additional avenues of research. Also the more names that get added, then the more likely the DB becomes the first point of reference for someone compiling a 'nominal' list of a unit.

    Therefore, i would definately be interested in your Rangoon prisoner lists. Only issue i see is backtracking the men listed to the appropriate WO 361 file reference (unless you've already done it elsewhere)

    Cheers
     

Share This Page