DLXS Workshop Materials

From DLXS Documentation

(Difference between revisions)
Jump to: navigation, search
Line 2: Line 2:
==[[Finding Aids]]==
==[[Finding Aids]]==
 +
= Findaid Class Collection Implementation =
 +
 +
== [../index.html DLXS Workshop] ==
 +
 +
Findaid Class Instructor: '''Tom Burton-West'''
 +
 +
This portion of the DLXS Workshop focuses on implementing a collection in the Findaid Class. Since many of the steps in implementing a Findaid Class collection are very similar to implementing a Text Class collection, the emphasis at the workshop will be on the differences between Text Class and Findaid Class and on dealing with practical implementation issues. Since EAD encoding practices vary widely we will highlight issues arising from different encoding practices and how to resolve them.
 +
 +
<span class="sectionheader">''More general documentation: ''[http://www.dlxs.org/docs/13/class/findaid/index.html Findaid Class][http://www.dlxs.org/docs/13/class/findaid/pis.html ]</span>
 +
 +
----
 +
 +
* [#Overview Overview and Introduction]
 +
* Comparison with Text Class
 +
* [#FindaidEncodingPractices Findaid Class Encoding Practices and Processes]  Brief version of this document [fc_brief.html Brief version]
 +
** [#ead2002DTDOverview EAD 2002 DTD Overview]
 +
** [#PracticalEADEncodingIssues Practical EAD Encoding Issues]
 +
** [#DataPrep Data Preparation]
 +
* [#FindaidClassIndexBuildingwithXPAT Findaid Class Index Building with XPAT]
 +
** [#SetupDirectoriesforXPATIndex Set Up Directories and Files for XPAT Indexing]
 +
** [#BuildtheXPATIndex Build the XPAT Index]
 +
** [#FabRegions Fabricated Regions in FindaidClass]
 +
* [#FindaidClassCollectiontoWeb Findaid Class Collection to Web]
 +
** [#ReviewtheCollectionDatabaseEntrywithCollMgr Review the Collection Database Entry with CollMgr ]
 +
** [#ReviewtheGroupsDatabaseEntrywithCollMgr Review the Groups Database Entry with CollMgr]
 +
** [#MakeCollectionmaps Make Collection Map]
 +
** [#SetuptheCollectionsWebDirectory Set Up the Collection's Web Directory]
 +
** [#TryItOut Try It Out]
 +
* [#linking Linking from Finding Aids Using the ID Resolver]
 +
 +
== Overview ==
 +
 +
[#Top go to table of contents]
 +
 +
''' Outline'''
 +
 +
* Overview
 +
* Comparison with Text Class
 +
* Run through with samplefa/workshopfa
 +
** Data Prep
 +
** Indexing
 +
** Collection to Web
 +
* Repeat with non-Bentley finding aids
 +
** Troubleshooting and resolving problems
 +
** (prep scripts, config files, perl code, XSL)
 +
** Discussion of tips from other Finding Aids implementors
 +
 +
=== Overview of Preprocessing, Data Preparation and Indexing steps: ===
 +
 +
'''Data Preparation'''
 +
 +
# [#DataPrepStep1 validating the files individually] against the EAD ''2002'' DTD<br />'''make validateeach'''<br />
 +
# [#DataPrepStep2 concatenating the files into one larger XML file]<br />'''make prepdocs'''<br />
 +
# [#DataPrepStep3 validating the concatenated file] against the ''dlxsead2002'' DTD:<br />'''make validate'''<br />
 +
# [#DataPrepStep4 "normalizing" the concatenated file.]<br />'''make norm'''<br />
 +
# [#DataPrepStep5 validating the normalized concatenated file against the ''dlxsead2002'' DTD]<br />'''make validate'''<br />
 +
 +
The end result of these steps is a file containing the concatenated EADs wrapped in a &lt;COLL&gt; element which validates against the dlxsead2002 and is ready for indexing:
 +
 +
&lt;COLL&gt;<br />&lt;ead&gt;&lt;eadheader&gt;&lt;eadid&gt;1&lt;/eadid&gt;...&lt;/eadheader&gt;... content&lt;/ead&gt;<br />&lt;ead&gt;&lt;eadheader&gt;&lt;eadid&gt;2&lt;/eadid&gt;...&lt;/eadheader&gt;... content&lt;/ead&gt;<br />&lt;ead&gt;&lt;eadheader&gt;&lt;eadid&gt;3&lt;/eadid&gt;...&lt;/eadheader&gt;... content&lt;/ead&gt;<br />&lt;/COLL&gt;
 +
 +
 +
'''WARNING!'''    If are extra characters or some other problem with the part of the program that strips out the xml declaration and the docytype declearation the file will end up like:
 +
 +
 +
&lt;COLL&gt;<br />baddata&lt;ead&gt;&lt;eadheader&gt;&lt;eadid&gt;1&lt;/eadid&gt;...&lt;/eadheader&gt;... content&lt;/ead&gt;<br />baddata&lt;ead&gt;&lt;eadheader&gt;&lt;eadid&gt;2&lt;/eadid&gt;...&lt;/eadheader&gt;... content&lt;/ead&gt;<br />baddata&lt;ead&gt;&lt;eadheader&gt;&lt;eadid&gt;3&lt;/eadid&gt;...&lt;/eadheader&gt;... content&lt;/ead&gt;<br />&lt;/COLL&gt;
 +
 +
In this case you will get "character data not allowed" or similar errors during the make validate step. You can troubleshoot by looking at the concatenated file and/or checking your original EADs.
 +
 +
'''Indexing'''
 +
 +
# '''make singledd''' indexes words for texts that have been concatenated into on large file for a collection. This is the recommended process.
 +
# '''make xml''' indexes the XML structure by reading the DTD. Validates as it indexes.
 +
# '''make post''' builds and indexes fabricated regions based on the XPAT queries stored in the workshopfa.extra.srch file.
 +
 +
=== Types of changes to accomodate differing encoding practices and/or interface changes ===
 +
 +
* Custom preprocessing
 +
* Add dummy EAD to data
 +
* Modify prep scripts (Makefile, preparedocs.pl, validateeach.csh)
 +
* Modify *inp files (DOCTYPE declarations and entities)
 +
* Modify fabricated regions (*.extra.srch)
 +
* Modify CollMgr entries
 +
* Modify findaidclass.cfg (change table of contents sections)
 +
* Subclass FindaidClass.pm
 +
* Modify XSL
 +
* Modify XML templates
 +
* Modify CSS
 +
 +
==  ==
 +
 +
== Findaid Class Encoding Practices and Processes ==
 +
 +
[#Top go to table of contents]
 +
 +
In Findaid Class Encoding Practices and Processes we discuss the elements and attributes required for "out of the box" Findaid Class delivery, various encoding issues, and preparing the work environment and validating the data.
 +
 +
----
 +
 +
=== EAD 2002 DTD Overview ===
 +
 +
These instructions assume that you have already encoded your finding aids files in the XML-based [http://www.loc.gov/ead/ EAD 2002 DTD]. If you have finding aids encoded using the older EAD 1.0 standard or are using the SGML version of EAD2002, you will need to convert your files to the XML version of EAD2002.  When converting from SGML to XML a number of character set issues may arise.  These are pretty much the same issues that were described for text class see [../conversion/index.html Data Conversion: Unicode, XML, and Normalization] .
 +
 +
Resources for converting from EAD 1.0 to EAD2002 and/or from SGML EAD to XML EAD are available from:
 +
 +
* The Society of American Archivists EAD Tools page:http://www.archivists.org/saagroups/ead/tools.html
 +
* Library of Congress EAD conversion toolshttp://lcweb2.loc.gov/music/eadmusic/eadconv12/ead2002_r.html
 +
 +
Other good sources of information about EAD encoding practices and practical issues involved with EADs are:
 +
 +
* Library of Congress EAD page http://www.loc.gov/ead/ (This is the home of the EAD standard
 +
* EAD2002 tag library http://www.loc.gov/ead/tglib/index.html
 +
* The Society of American Archivists EAD Help page: http://www.archivists.org/saagroups/ead/
 +
* Various EAD Best Practice Guidelines listed on the Society of American Archivists EAD essentials page: [http://www.archivists.org/saagroups/ead/ http://www.archivists.org/saagroups/ead/essentials.html] (the links to BPGs are at the bottom of the page)
 +
* The EAD listserv http://listserv.loc.gov/listarch/ead.html
 +
 +
The EAD standard was designed as a ´loose¡ standard in order to accommodate the large variety in local practices for paper finding aids and make it easy for archives to convert from paper to electronic form.  As a result, conformance with the EAD standard still allows a great deal of variety in encoding practices.
 +
 +
The DLXS software is primarily designed as a system for mounting University of Michigan collections.  In the case of finding aids, the software has been designed to accommodate the encoding practices of the Bentley Historical Library. The more similar your data and setup is to the Bentley’s, the easier is will be to integrate your finding aids collection with DLXS.  If your practices differ significantly from the Bentley’s, you will probably need to do some preprocessing of your files and/or modifications to various files in DLXS.  We have found that the largest number of issues in implementing Findaid Class for member institutions is dealing with differences in encoding practices. We will cover various issues that commonly arise.
 +
 +
More information on the Bentley's encoding practices and workflow:
 +
 +
* Overview of Bentley's workflow process for Finding Aids http://bentley.umich.edu/EAD/eadproj.htm
 +
* Description of Bentley Finding Aids and their presentation on the web http://bentley.umich.edu/EAD/findaids.htm
 +
* Bentley MS Word EAD templates and macros http://bentley.umich.edu/EAD/bhlfiles.htm
 +
* Description of EAD tags used in Bentley EADs http://bentley.umich.edu/EAD/bhltags.htm
 +
 +
----
 +
 +
=== Practical EAD Encoding Issues ===
 +
 +
There are a number of encoding issues that may affect the data preparation, indexing, searching, and rendering of your finding aids. Some of them are:
 +
 +
* [fc_char.html Character Encoding issues]
 +
* [fc_ids Attribute ids must be unique within the entire collection ]
 +
* If you use attribute ids and corresponding targets within your EADs preparedocs.pl may need to be modified.
 +
* &lt;eadid&gt; should be less than about 20 characters in length
 +
* UTF-8 Byte Order Marks (BOM) should be removed from EADs prior to concatenation
 +
* XML processing instructions should be removed from EADs prior to concatenation
 +
* Multiline DOCTYPE declarations are currently not properly handled by the data prep scripts
 +
* If your DOCTYPE declaration contains entitys, you need to modify the appropriate *inp files accordingly
 +
* Out-of-the-box &lt;dao&gt; handling may need to be modified for your needs
 +
* If your &lt;unititle&gt; element precedes your &lt;origination&gt; element in <span class="unixcommand">the top level &lt;did&gt;, you will have to modify the maintitle fabricated region query in xxx.extra.srch </span>
 +
* If you have encoded &lt;unitdate&gt;s as siblings of &lt;unittitle&gt;s, you may have to modify the appropriate XSL templates
 +
* If you do not use a &lt;frontmatter&gt; element, you will have to make modifications to various files to provide an appropriate "Title Page" region based on the &lt;eadheader&gt;
 +
* If your encoding practices for &lt;biohist&gt; differ from the Bentley's, you may need to make changes in findaidclass.cfg or create a subclass of FindaidClass and override FindaidClass:: GetBioghistTocHead, and/or change the appropriate XSL files.
 +
* If you want &lt;relatedmaterial&gt;,&lt;separatedmaterial&gt; to show up in the table of contents (TOC) on the left hand side of the Finding Aids, you may have to modify findaidclass.cfg and make other modifications to the code. This also applies if there are other sections of the finding aid not listed in the out-of-the-box findaidclass.cfg %gSectHeadsHash.
 +
* If you want the middleware to use the &lt;head&gt; element for labeling sections instead of the default hard-coded values in findaidclass.cfg, you may need to make changes to the XSL and possibly modify other files.
 +
 +
----
 +
 +
=== Data Preparation ===
 +
 +
For today, we are going to be working with some texts that are already in Findaid Class. We will be building them into a collection we are going to call '''workshopfa'''.
 +
 +
This documentation will make use of the concept of the <span class="unixcommand">$[../overview/dirstructure.html DLXSROOT]</span>, which is the place at which your DLXS directory structure starts. We generally use <span class="unixcommand">/l1/</span>, but for the workshop, we each have our own <span class="unixcommand">$DLXSROOT</span> in the form of <span class="unixcommand">/l1/workshop/userX/dlxs/</span>. To check your <span class="unixcommand">$DLXSROOT</span>, type the following commands at the command prompt:
 +
 +
<blockquote>
 +
 +
cd $DLXSROOT<br />pwd
 +
 +
</blockquote>
 +
 +
The <span class="unixcommand">prep</span> directory under <span class="unixcommand">$DLXSROOT</span> is the space for you to take your encoded finding aids and "package them up" for use with the DLXS middleware. Create your basic directory <span class="unixcommand">$DLXSROOT/prep/w/workshopfa</span> and its <span class="unixcommand">data</span> subdirectory with the following command:
 +
 +
<blockquote>
 +
 +
mkdir -p $DLXSROOT/prep/w/workshopfa/data
 +
 +
</blockquote>
 +
 +
Move into the <span class="unixcommand">prep</span> directory with the following command:
 +
 +
<blockquote>
 +
 +
cd $DLXSROOT/prep/w/workshopfa
 +
 +
</blockquote>
 +
 +
This will be your staging area for all the things you will be doing to your texts, and ultimately to your collection. At present, all it contains is the <span class="unixcommand">data</span> subdirectory you created a moment ago. We will be populating it further over the course of the next two days. Unlike the contents of other collection-specific directories, everything in <span class="unixcommand">prep</span> should be ultimately expendable in the production environment.
 +
 +
Copy the necessary files into your <span class="unixcommand">data</span> directory with the following commands:
 +
 +
<blockquote>
 +
 +
cp $DLXSROOT/prep/s/samplefa/data/*.xml $DLXSROOT/prep/w/workshopfa/data/.
 +
 +
</blockquote>
 +
 +
We'll also need a few files to get us started working. They will need to be copied over as well, and also have paths adapted and collection identifiers changed. Follow these commands:
 +
 +
<blockquote>
 +
 +
 +
cp $DLXSROOT/prep/s/samplefa/validateeach.csh $DLXSROOT/prep/w/workshopfa/.
 +
cp $DLXSROOT/prep/s/samplefa/samplefa.xml.inp $DLXSROOT/prep/w/workshopfa/workshopfa.xml.inp
 +
cp $DLXSROOT/prep/s/samplefa/samplefa.text.inp $DLXSROOT/prep/w/workshopfa/workshopfa.text.inp
 +
mkdir -p $DLXSROOT/obj/w/workshopfa
 +
mkdir -p $DLXSROOT/bin/w/workshopfa
 +
cp $DLXSROOT/bin/s/samplefa/preparedocs.pl $DLXSROOT/bin/w/workshopfa/.
 +
cp $DLXSROOT/bin/s/samplefa/Makefile $DLXSROOT/bin/w/workshopfa/Makefile
 +
 +
</blockquote>
 +
 +
Now you'll need to edit these files to ensure that the paths match your <span class="unixcommand">$DLXSROOT</span> and that the collection name is ''workshopfa'' instead of ''samplefa''.
 +
 +
''STOP!! Make sure you edit the files before going to the next steps!! ''
 +
 +
Make sure you change these files:
 +
 +
* $DLXSROOT/prep/w/workshopfa/validateeach.csh
 +
* $DLXSROOT/bin/w/workshopfa/Makefile (see below for details)
 +
 +
You can run this command to check to see if you forgot to change samplefa to workshopfa:
 +
 +
grep "samplefa" $DLXSROOT/bin/w/workshopfa/* $DLXSROOT/prep/w/workshopfa/* |grep -v "#"
 +
 +
With the ready-to-go ead2002 encoded finding aids files in the <span class="unixcommand">data</span> directory, we are ready to begin the preparation process. This will include:
 +
 +
# [#DataPrepStep1 validating the files individually] against the EAD ''2002'' DTD
 +
# [#DataPrepStep2 concatenating the files into one larger XML file]
 +
# [#DataPrepStep3 validating the concatenated file] against the ''dlxsead2002'' DTD
 +
# [#DataPrepStep4 "normalizing" the concatenated file.]
 +
# [#DataPrepStep5 validating the normalized concatenated file against the ''dlxsead2002'' DTD]
 +
 +
These steps are generally handled via the <span class="unixcommand">Makefile</span> in <span class="unixcommand">$DLXSROOT/bin/s/samplefa</span> which we have copied to $DLXSROOT/bin/w/workshopfa. To see the Makefile and how it is used, [makefile.html click here].
 +
 +
Make sure you changed your copy of the Makefile to reflect
 +
 +
/w/workshopfa instead of /s/samplefa. You will want to change lines 2 and 3 accordingly
 +
 +
 +
    1 
 +
    2  NAMEPREFIX = samplefa
 +
    3  FIRSTLETTERSUBDIR = s
 +
 +
Tip: Be sure not to add any space after the workshopfa or w. The Makefile ignores space immediately before and after the equals sign but treats all other space as part of the string. I you accidentally put a space after the FIRSTLETTERSUBDIR = s , you will get an error like "[validateeach] Error 127" If you look closely at the first line of what the Makefile reported to standard output (see below) you will see that instead of running the command:
 +
 +
/l1/workshop/tburtonw/dlxs/prep/w/workshopfa/validateeach.csh
 +
 +
which just calls the validateeach c-shell script
 +
 +
it tried to run a directory name: "/l1/workshop/tburtonw/dlxs/prep/w" with the argument "/workshopfa/validateeach.csh" which does not make sense
 +
 +
  % make validateeach
 +
/l1/workshop/tburtonw/dlxs/prep/w /workshopfa/validateeach.csh
 +
make: execvp: /l1/workshop/tburtonw/dlxs/prep/w: Permission denied
 +
make: [validateeach] Error 127 (ignored)
 +
 +
Further note on editing the Makefile: If you modify or write your own Make targets, you need to make sure that a real "tab" starts each command line rather than spaces. The easiest way to check for these kinds of errors is to use "cat -vet Makefile" to show all spaces, tabs and newlines.
 +
 +
If you are doing this at your home institution you will also want to make sure you change $DLXSROOT, and the locations of the various binaries to match your installation. We will not need to do this for the workshop.
 +
 +
''These changes do not apply for the workshop''
 +
* Change $DLXSROOT /l1/dev/userxx to your $DLXSROOT on every line that uses it
 +
* Change XPATBINDIR = /l/local/bin/ to the location of the <span class="unixcommand">xpat</span> binary in your installation
 +
* Change the location of the <span class="unixcommand">osx</span> binary from
 +
OSX = /l/local/bin/osx
 +
to the location in your installation
 +
* Change the location of the <span class="unixcommand">osgmlnorm</span> binary from
 +
OSGMLNORM = /l/local/bin/osgmlnorm
 +
to the location in your installation
 +
 +
Tip: oxs and osgmlnorm are installed as part of the OpenSP package. If you are using linux, make sure that the OpenSP package for your version of linux is installed and make sure the paths above are changed to match your installation. If you are using Solaris you will have to install (and possibly compile) OpenSP. You may also need to make sure the $LD_LIBRARY_PATH environment variable is set so that the OpenSP programs can find the required libraries. For troubleshooting such problems the unix '''ldd''' utility is invaluble. [../troubleshooting/tools.html Information on OpenSP]
 +
 +
----
 +
 +
'''Step 1: Validating the files individually against the EAD 2002 DTD'''
 +
 +
<blockquote>
 +
 +
cd $DLXSROOT/bin/w/workshopfa
 +
make validateeach
 +
 +
 +
The Makefile runs the following command:
 +
% /l1/workshop/userXX/dlxs/prep/w/workshopfa/validateeach.csh
 +
 +
</blockquote>
 +
 +
What's happening: The makefile is running the c-shell script [validateeach.html validateeach.sh] in the prep directory. The script creates a temporary file without the public DOCTYPE declaration, runs <span class="unixcommand">onsgmls</span> on each of the resulting XML files in the <span class="unixcommand">data</span> subdirectory to make sure they conform with the EAD 2002 DTD. If validation errors occur, error files will be in the <span class="unixcommand">data</span> subdirectory with the same name as the finding aids file but with an extension of <span class="unixcommand">.err</span>. If there are validation errors, fix the problems in the source XML files and re-run.
 +
 +
Check the error files by running the following commands
 +
 +
<blockquote>
 +
 +
  ls -l $DLXSROOT/prep/w/workshopfa/data/*err
 +
 +
if there are any *err files, you can look at them with the following command: </blockquote><blockquote>
 +
 +
  less  $DLXSROOT/prep/w/workshopfa/data/*err
 +
 +
</blockquote>
 +
 +
There are not likely to be any errors with the '''workshopfa''' data, but tell the instructor if there are.
 +
 +
----
 +
 +
'''Step 2: Concatentating the files into one larger XML file (and running some preprocessing commands) '''
 +
 +
<blockquote>
 +
 +
 +
cd $DLXSROOT/bin/w/workshopfa
 +
make prepdocs
 +
 +
</blockquote><blockquote>
 +
 +
The Makefile runs the following command:
 +
$DLXSROOT/bin/w/workshopfa/preparedocs.pl $DLXSROOT/prep/w/workshopfa/data $DLXSROOT/obj/w/workshopfa/workshopfa.xml $DLXSROOT/prep/w/workshopfa/logfile.txt
 +
 +
This runs the preparedocs.pl script on all the files in the specified data directory and writes the output to the workshopfa.xml file in the appropriate /obj subdirectory. It also outputs a logfile to the /prep directory:</blockquote>
 +
 +
The Perl script does two sets of things:
 +
 +
# Concatenates all the files
 +
# Runs a number of preprocessing steps on all the files
 +
 +
'''Concatenating the files '''
 +
 +
The script finds all XML files in the <span class="unixcommand">data</span> subdirectory,and then strips off and xml declaration and doctype declaration from each file before concatenating them together. It also wraps the concatenated EADs in a &lt;COLL&gt; tag . The end result looks like:
 +
 +
 +
&lt;COLL&gt;<br />&lt;ead&gt;&lt;eadheader&gt;&lt;eadid&gt;1&lt;/eadid&gt;...&lt;/eadheader&gt;... content&lt;/ead&gt;<br />&lt;ead&gt;&lt;eadheader&gt;&lt;eadid&gt;2&lt;/eadid&gt;...&lt;/eadheader&gt;... content&lt;/ead&gt;<br />&lt;ead&gt;&lt;eadheader&gt;&lt;eadid&gt;3&lt;/eadid&gt;...&lt;/eadheader&gt;... content&lt;/ead&gt;<br />&lt;/COLL&gt;
 +
 +
 +
'''WARNING!'''    If are extra characters or some other problem with the part of the program that strips out the xml declaration and the docytype declearation the file will end up like:
 +
 +
 +
&lt;COLL&gt;<br />baddata&lt;ead&gt;&lt;eadheader&gt;&lt;eadid&gt;1&lt;/eadid&gt;...&lt;/eadheader&gt;... content&lt;/ead&gt;<br />baddata&lt;ead&gt;&lt;eadheader&gt;&lt;eadid&gt;2&lt;/eadid&gt;...&lt;/eadheader&gt;... content&lt;/ead&gt;<br />baddata&lt;ead&gt;&lt;eadheader&gt;&lt;eadid&gt;3&lt;/eadid&gt;...&lt;/eadheader&gt;... content&lt;/ead&gt;<br />&lt;/COLL&gt;
 +
 +
This will cause the document to be invalid since the dlxsead2002.dtd does not allow anything between the closing tag of one &lt;/ead&gt; and the opening tag of the next one &lt;ead&gt;
 +
 +
Some of the possible causes of such a problem are:
 +
 +
* UTF-8 Byte Order Marks at the beginning of the file
 +
* DOCTYPE declaration on more than one line
 +
* XML processing instructions
 +
 +
'''Preprocessing steps'''
 +
 +
The perl program also does some preprocessing on all the files. These steps are customized to the needs of the Bentley. You should look at the perl code and modify it so it is appropriate for your encoding practices.
 +
 +
The preprocessing steps are:
 +
 +
* finds all id attributes and prepends a number to them
 +
* adds a prefix string "dao-bhl" to all DAO links (You probably will want to change this)
 +
* removes empty <span class="unixcommand">persname</span>, <span class="unixcommand">corpname</span>, and <span class="unixcommand">famname</span> elements
 +
 +
The output of the combined concatenation and preprocessing steps will be the one collection named xml file which is deposited into the obj subdirectory.
 +
 +
If your collections need to be transformed in any way, or if you do not want the transformations to take place (the DAO changes, for example), edit preparedocs.pl file to effect the changes. Some changes you may want to make include:
 +
 +
* Changing the algorithm used to make id attibute unique. For example if your encoding practices use id attributes and targets, the out-of-the-box algorithm will remove the relationship between the attributes and targets. One possible modification might be to modify the algorithm to prepend the eadid or filename to all id and target attributes.
 +
* Modifying the program to read a list of files or list of eadids so that the files are concatenated in a particular order. The default sort order for search results is in occurance order, which translates to the order in which the eads are concatenated. If you write a script which looks at the eads for some element that you want to sort by and then outputs a list of filenames sorted by that order, you could then pass that file to a modified preparedocs.pl so it would concatenate the files in the order listed.
 +
 +
----
 +
 +
'''Step 3: Validating the concatenated file against the dlxsead2002 DTD'''
 +
 +
<blockquote>
 +
 +
make validate
 +
 +
The Makefile runs the following command:
 +
onsgmls -wxml -s -f $DLXSROOT/prep/w/workshopfa/workshopfa.errors $DLXSROOT/misc/sgml/xml.dcl $DLXSROOT/prep/w/workshopfa/workshopfa.xml.inp $DLXSROOT/obj/w/workshopfa/workshopfa.xml
 +
 +
</blockquote>
 +
 +
This runs the onsgmls command against the concatenated file using the dlxs2002dtd, and writes any errors to the workshopfa.errors file in the appropriate subdirectory in $DLXSROOT/prep/c/collection.. [validate.html More details]
 +
 +
Note that we are running this using <span class="unixcommand">'''workshopfa.xml.inp'''</span> not <span class="unixcommand">'''workshop.text.inp'''</span>. The '''workshopfa.xml.inp '''file points to '''$DLXSROOT/misc/sgml/dlxsead2002.ead''' which is the ''dlxsead2002'' DTD. The ''dlxsead2002'' DTDis exactly the same as the ''EAD2002'' DTD, but adds a wrapping element, <span class="unixcommand">&lt;COLL&gt;</span>, to be able to combine more than one <span class="unixcommand">ead</span> element, more than one finding aid, into one file. The larger file will be indexed with XPAT tomorrow. It is, of course, a good idea to validate the file now before going further.
 +
 +
Check for errors by looking for the file <span class="unixcommand">'''$DLXSROOT/prep/w/workshopfa/workshopfa.errors'''</span> which will be present and contain messages about what caused the file to be considered invalid if there are errors.
 +
 +
If you see errors at this point (assuming there were no errors during the validateeach step) is that there was a problem with the preparedocs.pl processing. Some common causes of problems are:
 +
 +
* The DOCTYPE declaration did not get completely removed. (The current scripts don't always remove multiline DOCTYPE declearations)
 +
* There was a UTF-8 Byte Order Mark at the begginning of one or more of the concatenated files
 +
 +
Run the following command
 +
 +
<blockquote>
 +
 +
  ls -l $DLXSROOT/prep/w/workshopfa/workshopfa.errors
 +
 +
</blockquote>
 +
 +
If there is a workshopfa.errors file then run the following command to look at the errors reported
 +
 +
<blockquote>
 +
 +
  less $DLXSROOT/prep/w/workshopfa/workshopfa.errors
 +
 +
$ less $DLXSROOT/prep/w/workshopfa/workshopfa.errors<br /> onsgmls:/l1/dev/tburtonw/misc/sgml/xml.dcl:1:W: SGML declaration was not implied<br />
 +
 +
The above error can be ignored, but if you see any other errors '''STOP!''' You need to determine the cause of the problem, fix it, and rerun the steps until there are no errors from make validate. If you continue with the next steps in the process with an invalid xml document, the errors will compound and it will be very difficult to trace the cause of the problem.
 +
 +
Note: To avoid seeing this error add the "-w no-explicit-sgml-decl" flag to the Makefile on line 83. Change line 83 of the Makefile
 +
 +
from
 +
 +
<blockquote>
 +
 +
onsgmls -wxml -s -f $(PREPDIR)$(NAMEPREFIX).errors $(XMLDECL) $(XMLDOCTYPE) $(XMLFILE)
 +
 +
</blockquote> to <blockquote>
 +
 +
onsgmls -wxml -w no-explicit-sgml-decl -s -f $(PREPDIR)$(NAMEPREFIX).errors $(XMLDECL) $(XMLDOCTYPE) $(XMLFILE)
 +
 +
</blockquote>
 +
 +
''This will be fixed in the next release of DLXS Findaid Class. ''
 +
 +
'''Step 4: Normalizing the concatenated file'''
 +
 +
<blockquote>
 +
 +
make norm
 +
 +
The Makefile runs a series of copy statements and two main commands:
 +
 +
 +
1.)  /l/local/bin/osgmlnorm -f $DLXSROOT/prep/s/samplefa/samplefa.errors $DLXSROOT/misc/sgml/xml.dcl $DLXSROOT$DLXSROOT/prep/s/samplefa/samplefa.xml.inp $DLXSROOT/obj/s/samplefa/samplefa.xml.prenorm &gt; /l1/dev/tburtonw/obj/s/samplefa/samplefa.xml.postnorm
 +
 +
2.)  /l/local/bin/osx -bUTF-8 -xlower -xempty -xno-nl-in-tag -f /l1/dev/tburtonw/prep/s/samplefa/samplefa.errors /l1/dev/tburtonw/misc/sgml/xml.dcl /l1/dev/tburtonw/prep/s/samplefa/samplefa.xml.inp /l1/dev/tburtonw/obj/s/samplefa/samplefa.xml.postnorm &gt; /l1/dev/tburtonw/obj/s/samplefa/samplefa.xml.postnorm.osx
 +
 +
</blockquote>
 +
 +
These commands ensure that your collection data is normalized. What this means is that any attributes are put in the order in which they were defined in the DTD. Even though your collection data is XML and attribute order should be irrelevant (according to the XML specification), due to a bug in one of the supporting libraries used by xmlrgn (part of the indexing software), attributes must appear in the order that they are definded in the DTD. If you have "out-of-order" attributes and don't run make norm, you will get ''"invalid endpoints"'' errors during the make post step.
 +
 +
Step one, which normalizes the document writes its errors to <span class="unixcommand">$DLXSROOT/prep/s/samplefa/samplefa.errors</span>. Be sure to check this file.
 +
 +
Step 2, which runs osx to convert the normalized document back into XML produces lots of error messages which are written to standard output. These are caused because we are using an XML DTD (the EAD 2002 DTD) and osx is using it to validate against the SGML document created by the osgmlnorm step. These are the only errors which may generally be ignored. However, if the next recommended step, which is to run "make validate" again reveals an invalid document, you may want to rerun osx and look at the errors for clues. (Only do this if you are sure that the problem is not being caused by XML processing instructions in the documents as explained below)
 +
 +
'''Step 5: Validating the normalized file against the dlxsead2002 DTD'''
 +
 +
<blockquote>
 +
 +
make validate
 +
 +
</blockquote>
 +
 +
We run this step again to make sure that the normalization process did not produce an invalid document. This is necessary because under some circumstances the "make norm" step can result in invalid XML. One known cause of this is the presense of XML processing instructions. For example: '''"&lt;?Pub Caret1?&gt;"'''. Although XML processing instructions are supposed to be ignored by any XML application that does not understand them, the problem is that when we use sgmlnorm and osx, which are SGML tools, they end up munging the output XML. The recommended workaround is to add a preprocessing step to remove any XML processing instructions from your EADs before you run "make prepdocs", or to include some code in preparedocs.pl that will strip out XML priocessing instructions prior to concatenating the EADs.
 +
 +
=== ''More Documentation'' ===
 +
 +
* [http://www.dlxs.org/docs/13/class/findaid/dtd.html The Findaid Class DTD]
 +
* [http://www.dlxs.org/docs/13/class/findaid/prep.html Preparation for Index Building (Finding Aids)]
 +
 +
== <span class="dayheader"> </span>Findaid Class Index Building with XPAT ==
 +
 +
[#Top go to table of contents]
 +
 +
In this section the '''workshopfa''' XML will be indexed with the XPAT search engine, preparing it for use with the DLXS middleware.
 +
 +
----
 +
 +
=== Set Up Directories and Files for XPAT Indexing ===
 +
 +
First, we need to create the rest of the directories in the '''workshopfa''' environment with the following commands:
 +
 +
<blockquote>
 +
 +
mkdir -p $DLXSROOT/idx/w/workshopfa
 +
 +
</blockquote>
 +
 +
The <span class="unixcommand">bin</span> directory we created yesterday holds any scripts or tools used for the collection specifically; <span class="unixcommand">obj</span> ( created earlier) holds the "object" or XML file for the collection, and <span class="unixcommand">idx</span> holds the XPAT indexes. Now we need to finish populating the directories.
 +
 +
<blockquote>
 +
 +
<br />cp $DLXSROOT/prep/s/samplefa/samplefa.blank.dd $DLXSROOT/prep/w/workshopfa/workshopfa.blank.dd<br />cp $DLXSROOT/prep/s/samplefa/samplefa.extra.srch $DLXSROOT/prep/w/workshopfa/workshopfa.extra.srch
 +
 +
</blockquote>
 +
 +
'''Each of these files need to be edited '''to reflect the new collection name and the paths to your particular directories. This will be true when you use these at your home institution as well, even if you use the same directory architecture as we do, because they will always need to reflect the unique name of each collection. Failure to change even one file can result in puzzling errors, because the scripts ''are'' working, just not necessarily in the directories you are looking at.
 +
 +
grep -l "samplefa" <span class="unixcommand">$DLXSROOT/prep/w/workshopfa/</span><nowiki>*</nowiki>
 +
 +
will check for changing s/samplefa to w/workshopfa. If you are at the workshop that should be all you need. However if you are doing this at your home institution you need to replace "/l1/" by whatever $DLXSROOT is on your server. If you don't have an /l1 directory on your server (which is very likely if you are not here using a DLPS machine) you can check with:
 +
 +
grep -l "l1" <span class="unixcommand">$DLXSROOT/prep/w/workshopfa/</span><nowiki>*</nowiki>
 +
 +
----
 +
 +
=== Build the XPAT Index ===
 +
 +
Everything is now set up to build the XPAT index. The <span class="unixcommand">Makefile</span> in the <span class="unixcommand">bin</span> directory contains the commands necessary to build the index, and can be executed easily.
 +
 +
To create an index for use with the Findaid Class interface, you will need to index the words in the collection, then index the XML (the structural metadata, if you will), and then finally "fabricate" structures based on a combination of elements (for example, defining who the "main author" of a finding aid is, without adding a <span class="unixcommand">&lt;mainauthor&gt;</span> tag around the appropriate <span class="unixcommand">&lt;author&gt;</span> in the <span class="unixcommand">eadheader</span> element). The following commands can be used to make the index:
 +
 +
'''make singledd''' indexes words for texts that have been concatenated into on large file for a collection. This is the recommended process.
 +
 +
'''make xml''' indexes the XML structure by reading the DTD. Validates as it indexes.
 +
 +
'''make post''' builds and indexes fabricated regions based on the XPAT queries stored in the workshopfa.extra.srch file.
 +
 +
<blockquote>
 +
 +
cd $DLXSROOT/bin/w/workshopfa
 +
make singledd
 +
 +
</blockquote><blockquote>
 +
 +
cp /l1/workshop/test02/dlxs/prep/w/workshopfa/workshopfa.blank.dd
 +
/l1/workshop/test02/dlxs/idx/w/workshopfa/workshopfa.dd
 +
/l/local/xpat/bin/xpatbld -m 256m -D /l1/workshop/test02/dlxs/idx/w/workshopfa/workshopfa.dd
 +
cp /l1/workshop/test02/dlxs/idx/w/workshopfa/workshopfa.dd
 +
/l1/workshop/test02/dlxs/prep/w/workshopfa/workshopfa.presgml.dd
 +
 +
</blockquote><blockquote>
 +
 +
make xml
 +
 +
</blockquote><blockquote>
 +
 +
cp /l1/workshop/test02/dlxs/prep/w/workshopfa/workshopfa.presgml.dd
 +
/l1/workshop/test02/dlxs/idx/w/workshopfa/workshopfa.dd
 +
/l/local/xpat/bin/xmlrgn -D /l1/workshop/test02/dlxs/idx/w/workshopfa/workshopfa.dd
 +
/l1/workshop/test02/dlxs/misc/sgml/xml.dcl
 +
/l1/workshop/test02/dlxs/prep/w/workshopfa/workshopfa.inp
 +
/l1/workshop/test02/dlxs/obj/w/workshopfa/workshopfa.xml
 +
 +
cp /l1/workshop/test02/dlxs/idx/w/workshopfa/workshopfa.dd
 +
/l1/workshop/test02/dlxs/idx/w/workshopfa/workshopfa.prepost.dd
 +
 +
</blockquote><blockquote>
 +
 +
make post
 +
 +
</blockquote><blockquote>
 +
 +
cp /l1/workshop/test02/dlxs/prep/w/workshopfa/workshopfa.prepost.dd
 +
/l1/workshop/test02/dlxs/idx/w/workshopfa/workshopfa.dd
 +
touch /l1/workshop/test02/dlxs/idx/w/workshopfa/workshopfa.init
 +
/l/local/xpat/bin/xpat -q /l1/workshop/test02/dlxs/idx/w/workshopfa/workshopfa.dd
 +
&lt; /l1/workshop/test02/dlxs/prep/w/workshopfa/workshopfa.extra.srch
 +
| /l1/workshop/test02/dlxs/bin/t/text/output.dd.frag.pl
 +
/l1/workshop/test02/dlxs/idx/w/workshopfa/
 +
&gt; /l1/workshop/test02/dlxs/prep/w/workshopfa/workshopfa.extra.dd
 +
/l1/workshop/test02/dlxs/bin/t/text/inc.extra.dd.pl
 +
/l1/workshop/test02/dlxs/prep/w/workshopfa/workshopfa.extra.dd
 +
/l1/workshop/test02/dlxs/idx/w/workshopfa/workshopfa.dd
 +
 +
</blockquote>
 +
 +
If you get an ''"invalid endpoints"'' message from "make post", the most likely cause is XML processing instructions or some other corruption. The second "make validate" step should have caught these. Other possible causes of errors during the "make post" step include syntax errors in workshopfa.extra.srch, or the absense of a particular region that is listed in the *.extra.srch file but not present in your collection. For example if you do not have any &lt;corpname&gt; elements in any of the EADs in your collection and you are using the out-of-the-box samplefa.extra.srch, you will see an error message when xpat tries to index the mainauthor region using this rule:
 +
 +
<blockquote>
 +
 +
 +
((region "persname" + region "corpname" + region "famname" + region "name") within (region "origination" within ( re
 +
gion "did" within (region "archdesc")))); {exportfile "/l1/workshop/user11/dlxs/idx/s/samplefa/mainauthor.rgn"}; exp
 +
ort; ~sync "mainauthor";
 +
 +
</blockquote>
 +
 +
The easiest solution is to modify *extra.srch to match the characteristics of your collection. An alternative is to include a "dummy" EAD that contains all the elements that you expect in your collection with no content
 +
 +
----
 +
 +
=== Testing the index ===
 +
 +
At this point it is a good idea to do some testing of the newly created index. Invoke xpat with the following commands
 +
 +
xpatu $DLXSROOT/idx/w/workshopfa/workshopfa.dd
 +
 +
Try searching for some likely regions. Its a good idea to test some of the fabricated regions. Here are a few sample queries:
 +
 +
&gt;&gt; region "ead"
 +
  1: 3 matches
 +
 +
&gt;&gt; region "eadheader"
 +
  2: 3 matches
 +
 +
&gt;&gt; region "mainauthor"
 +
  3: 3 matches
 +
 +
&gt;&gt; region "maintitle"
 +
  4: 3 matches
 +
 +
&gt;&gt; region "admininfo"
 +
  5: 3 matches
 +
 +
 +
----
 +
 +
=== Fabricated Regions in FindaidClass ===
 +
 +
The make post step and the testing steps above leads us into a discussion of the use of '''fabricated regions''' in FindaidClass. uses the <span class="unixcommand">workshopfa.extra.srch</span> file to add to the XPAT index.
 +
 +
"Fabricated" is a term we use to describe what are essentially virtual regions in an XPat indexed text. See a [../xpat/fabregions.html basic description of what a fabricated region is and how they are created].
 +
 +
In Finding Aids, we use fabricated regions for certain uninteresting regions simply so that some code can be shared. For example, the fabricated region "main" is set to refer to <span class="unixcommand">&lt;ead&gt;</span> in FindaidClass with:
 +
 +
(region ead); {exportfile "/l1/idx/b/bhlead/main.rgn"}; export; ~sync "main";
 +
 +
whereas in TextClass "main" can refer to <span class="unixcommand">&lt;TEXT&gt;</span>. Therfore, both FindaidClass and TextClass can share the Perl code, in a higher level subclass, that creates searches for "main".
 +
 +
Other fabricated regions are used for searching such as the maintitle and mainauthor regions.
 +
 +
The majority of the fabricated regions for Findaid Class are used for the creation and display of the left hand table of contents in the "outline" view. The findaidclass.cfg file contains a hash called %gSectHeadsHash which is normally loaded into FindaidClass.pm's tocheads hash in the FindaidClass::_initialize method. The elements of the hash and the corresponding fabiricated regions are used to create the table of contents and to output the XML for the corresponding section of the EAD when one of the TOC links is clicked on by a user. The fabricated regions are used so XPAT can have binary indexes ready to use for fast retrieval of these EAD sections.
 +
 +
Some of the more interesting regions extracted from the samplefa.extra.srch file are listed below.
 +
 +
One of these regions is the add. This used to be <span class="unixcommand">&lt;ADD&gt;</span> in the EAD 1.0 DTD, but now, is created based on the ead2002 DTD's <span class="unixcommand">&lt;descgrp&gt;</span> tag which contains a <span class="unixcommand">type</span> attribute of <span class="unixcommand">add</span>.
 +
 +
A number of issues related to varying encoding practices can be resolved by the appropriate edits to the *.extra.srch file. (Although some of them may require changes to other files as well)
 +
 +
* If your &lt;unititle&gt; element precedes your &lt;origination&gt; element in <span class="unixcommand">the top level &lt;did&gt;, you will have to modify the "maintitle" fabricated region query in *.extra.srch </span>
 +
* If you do not use a &lt;frontmatter&gt; element, you will have to make modifications to various files including modifying *.extra.srch to provide an appropriate "Title Page" region based on the &lt;eadheader&gt;
 +
* If your encoding practices for &lt;biohist&gt; differ from the Bentley's, you may need to make changes in the &lt;bioghist&gt; fabricated region although changes to other files may be suffient. The changes might include: modifying findaidclass.cfg or creating a subclass of FindaidClass and override FindaidClass:: GetBioghistTocHead, and/or changing the appropriate XSL files.
 +
* If you want sections of the finding aid that are not completely within a well-defined element such as &lt;relatedmaterial&gt;or &lt;separatedmaterial&gt; to show up in the table of contents, you may have to create a fabricated region using the appropriate xpat query and then modify findaidclass.cfg and make other modifications to the code.
 +
 +
----
 +
<blockquote>
 +
 +
 +
 +
 +
  (region ead); {exportfile "/l1/workshop/user11/dlxs/idx/s/samplefa/main.rgn"}; export; ~sync "main";
 +
   
 +
    ##
 +
    (((region "&lt;c01".."&lt;/did&gt;" + region "&lt;c02".."&lt;/did&gt;" + region "&lt;c03".."&lt;/did&gt;" + region "&lt;c04".."&lt;/did&gt;" + region "&lt;c05".."&lt;/did&gt;" + region "&lt;c06".."&lt;/did&gt;" + region "&lt;c07".."&lt;/did&gt;" + region "&lt;c08".."&lt;/did&gt;" + region "&lt;c09".."&lt;/did&gt;") not incl ("level=file" + "level=item")) incl "level="); {exportfile "/l1/workshop/user11/dlxs/idx/s/samplefa/c0xhead.rgn"}; export; ~sync "c0xhead";
 +
        ##
 +
    ((region "&lt;origination".."&lt;/unittitle&gt;") within ((region did within region archdesc) not within region dsc)); {exportfile "/l1/workshop/user11/dlxs/idx/s/samplefa/maintitle.rgn"}; export; ~sync "maintitle";
 +
    ##
 +
       
 +
    ((region "persname" + region "corpname" + region "famname" + region "name") within (region "origination" within ( region "did" within (region "archdesc")))); {exportfile "/l1/workshop/user11/dlxs/idx/s/samplefa/mainauthor.rgn"}; export; ~sync "mainauthor";
 +
    ##
 +
   
 +
    (region "abstract" within ((region did within region archdesc) not within region "c01")); {exportfile "/l1/workshop/user11/dlxs/idx/s/samplefa/mainabstract.rgn"}; export; ~sync "mainabstract";
 +
        ##
 +
        ((region unitdate incl "encodinganalog=245$f") within ((region did within region archdesc) not within region dsc)); {exportfile "/l1/workshop/user11/dlxs/idx/s/samplefa/colldate.rgn"}; export; ~sync "colldate";
 +
    ##
 +
   
 +
    (region dsc); {exportfile "/l1/workshop/user11/dlxs/idx/s/samplefa/contentslist.rgn"}; export; ~sync "contentslist";
 +
    ##
 +
      ########## admininfo ########
 +
    admininfot = (region "descgrp-T" incl (region "A-type" incl "admin")); {exportfile "/l1/workshop/user11/dlxs/idx/s/samplefa/admininfo-t.rgn"}; export; ~sync "admininfo-t";
 +
    ##
 +
    ## ########## add ######
 +
    addt = (region "descgrp-T" incl (region "A-type" incl "add")); {exportfile "/l1/workshop/user11/dlxs/idx/s/samplefa/add-t.rgn"}; export; ~sync "add-t";
 +
  ## ########## frontmatter/titlepage ########
 +
  frontmattert = region "frontmatter-T"; {exportfile "/l1/workshop/user11/dlxs/idx/s/samplefa/frontmatter-t.rgn"}; export; ~sync "frontmatter-t";
 +
    ##
 +
    # frontmatter itself not needed as fabricated region since it exists
 +
    # as a regular xml region
 +
    ##
 +
  ## ########## bioghist ########
 +
    bioghist = ((region "bioghist" within region "archdesc") not within region "dsc"); {exportfile "/l1/workshop/user11/dlxs/idx/s/samplefa/bioghist.rgn"}; export; ~sync "bioghist";
 +
   
 +
  ##bioghisthead = ((region "&lt;bioghist" .. "&lt;/head&gt;" within region "archdesc") not within region "dsc"); {exportfile "/l1/workshop/user11/dlxs/idx/s/samplefa/bioghisthead.rgn"}; export; ~sync "bioghisthead";
 +
    ##
 +
  ((region did within region archdesc) not within region dsc); {exportfile "/l1/workshop/user11/dlxs/idx/s/samplefa/summaryinfo.rgn"}; export; ~sync "summaryinfo";;
 +
    ##
 +
  ##
 +
  #############################
 +
  (region "subject" + region "corpname" + region "famname" + region "name" + region "persname" + region "geogname"); {exportfile "/l1/workshop/user11/dlxs/idx/s/samplefa/subjects.rgn"}; export; ~sync "subjects";
 +
  (region "corpname" + region "famname" + region "name" + region "persname"); {exportfile "/l1/workshop/user11/dlxs/idx/s/samplefa/names.rgn"}; export; ~sync "names";
 +
 
 +
   
 +
  #(region "odd-T" ^ (region odd not within region dsc)); {exportfile "/l1/workshop/user11/dlxs/idx/s/samplefa/odd-t.rgn"}; export; ~sync "odd-t"; 
 +
 +
 +
</blockquote>
 +
 +
See a [bhl.extra.srch.txt full listing of the <span class="unixcommand">extra.srch</span> file of the Bentley Historical Library's finding aids].
 +
 +
----
 +
 +
=== ''More Documentation'' ===
 +
 +
* [http://www.dlxs.org/docs/13/class/findaid/indexing.html Indexing the Collection]
 +
* [http://www.dlxs.org/docs/13/class/findaid/fabrgn.html Fabricated regions in Findaid Class]
 +
 +
== Findaid Class Collection to Web ==
 +
 +
[#Top go to table of contents]
 +
 +
These are the final steps in deploying an Findaid Class collection online. Here the '''Collection Manager''' will be used to review the '''Collection Database''' entry for '''workshopfa''' . The '''Collection Manager''' will also be used to check the '''Group Database'''. Finally, we need to work with the collection map and the set up the collection's web directory.
 +
 +
----
 +
 +
=== Review the Collection Database Entry with CollMgr ===
 +
 +
Each collection has a record in the collection database that holds collection specific configurations for the middleware. CollMgr (Collection Manager) is a web based interface to the collection database that provides functionality for editing each collection's record. Collections can be checked-out for editing, checked-in for testing, and released to production. In general, a new collection needs to have a CollMgr record created from scratch before the middleware can be used. If you are starting with the samplefa collmgr as a model make sure to change references from '''s/samplefa''' to '''w/workshopfa''' or whatever you are using for your collection name.
 +
 +
''More Documentation''
 +
 +
* [http://www.dlxs.org/docs/13/collmeta/collmgr-fields.html Collection Manager Field Descriptions]
 +
 +
----
 +
 +
=== Review the Groups Database Entry with CollMgr ===
 +
 +
Another function of CollMgr allows the grouping of collections for cross-collection searching. Any number of collection groups may be created for Findaid Class. Findaid Class supports a group with the groupid "all". It is not a requirement that all collections be in this group, though that's the basic idea. Groups are created and modified using CollMgr.
 +
 +
http://username.ws.umdl.umich.edu/cgi/c/collmgr/collmgr
 +
 +
We won't be doing anything with groups;
 +
 +
----
 +
 +
=== Make Collection Map ===
 +
 +
Collection mapper files exist to identify the regions and operators used by the middleware when interacting with the search forms. Each collection will need one, but most collections can use a fairly standard map file, such as the one in the '''samplefa''' collection. The map files for all Findaid Class collections are stored in $DLXSROOT/misc/f/findaid/maps
 +
 +
Map files take language that is used in the forms and translates it into language for the cgi and for XPAT. For example, if you want your users to be able to search within names, you would need to add a mapping for how you want it to appear in the search interface (case is important, as is pluralization!), how the cgi variable would be set (usually all caps, and not stepping on an existing variable), and how XPAT will identify and retrieve this natively (in XPAT search language).
 +
 +
The first part of the map file is operator mapping, for the form, the cgi, and XPAT. The second part is for region mapping, as in the example above.
 +
 +
<blockquote>
 +
 +
cd $DLXSROOT/misc/f/findaid/maps<br />cp samplefa.map workshopfa.map
 +
 +
</blockquote>
 +
 +
You might note that some of the fields that are defined in the map file correspond to some of the [#FabRegions fabricated regions].
 +
 +
=== ''More Documentation'' ===
 +
 +
* [http://www.dlxs.org/docs/13/collmeta/maps.html DLXS Map Files]
 +
* [http://www.dlxs.org/docs/13/class/findaid/map.html Collection Map Files (Finding Aids)]
 +
 +
----
 +
 +
=== Set Up the Collection's Web Directory ===
 +
 +
Each collection may have a <span class="unixcommand">web</span> directory with custom Cascading Style Sheets, interface templates, graphics, and javascript. The default is for a collection to use the web templates at<span class="unixcommand"> $DLXSROOT/web/f/findaid</span>. Of course, collection specific templates and other files can be placed in a collection specific web directory, and it is necessary if you have any customization at all. ''DLXS Middleware uses [../ui/index.html#fallback fallback] to find HTML related templates, chunks, graphics, js and css files.''
 +
 +
For a minimal collection, you will want two files: index.html and <span class="unixcommand">FindaidClass-specific.css</span>.
 +
 +
<blockquote>
 +
 +
 +
mkdir -p $DLXSROOT/web/w/workshopfa
 +
cp $DLXSROOT/web/s/samplefa/index.html $DLXSROOT/web/w/workshopfa/index.html
 +
cp $DLXSROOT/web/s/samplefa/findaidclass-specific.css $DLXSROOT/web/w/workshopfa/findaidclass-specific.css
 +
 +
</blockquote>
 +
 +
As always, we'll need to change the collection name and paths. You might want to change the look radically, if your HTML skills are up to it.
 +
 +
Note that the browse link on the index.html page is hard-coded to go to the samplefa hard-coded browse.html page. You may want to change this to point to a dynamic browse page (see below). The url for the dynamic browse page is ".../cgi/f/findaid/findaid-idx?c=workshopfa;page=browse".
 +
 +
If you would prefer a dynamic home page, you can copy and modify the home.xml and home.xsl files from $DLXSROOT/web/f/findaid/. Note that they are currently set up to be the home page for all finding aids collections, so you will have to do some considerable editing. However they contain a number of PIs that you may find useful. In order to have these pages actually be used by DLXS, they have to be present in your $DLXSROOT/web/w/workshopfa/ directory and '''there can't be an index.html page in that directory.''' The easiest thing to do, if you have an existing index.html page is to rename it to "index.html.foobar" or something. <br />
 +
 +
=== Create a browse page ===
 +
 +
See the documentation: http://www.dlxs.org/docs/13/collmeta/browse.html
 +
----
 +
 +
=== Try It Out ===
 +
 +
http://''username''.ws.umdl.umich.edu/cgi/f/findaid/findaid-idx
 +
 +
== Linking from Finding Aids Using the ID Resolver ==
 +
 +
[#Top go to table of contents]
 +
 +
How do you do this?
 +
 +
Findaid Class is coded so that if there is an href attribute to the  &lt;dao&gt; element, it will check to see if it contains the string "http". If it does, FindaidClass will not us ID Resolver, but will create a link based on the content of the href attribute of the &lt;dao&gt;. If there is no "http" string in the href attribute, FindaidClass assumes that the href attribute is actully an id and will look up that id in in the idresolver and build a link if it finds the ID in the IDRESOLVER table. The method FilterAllDaos_XML in $DLXSROOT/cgi/f/findaid/FindaidClass.pm can be overridden per collection if different behavior is needed.
 +
 +
If you decide to use this feature, you will want to modify the preprocessing script preparedocs.pl which out-of-the-box inserts the string 'dao-bhl-' after the href. Below is an example of a Bentley &lt;dao&gt; where the id number is 91153-1.
 +
 +
&lt;dao linktype="simple" href="91153-1" show="new" actuate="onrequest"&gt;<br />        &lt;daodesc&gt;<br />          &lt;p&gt;[view selected images]&lt;/p&gt;<br />        &lt;/daodesc&gt;<br />        &lt;/dao&gt;
 +
 +
The preparedocs.pl program would change this to:
 +
 +
&lt;dao linktype="simple" href="dao-bhl-91153-1" show="new" actuate="onrequest"&gt;<br />        &lt;daodesc&gt;<br />          &lt;p&gt;[view selected images]&lt;/p&gt;<br />        &lt;/daodesc&gt;<br />        &lt;/dao&gt;
 +
 +
The ID resolver would look up the id "dao-bhl-91153-1" and replace it with the appropriate URL.
 +
 +
<font color="#0000A0">ID Resolver Data Transformation and Deployment</font>
 +
 +
The ID Resolver is a CGI that takes as input a unique identifier and returns a URI. It is used, for example, by Harper's Weekly to link the text pages in Text Class middleware to the image pages in the Image Class middleware, and vice versa.
 +
 +
Plug something like the following in to your web browser and you should get something back. If you choose to test middleware on a development machine that uses the id resolver, make sure that the middleware on that machine is calling the resolver on the machine with the data, and not the resolver on the production server.
 +
 +
* [http://clamato.umdl.umich.edu/cgi/i/idresolver/idresolver?id=dao-bhl-bl000684 http://clamato.hti.umich.edu/cgi/i/idresolver/idresolver?id=dao-bhl-bl000684]
 +
* which should yield...<br /><code>'''&lt;ITEM MTIME="20030728142225"&gt;&lt;ID&gt;dao-bhl-bl000684 &lt;/ID&gt;&lt;URI&gt;http://images.umdl.umich.edu/cgi/i/image/image-idx?&amp;q1=bl000684&amp;rgn1=bhl_href&amp;type=boolean&amp;med=1&amp;view=thumbnail&amp;c=bhl &lt;/URI&gt;&lt;/ITEM&gt;'''</code>
 +
 +
[http://www.dlxs.org/docs/13/ancil/idresolver.html Information on how to set up the ID resolver]
 +
 +
</blockquote>

Revision as of 17:32, 8 August 2007

You can find the 2007 DLXS Workshop Materials on this web site.

Contents

Finding Aids

Findaid Class Collection Implementation

[../index.html DLXS Workshop]

Findaid Class Instructor: Tom Burton-West

This portion of the DLXS Workshop focuses on implementing a collection in the Findaid Class. Since many of the steps in implementing a Findaid Class collection are very similar to implementing a Text Class collection, the emphasis at the workshop will be on the differences between Text Class and Findaid Class and on dealing with practical implementation issues. Since EAD encoding practices vary widely we will highlight issues arising from different encoding practices and how to resolve them.

More general documentation: Findaid Class[1]

  • [#Overview Overview and Introduction]
  • Comparison with Text Class
  • [#FindaidEncodingPractices Findaid Class Encoding Practices and Processes] Brief version of this document [fc_brief.html Brief version]
    • [#ead2002DTDOverview EAD 2002 DTD Overview]
    • [#PracticalEADEncodingIssues Practical EAD Encoding Issues]
    • [#DataPrep Data Preparation]
  • [#FindaidClassIndexBuildingwithXPAT Findaid Class Index Building with XPAT]
    • [#SetupDirectoriesforXPATIndex Set Up Directories and Files for XPAT Indexing]
    • [#BuildtheXPATIndex Build the XPAT Index]
    • [#FabRegions Fabricated Regions in FindaidClass]
  • [#FindaidClassCollectiontoWeb Findaid Class Collection to Web]
    • [#ReviewtheCollectionDatabaseEntrywithCollMgr Review the Collection Database Entry with CollMgr ]
    • [#ReviewtheGroupsDatabaseEntrywithCollMgr Review the Groups Database Entry with CollMgr]
    • [#MakeCollectionmaps Make Collection Map]
    • [#SetuptheCollectionsWebDirectory Set Up the Collection's Web Directory]
    • [#TryItOut Try It Out]
  • [#linking Linking from Finding Aids Using the ID Resolver]

Overview

[#Top go to table of contents]

Outline

  • Overview
  • Comparison with Text Class
  • Run through with samplefa/workshopfa
    • Data Prep
    • Indexing
    • Collection to Web
  • Repeat with non-Bentley finding aids
    • Troubleshooting and resolving problems
    • (prep scripts, config files, perl code, XSL)
    • Discussion of tips from other Finding Aids implementors

Overview of Preprocessing, Data Preparation and Indexing steps:

Data Preparation

  1. [#DataPrepStep1 validating the files individually] against the EAD 2002 DTD
    make validateeach
  2. [#DataPrepStep2 concatenating the files into one larger XML file]
    make prepdocs
  3. [#DataPrepStep3 validating the concatenated file] against the dlxsead2002 DTD:
    make validate
  4. [#DataPrepStep4 "normalizing" the concatenated file.]
    make norm
  5. [#DataPrepStep5 validating the normalized concatenated file against the dlxsead2002 DTD]
    make validate

The end result of these steps is a file containing the concatenated EADs wrapped in a <COLL> element which validates against the dlxsead2002 and is ready for indexing: 

<COLL>
<ead><eadheader><eadid>1</eadid>...</eadheader>... content</ead>
<ead><eadheader><eadid>2</eadid>...</eadheader>... content</ead>
<ead><eadheader><eadid>3</eadid>...</eadheader>... content</ead>
</COLL>

WARNING! If are extra characters or some other problem with the part of the program that strips out the xml declaration and the docytype declearation the file will end up like: 


<COLL>
baddata<ead><eadheader><eadid>1</eadid>...</eadheader>... content</ead>
baddata<ead><eadheader><eadid>2</eadid>...</eadheader>... content</ead>
baddata<ead><eadheader><eadid>3</eadid>...</eadheader>... content</ead>
</COLL>

In this case you will get "character data not allowed" or similar errors during the make validate step. You can troubleshoot by looking at the concatenated file and/or checking your original EADs.

Indexing

  1. make singledd indexes words for texts that have been concatenated into on large file for a collection. This is the recommended process.
  2. make xml indexes the XML structure by reading the DTD. Validates as it indexes.
  3. make post builds and indexes fabricated regions based on the XPAT queries stored in the workshopfa.extra.srch file.

Types of changes to accomodate differing encoding practices and/or interface changes

  • Custom preprocessing
  • Add dummy EAD to data
  • Modify prep scripts (Makefile, preparedocs.pl, validateeach.csh)
  • Modify *inp files (DOCTYPE declarations and entities)
  • Modify fabricated regions (*.extra.srch)
  • Modify CollMgr entries
  • Modify findaidclass.cfg (change table of contents sections)
  • Subclass FindaidClass.pm
  • Modify XSL
  • Modify XML templates
  • Modify CSS

Findaid Class Encoding Practices and Processes

[#Top go to table of contents]

In Findaid Class Encoding Practices and Processes we discuss the elements and attributes required for "out of the box" Findaid Class delivery, various encoding issues, and preparing the work environment and validating the data.

EAD 2002 DTD Overview

These instructions assume that you have already encoded your finding aids files in the XML-based EAD 2002 DTD. If you have finding aids encoded using the older EAD 1.0 standard or are using the SGML version of EAD2002, you will need to convert your files to the XML version of EAD2002. When converting from SGML to XML a number of character set issues may arise. These are pretty much the same issues that were described for text class see [../conversion/index.html Data Conversion: Unicode, XML, and Normalization] .

Resources for converting from EAD 1.0 to EAD2002 and/or from SGML EAD to XML EAD are available from:

Other good sources of information about EAD encoding practices and practical issues involved with EADs are:

The EAD standard was designed as a ´loose¡ standard in order to accommodate the large variety in local practices for paper finding aids and make it easy for archives to convert from paper to electronic form. As a result, conformance with the EAD standard still allows a great deal of variety in encoding practices.

The DLXS software is primarily designed as a system for mounting University of Michigan collections. In the case of finding aids, the software has been designed to accommodate the encoding practices of the Bentley Historical Library. The more similar your data and setup is to the Bentley’s, the easier is will be to integrate your finding aids collection with DLXS. If your practices differ significantly from the Bentley’s, you will probably need to do some preprocessing of your files and/or modifications to various files in DLXS. We have found that the largest number of issues in implementing Findaid Class for member institutions is dealing with differences in encoding practices. We will cover various issues that commonly arise.

More information on the Bentley's encoding practices and workflow:

Practical EAD Encoding Issues

There are a number of encoding issues that may affect the data preparation, indexing, searching, and rendering of your finding aids. Some of them are:

  • [fc_char.html Character Encoding issues]
  • [fc_ids Attribute ids must be unique within the entire collection ]
  • If you use attribute ids and corresponding targets within your EADs preparedocs.pl may need to be modified.
  • <eadid> should be less than about 20 characters in length
  • UTF-8 Byte Order Marks (BOM) should be removed from EADs prior to concatenation
  • XML processing instructions should be removed from EADs prior to concatenation
  • Multiline DOCTYPE declarations are currently not properly handled by the data prep scripts
  • If your DOCTYPE declaration contains entitys, you need to modify the appropriate *inp files accordingly
  • Out-of-the-box <dao> handling may need to be modified for your needs
  • If your <unititle> element precedes your <origination> element in the top level <did>, you will have to modify the maintitle fabricated region query in xxx.extra.srch
  • If you have encoded <unitdate>s as siblings of <unittitle>s, you may have to modify the appropriate XSL templates
  • If you do not use a <frontmatter> element, you will have to make modifications to various files to provide an appropriate "Title Page" region based on the <eadheader>
  • If your encoding practices for <biohist> differ from the Bentley's, you may need to make changes in findaidclass.cfg or create a subclass of FindaidClass and override FindaidClass:: GetBioghistTocHead, and/or change the appropriate XSL files.
  • If you want <relatedmaterial>,<separatedmaterial> to show up in the table of contents (TOC) on the left hand side of the Finding Aids, you may have to modify findaidclass.cfg and make other modifications to the code. This also applies if there are other sections of the finding aid not listed in the out-of-the-box findaidclass.cfg %gSectHeadsHash.
  • If you want the middleware to use the <head> element for labeling sections instead of the default hard-coded values in findaidclass.cfg, you may need to make changes to the XSL and possibly modify other files.

Data Preparation

For today, we are going to be working with some texts that are already in Findaid Class. We will be building them into a collection we are going to call workshopfa.

This documentation will make use of the concept of the $[../overview/dirstructure.html DLXSROOT], which is the place at which your DLXS directory structure starts. We generally use /l1/, but for the workshop, we each have our own $DLXSROOT in the form of /l1/workshop/userX/dlxs/. To check your $DLXSROOT, type the following commands at the command prompt:

cd $DLXSROOT
pwd

The prep directory under $DLXSROOT is the space for you to take your encoded finding aids and "package them up" for use with the DLXS middleware. Create your basic directory $DLXSROOT/prep/w/workshopfa and its data subdirectory with the following command:

mkdir -p $DLXSROOT/prep/w/workshopfa/data

Move into the prep directory with the following command:

cd $DLXSROOT/prep/w/workshopfa

This will be your staging area for all the things you will be doing to your texts, and ultimately to your collection. At present, all it contains is the data subdirectory you created a moment ago. We will be populating it further over the course of the next two days. Unlike the contents of other collection-specific directories, everything in prep should be ultimately expendable in the production environment.

Copy the necessary files into your data directory with the following commands:

cp $DLXSROOT/prep/s/samplefa/data/*.xml $DLXSROOT/prep/w/workshopfa/data/.

We'll also need a few files to get us started working. They will need to be copied over as well, and also have paths adapted and collection identifiers changed. Follow these commands:

cp $DLXSROOT/prep/s/samplefa/validateeach.csh $DLXSROOT/prep/w/workshopfa/. cp $DLXSROOT/prep/s/samplefa/samplefa.xml.inp $DLXSROOT/prep/w/workshopfa/workshopfa.xml.inp cp $DLXSROOT/prep/s/samplefa/samplefa.text.inp $DLXSROOT/prep/w/workshopfa/workshopfa.text.inp mkdir -p $DLXSROOT/obj/w/workshopfa mkdir -p $DLXSROOT/bin/w/workshopfa cp $DLXSROOT/bin/s/samplefa/preparedocs.pl $DLXSROOT/bin/w/workshopfa/. cp $DLXSROOT/bin/s/samplefa/Makefile $DLXSROOT/bin/w/workshopfa/Makefile

Now you'll need to edit these files to ensure that the paths match your $DLXSROOT and that the collection name is workshopfa instead of samplefa.

STOP!! Make sure you edit the files before going to the next steps!!

Make sure you change these files:

  • $DLXSROOT/prep/w/workshopfa/validateeach.csh
  • $DLXSROOT/bin/w/workshopfa/Makefile (see below for details)

You can run this command to check to see if you forgot to change samplefa to workshopfa: 

grep "samplefa" $DLXSROOT/bin/w/workshopfa/* $DLXSROOT/prep/w/workshopfa/* |grep -v "#"

With the ready-to-go ead2002 encoded finding aids files in the data directory, we are ready to begin the preparation process. This will include:

  1. [#DataPrepStep1 validating the files individually] against the EAD 2002 DTD
  2. [#DataPrepStep2 concatenating the files into one larger XML file]
  3. [#DataPrepStep3 validating the concatenated file] against the dlxsead2002 DTD
  4. [#DataPrepStep4 "normalizing" the concatenated file.]
  5. [#DataPrepStep5 validating the normalized concatenated file against the dlxsead2002 DTD]

These steps are generally handled via the Makefile in $DLXSROOT/bin/s/samplefa which we have copied to $DLXSROOT/bin/w/workshopfa. To see the Makefile and how it is used, [makefile.html click here].

Make sure you changed your copy of the Makefile to reflect

/w/workshopfa instead of /s/samplefa. You will want to change lines 2 and 3 accordingly 


   1  
   2  NAMEPREFIX = samplefa
   3  FIRSTLETTERSUBDIR = s

Tip: Be sure not to add any space after the workshopfa or w. The Makefile ignores space immediately before and after the equals sign but treats all other space as part of the string. I you accidentally put a space after the FIRSTLETTERSUBDIR = s , you will get an error like "[validateeach] Error 127" If you look closely at the first line of what the Makefile reported to standard output (see below) you will see that instead of running the command: 

/l1/workshop/tburtonw/dlxs/prep/w/workshopfa/validateeach.csh

which just calls the validateeach c-shell script

it tried to run a directory name: "/l1/workshop/tburtonw/dlxs/prep/w" with the argument "/workshopfa/validateeach.csh" which does not make sense 

 % make validateeach
/l1/workshop/tburtonw/dlxs/prep/w /workshopfa/validateeach.csh
make: execvp: /l1/workshop/tburtonw/dlxs/prep/w: Permission denied
make: [validateeach] Error 127 (ignored)

Further note on editing the Makefile: If you modify or write your own Make targets, you need to make sure that a real "tab" starts each command line rather than spaces. The easiest way to check for these kinds of errors is to use "cat -vet Makefile" to show all spaces, tabs and newlines.

If you are doing this at your home institution you will also want to make sure you change $DLXSROOT, and the locations of the various binaries to match your installation. We will not need to do this for the workshop.

These changes do not apply for the workshop

  • Change $DLXSROOT /l1/dev/userxx to your $DLXSROOT on every line that uses it
  • Change XPATBINDIR = /l/local/bin/ to the location of the xpat binary in your installation
  • Change the location of the osx binary from 
OSX = /l/local/bin/osx
to the location in your installation
  • Change the location of the osgmlnorm binary from 
OSGMLNORM = /l/local/bin/osgmlnorm
to the location in your installation

Tip: oxs and osgmlnorm are installed as part of the OpenSP package. If you are using linux, make sure that the OpenSP package for your version of linux is installed and make sure the paths above are changed to match your installation. If you are using Solaris you will have to install (and possibly compile) OpenSP. You may also need to make sure the $LD_LIBRARY_PATH environment variable is set so that the OpenSP programs can find the required libraries. For troubleshooting such problems the unix ldd utility is invaluble. [../troubleshooting/tools.html Information on OpenSP]

Step 1: Validating the files individually against the EAD 2002 DTD

cd $DLXSROOT/bin/w/workshopfa make validateeach The Makefile runs the following command: % /l1/workshop/userXX/dlxs/prep/w/workshopfa/validateeach.csh

What's happening: The makefile is running the c-shell script [validateeach.html validateeach.sh] in the prep directory. The script creates a temporary file without the public DOCTYPE declaration, runs onsgmls on each of the resulting XML files in the data subdirectory to make sure they conform with the EAD 2002 DTD. If validation errors occur, error files will be in the data subdirectory with the same name as the finding aids file but with an extension of .err. If there are validation errors, fix the problems in the source XML files and re-run.

Check the error files by running the following commands

ls -l $DLXSROOT/prep/w/workshopfa/data/*err if there are any *err files, you can look at them with the following command: 
 less  $DLXSROOT/prep/w/workshopfa/data/*err

There are not likely to be any errors with the workshopfa data, but tell the instructor if there are.

Step 2: Concatentating the files into one larger XML file (and running some preprocessing commands)

cd $DLXSROOT/bin/w/workshopfa make prepdocs 
The Makefile runs the following command:
$DLXSROOT/bin/w/workshopfa/preparedocs.pl $DLXSROOT/prep/w/workshopfa/data $DLXSROOT/obj/w/workshopfa/workshopfa.xml $DLXSROOT/prep/w/workshopfa/logfile.txt
This runs the preparedocs.pl script on all the files in the specified data directory and writes the output to the workshopfa.xml file in the appropriate /obj subdirectory. It also outputs a logfile to the /prep directory:

The Perl script does two sets of things:

  1. Concatenates all the files
  2. Runs a number of preprocessing steps on all the files

Concatenating the files

The script finds all XML files in the data subdirectory,and then strips off and xml declaration and doctype declaration from each file before concatenating them together. It also wraps the concatenated EADs in a <COLL> tag . The end result looks like: 


<COLL>
<ead><eadheader><eadid>1</eadid>...</eadheader>... content</ead>
<ead><eadheader><eadid>2</eadid>...</eadheader>... content</ead>
<ead><eadheader><eadid>3</eadid>...</eadheader>... content</ead>
</COLL>

WARNING! If are extra characters or some other problem with the part of the program that strips out the xml declaration and the docytype declearation the file will end up like: 


<COLL>
baddata<ead><eadheader><eadid>1</eadid>...</eadheader>... content</ead>
baddata<ead><eadheader><eadid>2</eadid>...</eadheader>... content</ead>
baddata<ead><eadheader><eadid>3</eadid>...</eadheader>... content</ead>
</COLL>

This will cause the document to be invalid since the dlxsead2002.dtd does not allow anything between the closing tag of one </ead> and the opening tag of the next one <ead>

Some of the possible causes of such a problem are:

  • UTF-8 Byte Order Marks at the beginning of the file
  • DOCTYPE declaration on more than one line
  • XML processing instructions

Preprocessing steps

The perl program also does some preprocessing on all the files. These steps are customized to the needs of the Bentley. You should look at the perl code and modify it so it is appropriate for your encoding practices.

The preprocessing steps are:

  • finds all id attributes and prepends a number to them
  • adds a prefix string "dao-bhl" to all DAO links (You probably will want to change this)
  • removes empty persname, corpname, and famname elements

The output of the combined concatenation and preprocessing steps will be the one collection named xml file which is deposited into the obj subdirectory.

If your collections need to be transformed in any way, or if you do not want the transformations to take place (the DAO changes, for example), edit preparedocs.pl file to effect the changes. Some changes you may want to make include:

  • Changing the algorithm used to make id attibute unique. For example if your encoding practices use id attributes and targets, the out-of-the-box algorithm will remove the relationship between the attributes and targets. One possible modification might be to modify the algorithm to prepend the eadid or filename to all id and target attributes.
  • Modifying the program to read a list of files or list of eadids so that the files are concatenated in a particular order. The default sort order for search results is in occurance order, which translates to the order in which the eads are concatenated. If you write a script which looks at the eads for some element that you want to sort by and then outputs a list of filenames sorted by that order, you could then pass that file to a modified preparedocs.pl so it would concatenate the files in the order listed.

Step 3: Validating the concatenated file against the dlxsead2002 DTD

make validate The Makefile runs the following command: onsgmls -wxml -s -f $DLXSROOT/prep/w/workshopfa/workshopfa.errors $DLXSROOT/misc/sgml/xml.dcl $DLXSROOT/prep/w/workshopfa/workshopfa.xml.inp $DLXSROOT/obj/w/workshopfa/workshopfa.xml

This runs the onsgmls command against the concatenated file using the dlxs2002dtd, and writes any errors to the workshopfa.errors file in the appropriate subdirectory in $DLXSROOT/prep/c/collection.. [validate.html More details]

Note that we are running this using workshopfa.xml.inp not workshop.text.inp. The workshopfa.xml.inp file points to $DLXSROOT/misc/sgml/dlxsead2002.ead which is the dlxsead2002 DTD. The dlxsead2002 DTDis exactly the same as the EAD2002 DTD, but adds a wrapping element, <COLL>, to be able to combine more than one ead element, more than one finding aid, into one file. The larger file will be indexed with XPAT tomorrow. It is, of course, a good idea to validate the file now before going further.

Check for errors by looking for the file $DLXSROOT/prep/w/workshopfa/workshopfa.errors which will be present and contain messages about what caused the file to be considered invalid if there are errors.

If you see errors at this point (assuming there were no errors during the validateeach step) is that there was a problem with the preparedocs.pl processing. Some common causes of problems are:

  • The DOCTYPE declaration did not get completely removed. (The current scripts don't always remove multiline DOCTYPE declearations)
  • There was a UTF-8 Byte Order Mark at the begginning of one or more of the concatenated files

Run the following command

ls -l $DLXSROOT/prep/w/workshopfa/workshopfa.errors

If there is a workshopfa.errors file then run the following command to look at the errors reported

less $DLXSROOT/prep/w/workshopfa/workshopfa.errors $ less $DLXSROOT/prep/w/workshopfa/workshopfa.errors
onsgmls:/l1/dev/tburtonw/misc/sgml/xml.dcl:1:W: SGML declaration was not implied
The above error can be ignored, but if you see any other errors STOP! You need to determine the cause of the problem, fix it, and rerun the steps until there are no errors from make validate. If you continue with the next steps in the process with an invalid xml document, the errors will compound and it will be very difficult to trace the cause of the problem. Note: To avoid seeing this error add the "-w no-explicit-sgml-decl" flag to the Makefile on line 83. Change line 83 of the Makefile from
onsgmls -wxml -s -f $(PREPDIR)$(NAMEPREFIX).errors $(XMLDECL) $(XMLDOCTYPE) $(XMLFILE)
to 
onsgmls -wxml -w no-explicit-sgml-decl -s -f $(PREPDIR)$(NAMEPREFIX).errors $(XMLDECL) $(XMLDOCTYPE) $(XMLFILE)

This will be fixed in the next release of DLXS Findaid Class.

Step 4: Normalizing the concatenated file

make norm The Makefile runs a series of copy statements and two main commands: 1.) /l/local/bin/osgmlnorm -f $DLXSROOT/prep/s/samplefa/samplefa.errors $DLXSROOT/misc/sgml/xml.dcl $DLXSROOT$DLXSROOT/prep/s/samplefa/samplefa.xml.inp $DLXSROOT/obj/s/samplefa/samplefa.xml.prenorm > /l1/dev/tburtonw/obj/s/samplefa/samplefa.xml.postnorm 2.) /l/local/bin/osx -bUTF-8 -xlower -xempty -xno-nl-in-tag -f /l1/dev/tburtonw/prep/s/samplefa/samplefa.errors /l1/dev/tburtonw/misc/sgml/xml.dcl /l1/dev/tburtonw/prep/s/samplefa/samplefa.xml.inp /l1/dev/tburtonw/obj/s/samplefa/samplefa.xml.postnorm > /l1/dev/tburtonw/obj/s/samplefa/samplefa.xml.postnorm.osx

These commands ensure that your collection data is normalized. What this means is that any attributes are put in the order in which they were defined in the DTD. Even though your collection data is XML and attribute order should be irrelevant (according to the XML specification), due to a bug in one of the supporting libraries used by xmlrgn (part of the indexing software), attributes must appear in the order that they are definded in the DTD. If you have "out-of-order" attributes and don't run make norm, you will get "invalid endpoints" errors during the make post step.

Step one, which normalizes the document writes its errors to $DLXSROOT/prep/s/samplefa/samplefa.errors. Be sure to check this file.

Step 2, which runs osx to convert the normalized document back into XML produces lots of error messages which are written to standard output. These are caused because we are using an XML DTD (the EAD 2002 DTD) and osx is using it to validate against the SGML document created by the osgmlnorm step. These are the only errors which may generally be ignored. However, if the next recommended step, which is to run "make validate" again reveals an invalid document, you may want to rerun osx and look at the errors for clues. (Only do this if you are sure that the problem is not being caused by XML processing instructions in the documents as explained below)

Step 5: Validating the normalized file against the dlxsead2002 DTD

make validate

We run this step again to make sure that the normalization process did not produce an invalid document. This is necessary because under some circumstances the "make norm" step can result in invalid XML. One known cause of this is the presense of XML processing instructions. For example: "<?Pub Caret1?>". Although XML processing instructions are supposed to be ignored by any XML application that does not understand them, the problem is that when we use sgmlnorm and osx, which are SGML tools, they end up munging the output XML. The recommended workaround is to add a preprocessing step to remove any XML processing instructions from your EADs before you run "make prepdocs", or to include some code in preparedocs.pl that will strip out XML priocessing instructions prior to concatenating the EADs.

More Documentation

Findaid Class Index Building with XPAT

[#Top go to table of contents]

In this section the workshopfa XML will be indexed with the XPAT search engine, preparing it for use with the DLXS middleware.

Set Up Directories and Files for XPAT Indexing

First, we need to create the rest of the directories in the workshopfa environment with the following commands:

mkdir -p $DLXSROOT/idx/w/workshopfa

The bin directory we created yesterday holds any scripts or tools used for the collection specifically; obj ( created earlier) holds the "object" or XML file for the collection, and idx holds the XPAT indexes. Now we need to finish populating the directories.


cp $DLXSROOT/prep/s/samplefa/samplefa.blank.dd $DLXSROOT/prep/w/workshopfa/workshopfa.blank.dd
cp $DLXSROOT/prep/s/samplefa/samplefa.extra.srch $DLXSROOT/prep/w/workshopfa/workshopfa.extra.srch

Each of these files need to be edited to reflect the new collection name and the paths to your particular directories. This will be true when you use these at your home institution as well, even if you use the same directory architecture as we do, because they will always need to reflect the unique name of each collection. Failure to change even one file can result in puzzling errors, because the scripts are working, just not necessarily in the directories you are looking at.

grep -l "samplefa" $DLXSROOT/prep/w/workshopfa/*

will check for changing s/samplefa to w/workshopfa. If you are at the workshop that should be all you need. However if you are doing this at your home institution you need to replace "/l1/" by whatever $DLXSROOT is on your server. If you don't have an /l1 directory on your server (which is very likely if you are not here using a DLPS machine) you can check with:

grep -l "l1" $DLXSROOT/prep/w/workshopfa/*

Build the XPAT Index

Everything is now set up to build the XPAT index. The Makefile in the bin directory contains the commands necessary to build the index, and can be executed easily.

To create an index for use with the Findaid Class interface, you will need to index the words in the collection, then index the XML (the structural metadata, if you will), and then finally "fabricate" structures based on a combination of elements (for example, defining who the "main author" of a finding aid is, without adding a <mainauthor> tag around the appropriate <author> in the eadheader element). The following commands can be used to make the index:

make singledd indexes words for texts that have been concatenated into on large file for a collection. This is the recommended process.

make xml indexes the XML structure by reading the DTD. Validates as it indexes.

make post builds and indexes fabricated regions based on the XPAT queries stored in the workshopfa.extra.srch file.

cd $DLXSROOT/bin/w/workshopfa make singledd 
cp /l1/workshop/test02/dlxs/prep/w/workshopfa/workshopfa.blank.dd
	/l1/workshop/test02/dlxs/idx/w/workshopfa/workshopfa.dd
/l/local/xpat/bin/xpatbld -m 256m -D /l1/workshop/test02/dlxs/idx/w/workshopfa/workshopfa.dd
cp /l1/workshop/test02/dlxs/idx/w/workshopfa/workshopfa.dd
	/l1/workshop/test02/dlxs/prep/w/workshopfa/workshopfa.presgml.dd

make xml

cp /l1/workshop/test02/dlxs/prep/w/workshopfa/workshopfa.presgml.dd
	/l1/workshop/test02/dlxs/idx/w/workshopfa/workshopfa.dd
/l/local/xpat/bin/xmlrgn -D /l1/workshop/test02/dlxs/idx/w/workshopfa/workshopfa.dd
	/l1/workshop/test02/dlxs/misc/sgml/xml.dcl
	/l1/workshop/test02/dlxs/prep/w/workshopfa/workshopfa.inp
	/l1/workshop/test02/dlxs/obj/w/workshopfa/workshopfa.xml

cp /l1/workshop/test02/dlxs/idx/w/workshopfa/workshopfa.dd
	/l1/workshop/test02/dlxs/idx/w/workshopfa/workshopfa.prepost.dd

make post

cp /l1/workshop/test02/dlxs/prep/w/workshopfa/workshopfa.prepost.dd
	/l1/workshop/test02/dlxs/idx/w/workshopfa/workshopfa.dd
touch /l1/workshop/test02/dlxs/idx/w/workshopfa/workshopfa.init
/l/local/xpat/bin/xpat -q /l1/workshop/test02/dlxs/idx/w/workshopfa/workshopfa.dd
	< /l1/workshop/test02/dlxs/prep/w/workshopfa/workshopfa.extra.srch
	| /l1/workshop/test02/dlxs/bin/t/text/output.dd.frag.pl
	/l1/workshop/test02/dlxs/idx/w/workshopfa/
	> /l1/workshop/test02/dlxs/prep/w/workshopfa/workshopfa.extra.dd
/l1/workshop/test02/dlxs/bin/t/text/inc.extra.dd.pl
	/l1/workshop/test02/dlxs/prep/w/workshopfa/workshopfa.extra.dd
	/l1/workshop/test02/dlxs/idx/w/workshopfa/workshopfa.dd

If you get an "invalid endpoints" message from "make post", the most likely cause is XML processing instructions or some other corruption. The second "make validate" step should have caught these. Other possible causes of errors during the "make post" step include syntax errors in workshopfa.extra.srch, or the absense of a particular region that is listed in the *.extra.srch file but not present in your collection. For example if you do not have any <corpname> elements in any of the EADs in your collection and you are using the out-of-the-box samplefa.extra.srch, you will see an error message when xpat tries to index the mainauthor region using this rule:

((region "persname" + region "corpname" + region "famname" + region "name") within (region "origination" within ( re gion "did" within (region "archdesc")))); {exportfile "/l1/workshop/user11/dlxs/idx/s/samplefa/mainauthor.rgn"}; exp ort; ~sync "mainauthor";

The easiest solution is to modify *extra.srch to match the characteristics of your collection. An alternative is to include a "dummy" EAD that contains all the elements that you expect in your collection with no content

Testing the index

At this point it is a good idea to do some testing of the newly created index. Invoke xpat with the following commands 

xpatu $DLXSROOT/idx/w/workshopfa/workshopfa.dd

Try searching for some likely regions. Its a good idea to test some of the fabricated regions. Here are a few sample queries:

>> region "ead"
  1: 3 matches

>> region "eadheader"
  2: 3 matches

>> region "mainauthor"
  3: 3 matches

>> region "maintitle"
  4: 3 matches

>> region "admininfo"
  5: 3 matches

Fabricated Regions in FindaidClass

The make post step and the testing steps above leads us into a discussion of the use of fabricated regions in FindaidClass. uses the workshopfa.extra.srch file to add to the XPAT index.

"Fabricated" is a term we use to describe what are essentially virtual regions in an XPat indexed text. See a [../xpat/fabregions.html basic description of what a fabricated region is and how they are created].

In Finding Aids, we use fabricated regions for certain uninteresting regions simply so that some code can be shared. For example, the fabricated region "main" is set to refer to <ead> in FindaidClass with: 

(region ead); {exportfile "/l1/idx/b/bhlead/main.rgn"}; export; ~sync "main";

whereas in TextClass "main" can refer to <TEXT>. Therfore, both FindaidClass and TextClass can share the Perl code, in a higher level subclass, that creates searches for "main".

Other fabricated regions are used for searching such as the maintitle and mainauthor regions.

The majority of the fabricated regions for Findaid Class are used for the creation and display of the left hand table of contents in the "outline" view. The findaidclass.cfg file contains a hash called %gSectHeadsHash which is normally loaded into FindaidClass.pm's tocheads hash in the FindaidClass::_initialize method. The elements of the hash and the corresponding fabiricated regions are used to create the table of contents and to output the XML for the corresponding section of the EAD when one of the TOC links is clicked on by a user. The fabricated regions are used so XPAT can have binary indexes ready to use for fast retrieval of these EAD sections.

Some of the more interesting regions extracted from the samplefa.extra.srch file are listed below.

One of these regions is the add. This used to be <ADD> in the EAD 1.0 DTD, but now, is created based on the ead2002 DTD's <descgrp> tag which contains a type attribute of add.

A number of issues related to varying encoding practices can be resolved by the appropriate edits to the *.extra.srch file. (Although some of them may require changes to other files as well)

  • If your <unititle> element precedes your <origination> element in the top level <did>, you will have to modify the "maintitle" fabricated region query in *.extra.srch
  • If you do not use a <frontmatter> element, you will have to make modifications to various files including modifying *.extra.srch to provide an appropriate "Title Page" region based on the <eadheader>
  • If your encoding practices for <biohist> differ from the Bentley's, you may need to make changes in the <bioghist> fabricated region although changes to other files may be suffient. The changes might include: modifying findaidclass.cfg or creating a subclass of FindaidClass and override FindaidClass:: GetBioghistTocHead, and/or changing the appropriate XSL files.
  • If you want sections of the finding aid that are not completely within a well-defined element such as <relatedmaterial>or <separatedmaterial> to show up in the table of contents, you may have to create a fabricated region using the appropriate xpat query and then modify findaidclass.cfg and make other modifications to the code.
(region ead); {exportfile "/l1/workshop/user11/dlxs/idx/s/samplefa/main.rgn"}; export; ~sync "main"; ## (((region "<c01".."</did>" + region "<c02".."</did>" + region "<c03".."</did>" + region "<c04".."</did>" + region "<c05".."</did>" + region "<c06".."</did>" + region "<c07".."</did>" + region "<c08".."</did>" + region "<c09".."</did>") not incl ("level=file" + "level=item")) incl "level="); {exportfile "/l1/workshop/user11/dlxs/idx/s/samplefa/c0xhead.rgn"}; export; ~sync "c0xhead"; ## ((region "<origination".."</unittitle>") within ((region did within region archdesc) not within region dsc)); {exportfile "/l1/workshop/user11/dlxs/idx/s/samplefa/maintitle.rgn"}; export; ~sync "maintitle"; ## ((region "persname" + region "corpname" + region "famname" + region "name") within (region "origination" within ( region "did" within (region "archdesc")))); {exportfile "/l1/workshop/user11/dlxs/idx/s/samplefa/mainauthor.rgn"}; export; ~sync "mainauthor"; ## (region "abstract" within ((region did within region archdesc) not within region "c01")); {exportfile "/l1/workshop/user11/dlxs/idx/s/samplefa/mainabstract.rgn"}; export; ~sync "mainabstract"; ## ((region unitdate incl "encodinganalog=245$f") within ((region did within region archdesc) not within region dsc)); {exportfile "/l1/workshop/user11/dlxs/idx/s/samplefa/colldate.rgn"}; export; ~sync "colldate"; ## (region dsc); {exportfile "/l1/workshop/user11/dlxs/idx/s/samplefa/contentslist.rgn"}; export; ~sync "contentslist"; ## ########## admininfo ######## admininfot = (region "descgrp-T" incl (region "A-type" incl "admin")); {exportfile "/l1/workshop/user11/dlxs/idx/s/samplefa/admininfo-t.rgn"}; export; ~sync "admininfo-t"; ## ## ########## add ###### addt = (region "descgrp-T" incl (region "A-type" incl "add")); {exportfile "/l1/workshop/user11/dlxs/idx/s/samplefa/add-t.rgn"}; export; ~sync "add-t"; ## ########## frontmatter/titlepage ######## frontmattert = region "frontmatter-T"; {exportfile "/l1/workshop/user11/dlxs/idx/s/samplefa/frontmatter-t.rgn"}; export; ~sync "frontmatter-t"; ## # frontmatter itself not needed as fabricated region since it exists # as a regular xml region ## ## ########## bioghist ######## bioghist = ((region "bioghist" within region "archdesc") not within region "dsc"); {exportfile "/l1/workshop/user11/dlxs/idx/s/samplefa/bioghist.rgn"}; export; ~sync "bioghist"; ##bioghisthead = ((region "<bioghist" .. "</head>" within region "archdesc") not within region "dsc"); {exportfile "/l1/workshop/user11/dlxs/idx/s/samplefa/bioghisthead.rgn"}; export; ~sync "bioghisthead"; ## ((region did within region archdesc) not within region dsc); {exportfile "/l1/workshop/user11/dlxs/idx/s/samplefa/summaryinfo.rgn"}; export; ~sync "summaryinfo";; ## ## ############################# (region "subject" + region "corpname" + region "famname" + region "name" + region "persname" + region "geogname"); {exportfile "/l1/workshop/user11/dlxs/idx/s/samplefa/subjects.rgn"}; export; ~sync "subjects"; (region "corpname" + region "famname" + region "name" + region "persname"); {exportfile "/l1/workshop/user11/dlxs/idx/s/samplefa/names.rgn"}; export; ~sync "names"; #(region "odd-T" ^ (region odd not within region dsc)); {exportfile "/l1/workshop/user11/dlxs/idx/s/samplefa/odd-t.rgn"}; export; ~sync "odd-t";

See a [bhl.extra.srch.txt full listing of the extra.srch file of the Bentley Historical Library's finding aids].

More Documentation

Findaid Class Collection to Web

[#Top go to table of contents]

These are the final steps in deploying an Findaid Class collection online. Here the Collection Manager will be used to review the Collection Database entry for workshopfa . The Collection Manager will also be used to check the Group Database. Finally, we need to work with the collection map and the set up the collection's web directory.

Review the Collection Database Entry with CollMgr

Each collection has a record in the collection database that holds collection specific configurations for the middleware. CollMgr (Collection Manager) is a web based interface to the collection database that provides functionality for editing each collection's record. Collections can be checked-out for editing, checked-in for testing, and released to production. In general, a new collection needs to have a CollMgr record created from scratch before the middleware can be used. If you are starting with the samplefa collmgr as a model make sure to change references from s/samplefa to w/workshopfa or whatever you are using for your collection name.

More Documentation

Review the Groups Database Entry with CollMgr

Another function of CollMgr allows the grouping of collections for cross-collection searching. Any number of collection groups may be created for Findaid Class. Findaid Class supports a group with the groupid "all". It is not a requirement that all collections be in this group, though that's the basic idea. Groups are created and modified using CollMgr.

http://username.ws.umdl.umich.edu/cgi/c/collmgr/collmgr

We won't be doing anything with groups;

Make Collection Map

Collection mapper files exist to identify the regions and operators used by the middleware when interacting with the search forms. Each collection will need one, but most collections can use a fairly standard map file, such as the one in the samplefa collection. The map files for all Findaid Class collections are stored in $DLXSROOT/misc/f/findaid/maps

Map files take language that is used in the forms and translates it into language for the cgi and for XPAT. For example, if you want your users to be able to search within names, you would need to add a mapping for how you want it to appear in the search interface (case is important, as is pluralization!), how the cgi variable would be set (usually all caps, and not stepping on an existing variable), and how XPAT will identify and retrieve this natively (in XPAT search language).

The first part of the map file is operator mapping, for the form, the cgi, and XPAT. The second part is for region mapping, as in the example above.

cd $DLXSROOT/misc/f/findaid/maps
cp samplefa.map workshopfa.map

You might note that some of the fields that are defined in the map file correspond to some of the [#FabRegions fabricated regions].

More Documentation

Set Up the Collection's Web Directory

Each collection may have a web directory with custom Cascading Style Sheets, interface templates, graphics, and javascript. The default is for a collection to use the web templates at $DLXSROOT/web/f/findaid. Of course, collection specific templates and other files can be placed in a collection specific web directory, and it is necessary if you have any customization at all. DLXS Middleware uses [../ui/index.html#fallback fallback] to find HTML related templates, chunks, graphics, js and css files.

For a minimal collection, you will want two files: index.html and FindaidClass-specific.css.

mkdir -p $DLXSROOT/web/w/workshopfa cp $DLXSROOT/web/s/samplefa/index.html $DLXSROOT/web/w/workshopfa/index.html cp $DLXSROOT/web/s/samplefa/findaidclass-specific.css $DLXSROOT/web/w/workshopfa/findaidclass-specific.css

As always, we'll need to change the collection name and paths. You might want to change the look radically, if your HTML skills are up to it.

Note that the browse link on the index.html page is hard-coded to go to the samplefa hard-coded browse.html page. You may want to change this to point to a dynamic browse page (see below). The url for the dynamic browse page is ".../cgi/f/findaid/findaid-idx?c=workshopfa;page=browse".

If you would prefer a dynamic home page, you can copy and modify the home.xml and home.xsl files from $DLXSROOT/web/f/findaid/. Note that they are currently set up to be the home page for all finding aids collections, so you will have to do some considerable editing. However they contain a number of PIs that you may find useful. In order to have these pages actually be used by DLXS, they have to be present in your $DLXSROOT/web/w/workshopfa/ directory and there can't be an index.html page in that directory. The easiest thing to do, if you have an existing index.html page is to rename it to "index.html.foobar" or something.

Create a browse page 

See the documentation: http://www.dlxs.org/docs/13/collmeta/browse.html

Try It Out

http://username.ws.umdl.umich.edu/cgi/f/findaid/findaid-idx

Linking from Finding Aids Using the ID Resolver

[#Top go to table of contents]

How do you do this?

Findaid Class is coded so that if there is an href attribute to the <dao> element, it will check to see if it contains the string "http". If it does, FindaidClass will not us ID Resolver, but will create a link based on the content of the href attribute of the <dao>. If there is no "http" string in the href attribute, FindaidClass assumes that the href attribute is actully an id and will look up that id in in the idresolver and build a link if it finds the ID in the IDRESOLVER table. The method FilterAllDaos_XML in $DLXSROOT/cgi/f/findaid/FindaidClass.pm can be overridden per collection if different behavior is needed.

If you decide to use this feature, you will want to modify the preprocessing script preparedocs.pl which out-of-the-box inserts the string 'dao-bhl-' after the href. Below is an example of a Bentley <dao> where the id number is 91153-1. 

<dao linktype="simple" href="91153-1" show="new" actuate="onrequest">
         <daodesc>
          <p>[view selected images]</p>
         </daodesc>
        </dao>

The preparedocs.pl program would change this to: 

<dao linktype="simple" href="dao-bhl-91153-1" show="new" actuate="onrequest">
         <daodesc>
          <p>[view selected images]</p>
         </daodesc>
        </dao>

The ID resolver would look up the id "dao-bhl-91153-1" and replace it with the appropriate URL.

ID Resolver Data Transformation and Deployment

The ID Resolver is a CGI that takes as input a unique identifier and returns a URI. It is used, for example, by Harper's Weekly to link the text pages in Text Class middleware to the image pages in the Image Class middleware, and vice versa.

Plug something like the following in to your web browser and you should get something back. If you choose to test middleware on a development machine that uses the id resolver, make sure that the middleware on that machine is calling the resolver on the machine with the data, and not the resolver on the production server.

Information on how to set up the ID resolver

Retrieved from "http://webservices.itcs.umich.edu/mediawiki/dlxs14/index.php/DLXS_Workshop_Materials"
Personal tools