Mounting a Text Class Collection

From DLXS Documentation

(Difference between revisions)
Jump to: navigation, search
Current revision (11:32, 4 November 2010) (edit) (undo)
 
(17 intermediate revisions not shown.)
Line 3: Line 3:
==[[Text Class Overview]]==
==[[Text Class Overview]]==
-
==[[Examples of Text Class Implementations and Practices]]==
+
==Examples of Text Class Implementations and Practices==
[http://moa.umdl.umich.edu Making of America], [http://qoud.lib.umich.edu/a/ampo20/ Twentieth Century American Poetry].
[http://moa.umdl.umich.edu Making of America], [http://qoud.lib.umich.edu/a/ampo20/ Twentieth Century American Poetry].
Line 18: Line 18:
This section describes the the Text Class DTD and mechanisms and programs used by DLXS for accessing and displaying images that are defined by the FIGURE element in TextClass document markup.
This section describes the the Text Class DTD and mechanisms and programs used by DLXS for accessing and displaying images that are defined by the FIGURE element in TextClass document markup.
-
 
-
 
==[[Working with Text Class Data and Directories]]==
==[[Working with Text Class Data and Directories]]==
-
= Setting up directories =
 
-
You will need to identify directories where you plan to store your source files, your converted and concatenated Text Class XML file, your index file (approximately 75% of the size of your SGML source), your "region" files and other information such as data dictionaries, and files you use to prepare your data. We recommend you use the following structure:
+
==[[Data Preparation and Conversion: Unicode, XML, and Normalization]]==
-
* Store specialized scripts for your collection and its Makefile in <tt>$DLXSROOT/bin/c/collid/</tt>where <tt>$DLXSROOT</tt> is the "tree" where you install all DLXS components, <tt>c</tt> is the first letter of the name of the collection you are indexing, and <tt>collid</tt> is the collection ID of the collection you are indexing. For example, if your collection ID is "moa" and your DLXSROOT is "/l1", you will place the Makefile in <tt>/l1/bin/m/moa/</tt>, e.g., <tt>/l1/bin/m/moa/Makefile</tt>. See [[directory conventions]] for more information.
+
==[[Working with Fabricated Regions]]==
-
* Store your source texts and any DTDs, doctype, and files for preparing your data in <tt>$DLXSROOT/prep/c/collid/</tt>. Unlike the contents of other directories, everything in prep should be ultimately expendable in the production environment.
+
-
* Store the finalized, concatenated Text Class XML file for your text collection in <tt>$DLXSROOT/obj/c/collid/</tt> , e.g., <tt>/l1/obj/m/moa/moa.xml</tt>.
+
-
* Store index, region, data dictionary, and init files in <tt>$DLXSROOT/idx/c/collid/</tt>, e.g., <tt>/l1/idx/m/moa/moa.idx</tt>. See the [[XPAT documentation]] for more on these types of files.
+
-
The files that are located in <tt>$DLXSROOT/bin/s/sampletc_utf8</tt> and<tt>$DLXSROOT/prep/s/sampletc_utf8</tt> should be copied into your collection directories and used to index your collection. The following files may need to be editted so that the #! points to your location of perl:
+
==[[Building the Text Class Index]]==
-
* $DLXSROOT/bin/t/text/isolat128bit.pl
+
==[[Mounting the Text Class Collection Online]]==
-
* $DLXSROOT/bin/t/text/output.dd.frag.pl
+
-
* $DLXSROOT/bin/t/text/inc.extra.dd.pl
+
-
The following files will need to be edited to reflect your collection names and paths:
+
==[[Adding Text Class Search Functionality]]==
-
 
+
-
* $DLXSROOT/bin/s/sampletc_utf8/Makefile
+
-
* $DLXSROOT/prep/s/sampletc_utf8/sampletc_utf8.blank.dd
+
-
* $DLXSROOT/prep/s/sampletc_utf8/sampletc_utf8.extra.srch
+
-
* $DLXSROOT/prep/s/sampletc_utf8/sampletc_utf8.inp
+
-
 
+
-
 
+
-
==[[Data Preparation and Conversion]]==
+
-
== Data Conversion: Unicode, XML, and Normalization ==
+
-
 
+
-
To make the most of Text Class and Finding Aids Class in DLXS Release 13, you will want to convert or otherwise handle the character entities, numeric entities, or Latin 1 8-bit characters that have been the staples of SGML (and XML, despite the default encoding of UTF-8) for so long. This is separate from the conversion of Text Class materials to the Text Class DTD; even with finding aids that are already in XML, you will probably need to do some testing of character encodings, conversion of these encodings to UTF-8, normalization, and conversion of SGML to XML (strange but true).
+
-
 
+
-
----
+
-
 
+
-
=== '''Determining the Character Encodings Present in Your Data''' ===
+
-
 
+
-
There are a number of possibilities you may encounter:
+
-
 
+
-
# Plain ASCII (aka the Basic Latin block)
+
-
# Character entity references (ISO and otherwise)
+
-
# Numeric character references (decimal and/or hexadecimal)
+
-
# Latin 1 characters
+
-
# UTF-8 characters
+
-
 
+
-
You may very well find a mixture of 1, 2, 3, and 4 or even 2, 3, and 5 in the wild, simply because many encoders are not clear on what they should be doing with special characters. One hopes you will not encounter a document with a mixture of Latin 1 and UTF-8 characters, although it's possible that misidentified files could end up concatenated together and create such a mess.
+
-
 
+
-
There are a number of tools you can use to identify what you have before you.
+
-
 
+
-
; findentities.pl
+
-
: A perl script written by Phil that is part of the DLXS package, it prints the names and frequencies of the entities (CERs and NCRs) it encounters. Fairly quick, regardless of the size of the file. Can be run on more than one file at once, which is handy if you have a batch of texts.
+
-
; xpatutf8check
+
-
: Another perl script written by Phil, it exists to answer the question, "Will xpatu index this?" It will report the line number of the first non-UTF character it encounters when it has failed and it runs very quickly, so it's great as a first step in checking your material, but it is not authoritative enough to identify all of the problems you may have.
+
-
; jhove
+
-
: The JSTOR/Harvard Object Validation Environment has a UTF-8 module that reports whether your document is or is not valid UTF-8, and which Unicode blocks are contained in the document. Can be slow checking large documents, but very informative. Available at http://hul.harvard.edu/jhove/ and invoked with
+
-
jhove -c /l/local/jhove/conf/jhove.conf -m utf8-hul file.xml
+
-
; utf8chars
+
-
: Yet another perl script written by Phil, it identifies the characters used in a document and the Unicode blocks to which they belong. It assumes your document is UTF-8 and will report each instance (by line number) where a non-UTF character is encountered. Because it is identifying and counting each character in a document, it is rather slow, but very very useful. Runs on one file at a time and prints to standard out, but can be invoked through a <tt>foreach</tt> to check many files in one command.
+
-
 
+
-
----
+
-
 
+
-
=== '''Converting Those Character Encodings to UTF-8''' ===
+
-
 
+
-
If you have a mixed bag of encodings and entities in your documents, there's a definite order in which you want to approach the conversion task, to avoid having a mixture of Latin 1 and UTF-8 in one document at any point in the transformation.
+
-
 
+
-
# First, if you have Latin 1 characters like ârun iconv, part of the Gnu C library, to convert files from one encoding to another.
+
-
iconv -f iso88591 -t utf8 oldfile &gt; newfile
+
-
# Next, convert character entity references like &amp;acirc; using isocer2utf8, a perl script written by Phil to convert character entity references to UTF-8 characters. Although it references ISO in the name, it's been expanded to handle all the CERs we've encountered, including TEI Greek and the Chadwyck-Healey custom entities.
+
-
/l1/bin/t/text/isocer2utf8 oldfile &gt; newfile
+
-
# Finally, if you have numeric character references like &amp;#226; or &amp;#xE2;, run ncr2utf8, also written by Phil, to convert decimal and hexadecimal entities to UTF-8 characters.
+
-
/l1/bin/t/text/ncr2utf8 oldfile &gt; newfile
+
-
 
+
-
This would be a good point to run findentities.pl again to see what (if anything) you have left, and to re-validate using jhove or utf8chars to ensure that you have done no harm.
+
-
 
+
-
----
+
-
 
+
-
=== '''Test Driving the Tools''' ===
+
-
 
+
-
In the directory /l1/workshop-samples/sooty, you will find four sample files that we'll examine for character encoding and then convert to UTF-8: findaid1.xml, findaid2.xml, text1.xml, and text2.sgm. Copy these to your own directory -- they are completely expendable and won't serve a purpose in tomorrow's Text Class implementation. They are merely illustrative of all the possibilities you might encounter and how you may want to handle them.
+
-
 
+
-
First, we'll look at which character or numeric entities, if any, are used in these documents.
+
-
 
+
-
<blockquote>
+
-
 
+
-
foreach file (findaid*)<br />echo $file <br />$DLXSROOT/bin/t/text/findEntities.pl $file <br />end
+
-
 
+
-
foreach file (text*)<br />echo $file <br />$DLXSROOT/bin/t/text/findEntities.pl $file <br />end
+
-
 
+
-
</blockquote>
+
-
 
+
-
We have some CERs and NCRs to deal with, aside from the five XML-approved entities (&amp;amp;, &amp;gt;, &amp;lt;, &amp;apos;, and &amp;quot;). So, we know we'll be needing both <tt>isocer2utf</tt> and <tt>ncr2utf</tt>. Next, we'll see what characters we have (Latin 1? UTF-8? something else?). We'll run through all three tools, just for the sake of completeness, in the order of speediness and terseness.
+
-
 
+
-
<blockquote>
+
-
 
+
-
foreach file (findaid*)<br />echo $file <br />xpatutf8check $file <br />end
+
-
 
+
-
foreach file (text*)<br />echo $file <br />xpatutf8check  $file <br />end
+
-
 
+
-
</blockquote>
+
-
 
+
-
We now know that both the text files are either UTF-8 or plain ASCII (because of the output of these two tests), but there's a problem with one of the finding aids. jhove will tell us a bit more about our materials. You'll note we don't need to echo the filename as that's part of the jhove report. You'll also notice jhove is not so fast.
+
-
 
+
-
<blockquote>
+
-
 
+
-
foreach file (findaid*)<br />jhove -c /l/local/jhove/conf/jhove.conf -m utf8-hul  $file <br />end
+
-
 
+
-
foreach file (text*)<br />jhove -c /l/local/jhove/conf/jhove.conf -m utf8-hul  $file <br />end
+
-
 
+
-
</blockquote>
+
-
 
+
-
So, the second file in each set is plain ASCII (the Basic Latin block) with entities, the first finding aid is not UTF-8, and the first text file is. Let's look a bit more at the two non-ASCII files with the slowest and most verbose tool of them all. We're not doing a foreach this time, but we wouldn't need to echo the filename either, as it is again part of what the tool reports.
+
-
 
+
-
<blockquote>
+
-
 
+
-
utf8chars findaid1.xml
+
-
 
+
-
utf8chars text1.xml
+
-
 
+
-
</blockquote>
+
-
 
+
-
We can see the exact problem with findaid1.xml -- there's an 8-bit Latin 1 e acute before Boekeloo on line 37. We also can see all the UTF-8 characters in text1.xml -- this is the sort of information that is useful when time comes to map characters and encodings in the xpatu data dictionary.
+
-
 
+
-
Now that we know which items need what character treatments, we'll convert them. text1.xml is completely fine, so we'll leave it as is. findaid1.xml has the one Latin 1 character, so we'll use iconv to convert it to UTF-8. It had no entities of any kind, so we'll be done with it after this step.
+
-
 
+
-
<blockquote>
+
-
 
+
-
iconv -f iso88591 -t utf8 findaid1.xml &gt; findaid1.xml.utf
+
-
 
+
-
</blockquote>
+
-
 
+
-
Next, findaid2.xml had numeric character references. It is fine and can be indexed as-is, but users would need to search for the hexadecimal string in the midst of words ( &amp;#xE9; for é, for example). So, we'll use ncr2utf to convert the entities into the characters. WARNING! &amp;#x26; is the ampersand (as is &amp;#38;) -- if you convert these to the character, you will run into validation problems down the road, as bare ampersands are not permitted in XML. Don't get carried away!
+
-
 
+
-
<blockquote>
+
-
 
+
-
ncr2utf8 findaid2.xml &gt; findaid2.xml.utf
+
-
 
+
-
</blockquote>
+
-
 
+
-
Finally, text2.sgm has ISO character entity references (from Latin 1, Greek, and Publishing) that need to be converted to UTF-8 with isocer2utf.
+
-
 
+
-
<blockquote>
+
-
 
+
-
isocer2utf8 text2.sgm &gt; text2.sgm.utf
+
-
 
+
-
</blockquote>
+
-
 
+
-
Note that the ampersand CER was not processed. This is perfectly correct.
+
-
 
+
-
=== ''More Documentation'' ===
+
-
 
+
-
* [http://www.dlxs.org/docs/13/class/unicode.html DLXS Unicode Data Preparation and Online Presentation Issues]
+
-
* [http://www.zvon.org/other/charSearch/PHP/search.php Zvon Character Search]
+
-
* [http://www.alanwood.net/unicode/ Unicode and Multilingual Support in HTML, Fonts, Web Browsers and Other Applications]
+
-
 
+
-
----
+
-
 
+
-
=== '''Normalization and Converting SGML to XML''' ===
+
-
 
+
-
Many of you may be in a position where you'll want to be converting your SGML files to XML. Many of you will be fortunate enough to have files already in XML -- say, finding aids in EAD 2002. However, these will have to be normalized, too, to avoid problems with xpatu and xmlrgn down the road by ensuring that all the attributes are in the same order as specified in the DTD. Because of known but uncorrected problems in the normalization tools, you will end up with SGML and will need to convert that to XML.
+
-
 
+
-
Because the file we want to work with is now UTF-8, we need to set some environment variables for the tools from the sp package to let them know this is UTF-8. It doesn't matter that you've set your puTTy window to UTF-8, if you are using osx, osgmlnorm, or onsgmls, you must set your environment properly.
+
-
 
+
-
<blockquote>
+
-
 
+
-
setenv SP_CHARSET_FIXED YES
+
-
 
+
-
setenv SP_ENCODING utf-8
+
-
 
+
-
</blockquote>
+
-
 
+
-
First we normalize, invoking a declaration to handle the non-SGML UTF-8 characters without claiming that the material itself is XML.
+
-
 
+
-
<blockquote>
+
-
 
+
-
osgmlnorm $DLXSROOT/misc/sgml/xmlentities.dcl sample.inp text2.sgm.utf &gt; text2.sgm.norm
+
-
 
+
-
</blockquote>
+
-
 
+
-
Now I'll test the output with one of the UTF-8 tools to make sure that it's come through unscathed, and with findEntities.pl to see what has happened with the remaining XML-friendly entities, and it's fine. Now to convert our SGML to XML using osx.
+
-
 
+
-
<blockquote>
+
-
 
+
-
osx -x no-nl-in-tag -x empty -E 500 -f errors $DLXSROOT/misc/sgml/xmlentities.dcl sample.inp  text2.sgm.norm &gt; text2.xml
+
-
 
+
-
</blockquote>
+
-
 
+
-
Again I'll test the output with one of the UTF-8 tools to make sure that it's come through unscathed, and with findEntities.pl to see what has happened with the remaining XML-friendly entities, and again it's fine.
+
-
 
+
-
Just for fun, we'll normalize the files already in XML, just to show that things get changes from XML to SGML against their will.
+
-
 
+
-
<blockquote>
+
-
 
+
-
osgmlnorm $DLXSROOT/misc/sgml/xml.dcl $DLXSROOT/prep/s/sampletc_utf8/sampletc_utf8.text.inp  text1.xml &gt; text1.xml.norm
+
-
 
+
-
</blockquote><blockquote>
+
-
 
+
-
osx -x no-nl-in-tag -x empty -E 5000 -f error $DLXSROOT/misc/sgml/xml.dcl $DLXSROOT/prep/s/sampletc_utf8/sampletc_utf8.text.inp  text1.xml.norm &gt; text1.xml.norm.xml
+
-
 
+
-
</blockquote>
+
-
 
+
-
=== ''More Documentation'' ===
+
-
 
+
-
* [http://www.dlxs.org/docs/13/class/unicode.html DLXS Unicode Data Preparation and Online Presentation Issues]
+
-
* [http://linuxcommand.org/man_pages/osgmlnorm1.html osgmlnorm Documentation]
+
-
* [http://linuxcommand.org/man_pages/osx1.html osx Documentation]
+
-
* [http://www.jclark.com/sp/ James Clark's Original sp Documentation]
+
-
 
+
-
----
+
-
 
+
-
=== '''Using Unconverted Collections with Current Middleware''' ===
+
-
 
+
-
Recognizing that there may be situations when you do not wish to migrate a collection to XML and UTF-8 immediately, there are mechanisms built into Release 13 to allow the middleware to handle SGML-style empty elements (aka singletons), Latin 1 characters, and character entity references. How do you make this happen? In DlpsUtils.pm, there is a subroutine called Sgml2XmlFilter that has a hard-coded list of empty elements (&lt;PB&gt;, &lt;LB&gt;, &lt;CAESURA&gt;, etc.) that are converted upon discovery to XML-style (&lt;PB/&gt;, &lt;LB/&gt;, &lt;CAESURA/&gt;, etc.), and a feature that converts Latin 1 (ISO-8859-1) characters to UTF-8. This subroutine comes into play if the locale field in collmgr is not set to en_US.UTF-8 (locale used to be optional but is now required if you are using UTF-8 and xpatu). In order to declare your entities, you need to put a file called <tt>entitiesdoctype.chnk</tt> in the web directory for your collection, declaring the entities like so:
+
-
 
+
-
<blockquote>
+
-
 
+
-
&lt;!DOCTYPE Top [
+
-
&lt;!ENTITY Dstrok  "&amp;#x0110;"&gt;
+
-
&lt;!ENTITY Sacute  "&amp;#x015A;"&gt;
+
-
&lt;!ENTITY Scaron  "&amp;#352;"&gt;
+
-
&lt;!ENTITY Ubreve  "&amp;#x016C;"&gt;
+
-
&lt;!ENTITY Zdot    "&amp;#x017B;"&gt;
+
-
]&gt;
+
-
 
+
-
</blockquote>
+
-
 
+
-
That being the case, why would anyone ever bother to go through the trouble of converting their material? First off, the value of having UTF-8 is apparent if you have material that used more than one entity set (and even the lowliest collections have both an e acute and an em-dash in them somewhere). Now that &amp;mdash; is one character that can be mapped to a space in the data dictionary like other punctuation, phrases that were obscured in searches now turn up, and characters that we used to flatten (transforming ? to a, for example) can be displayed. Second, this facility comes at a cost. All of the material returned needs to be run through this filter, which will take some time. In a results list, the lag is negligable, but in larger sections of text, it could be noticeable. Finally, some confusion might arise when a user cuts and pastes material he received as a result and cannot retrieve it again, because the results and input are UTF-8 (which is the encoding of the search form) but the material being searched is not.
+
-
 
+
-
==[[Working with Fabricated Regions]]==
+
-
When you use XPAT in combination with xmlrgn or sgmlrgn and a DTD, or multirgn and a tags file, you are identifying the elements and attributes in the DTD or tags file as "regions," containers of content rather like fields in a database. These separate regions are built into the regions file (collid.rgn) and are identified in the data dictionary (collid.dd). This is what is happening when you are running sgmlrgn and/or xmlrgn.
+
-
 
+
-
However, sometimes the things you want to identify collectively aren't so handily identified as elements in the DTD. For example, suppose you want to search within specific features of a book, such as a chapter, that can occur at different heirarchical levels in different volumes. Also, the element isn't even called CHAPTER; it's a numbered division with a type attribute telling you that it's a chapter.
+
-
 
+
-
In order to fabricate a region containing all the divisions in books that are chapters, for example you can first find all the regions with a query:
+
-
 
+
-
(region DIV1 incl (region "DIV1-T" incl "type=chapter"))+ (region DIV2 incl (region "DIV2-T" incl "type=chapter"))
+
-
 
+
-
You could do a more complex search based on treating attributes as regions instead of text strings, which is functionally the same:
+
-
 
+
-
(region DIV1 incl (region "DIV1-T" incl (region "A-TYPE" incl chapter)))+ (region DIV2 incl (region "DIV2-T" incl (region "A-TYPE" incl chapter)))
+
-
 
+
-
Finally, once you have a query that produces the results you want, create a file for export, export it, and sync it:
+
-
 
+
-
{exportfile "$DLXSROOT/idx/c/collid/chapter.rgn"}
+
-
export
+
-
~sync "chapter"
+
-
 
+
-
Why fabricate regions? Why not just put these queries in the map file and call them chapters? While you could, it's probably worth your time to build these succinctly-named and precompiled regions; query errors are more easily identified in the index building than in the CGI, and XPAT searches can be simpler and quicker for terms within the prebuilt regions.
+
-
 
+
-
Fabricated regions within Text Class can be found in the extra.srch file for the sample collection at <tt>$DLXSROOT/prep/s/sampletc_utf8/sampletc_utf8.extra.srch</tt> . As with any other elements used in the interface for a given collection, fabricated regions used must also be represented in the collmgr entry and the map file for that collection.
+
-
 
+
-
= Fabricated regions required in Text Class =
+
-
 
+
-
* main
+
-
* mainheader
+
-
* maintitle
+
-
* div1head
+
-
 
+
-
= Fabricated regions commonly found in Text Class =
+
-
 
+
-
* mainauthor
+
-
* maindate page (for collections with page images)
+
-
* id (for collections with a number of different IDNO elements)
+
-
* divxhead (for collections nested below DIV1)
+
-
 
+
-
==[[Customizing Text Class]]==
+
-
This could provide other topics, but I think it would be a good place for a discussion of the pieces of text class that are customizable ranked by level of difficulty. Then you could link to the places in the doc that discuss specific methods for customizing different bits.
+
-
 
+
-
==[[Building the Index]]==
+
-
After you have followed all the steps to set up your directories and prepare your files, as found in the [[Text Class preparation documentation]], indexing the collection is fairly straightforward. To create an index for use with the Text Class interface, you will need to index the words in the collection, then index the XML (the structural metadata, if you will), and then finally "fabricate" structures based on a combination of elements (for example, defining what the "main entry" is, without adding a &lt;MAINENTRY&gt; tag around the appropriate &lt;AUTHOR&gt; or &lt;TITLE&gt; element). The following commands can be used to make the index, alone or in combination.
+
-
 
+
-
# Ensure that your collection XML is valid by using the <tt>make validate</tt> command in the Makefile stored at $DLXSROOT/bin/c/collid/Makefile
+
-
# '''make singledd''' indexes words for texts that have been concatenated into one large file for a collection. This is the recommended process, as a data dictionary built from a single concatenated file is faster for searching and more reliable than one built using multi-file system indexing. Use the <tt>make singledd</tt> command in the Makefile stored at $DLXSROOT/bin/c/collid/Makefile .
+
-
# '''make xml''' indexes the XML structure by reading the DTD, and validates as it indexes. It is slower than multiregion indexing (see [[XPAT documentation]] for more information) for this reason. However, it is necessary for collections that have nested elements of the same name (even when separated by an intervening element, such as a &lt;P&gt; within &lt;NOTE1&gt; that is itself within a &lt;P&gt;). Use the <tt>make xml</tt>command in the Makefile stored at $DLXSROOT/bin/c/collid/Makefile
+
-
# '''make post''' builds and indexes fabricated regions based on the XPAT queries stored in the $DLXSROOT/prep/c/collid/collid.extra.srch file. Because every collection is different, this file will need to be adapted after you have determined what you want to use as a "poem" for text (e.g., perhaps every DIV1 TYPE="sonnet" and DIV2 TYPE="poem" in the collection) and how many levels of division heads you have in your collection (e.g., at least one text is nested to DIV4, so you'll need to fabricate up to div4head). If the extra.srch file references elements not used in your text collection, you will see errors like <tt>Error found: &lt;Error&gt;syntax error before: ")&lt;/Error&gt; </tt> when you use the <tt>make post</tt> command in the Makefile stored at $DLXSROOT/bin/c/collid/Makefile. Remove unnecessary lines.
+
-
 
+
-
You have now built indexes and region files for your collection. You can test that things are properly indexed by issuing the command<tt>xpatu $DLXSROOT/idx/c/collid/collid.dd</tt> and doing searches, such as for a common word like <tt>the</tt> or an element that should appear such as <tt>region "main"</tt> or <tt>region "HEADER"</tt>. Strategically, it is good to test this from a directory other than the one you indexed in, to ensure that relative or absolute paths are resolving appropriately.
+
-
 
+
-
Everything is now set up to build the XPATu index. The Makefile in the bin directory contains the commands necessary to build the index, and can be executed easily.
+
-
 
+
-
<blockquote>
+
-
 
+
-
make dd
+
-
 
+
-
</blockquote><blockquote>
+
-
 
+
-
cp /l1/workshop/sooty/dlxs/prep/w/workshoptc/workshoptc.blank.dd /l1/workshop/sooty/dlxs/idx/w/workshoptc/workshoptc.dd
+
-
/l/local/bin/xpatbldu -m 256m -D /l1/workshop/sooty/dlxs/idx/w/workshoptc/workshoptc.dd
+
-
cp /l1/workshop/sooty/dlxs/idx/w/workshoptc/workshoptc.dd /l1/workshop/sooty/dlxs/prep/w/workshoptc/workshoptc.presgml.dd
+
-
 
+
-
</blockquote><blockquote>
+
-
 
+
-
make xml
+
-
 
+
-
</blockquote><blockquote>
+
-
 
+
-
cp /l1/workshop/sooty/dlxs/prep/w/workshoptc/workshoptc.presgml.dd /l1/workshop/sooty/dlxs/idx/w/workshoptc/workshoptc.dd
+
-
/l/local/bin/xmlrgn -D /l1/workshop/sooty/dlxs/idx/w/workshoptc/workshoptc.dd /l1/workshop/sooty/dlxs/misc/sgml/xml.dcl /l1/workshop/sooty/dlxs/prep/w/workshoptc/workshoptc.inp /l1/workshop/sooty/dlxs/obj/w/workshoptc/workshoptc.xml
+
-
/l/local/bin/xmlrgn:/l1/workshop/sooty/dlxs/misc/sgml/xml.dcl:1:W: SGML declaration was not implied
+
-
cp /l1/workshop/sooty/dlxs/idx/w/workshoptc/workshoptc.dd /l1/workshop/sooty/dlxs/prep/w/workshoptc/workshoptc.prepost.dd
+
-
 
+
-
</blockquote><blockquote>
+
-
 
+
-
make post
+
-
 
+
-
</blockquote><blockquote>
+
-
 
+
-
cp /l1/workshop/sooty/dlxs/prep/w/workshoptc/workshoptc.prepost.dd /l1/workshop/sooty/dlxs/idx/w/workshoptc/workshoptc.dd
+
-
touch /l1/workshop/sooty/dlxs/idx/w/workshoptc/workshoptc.init
+
-
/l/local/bin/xpatu -q /l1/workshop/sooty/dlxs/idx/w/workshoptc/workshoptc.dd &lt; /l1/workshop/sooty/dlxs/prep/w/workshoptc/workshoptc.extra.srch | /l1/workshop/sooty/dlxs/bin/t/text/output.dd.frag.pl /l1/workshop/sooty/dlxs/idx/w/workshoptc/ &gt; /l1/workshop/sooty/dlxs/prep/w/workshoptc/workshoptc.extra.dd
+
-
/l1/workshop/sooty/dlxs/bin/t/text/inc.extra.dd.pl /l1/workshop/sooty/dlxs/prep/w/workshoptc/workshoptc.extra.dd /l1/workshop/sooty/dlxs/idx/w/workshoptc/workshoptc.dd
+
-
 
+
-
</blockquote>
+
-
 
+
-
<span><font color="#FF0000">Sorting and browse building require that you have only '''one''' maintitle, mainauthor and maindate per text, so that you have one value on which to sort. Your extra.srch files may need to be changed in order to be more specific. If you do not, some sort operations will give you a <tt>sortkey</tt> assertion failure.</font> </span>
+
-
 
+
-
<span><font color="#FF0000">Some examples of more specific searches in your extra.srch are provided below. The first relies on identifying metadata that has been specified through the use of attributes; the second merely chooses the first occurrence as an indication that it is the "main" value.</font></span>
+
-
 
+
-
(((region TITLE incl "type=main") within region TITLESTMT) within region SOURCEDESC);
+
-
{exportfile "/l1/idx/e/example/maintitle.rgn"}; export; ~sync "maintitle";
+
-
 
+
-
(((region AUTHOR within (region "&lt;TITLESTMT".."&lt;/AUTHOR&gt;")) within (region
+
-
TITLESTMT within region FILEDESC)) not within (region SOURCEDESC)); {exportfile
+
-
"/l1/idx/e/example/mainauthor.rgn"}; export; ~sync "mainauthor";
+
-
 
+
-
==[[Mounting the Collection Online]]==
+
-
Text Class Collection to Web from workshop doc would go in here
+
==[[Working with Page Image Access Mechanisms in Text Class]]==
==[[Working with Page Image Access Mechanisms in Text Class]]==
-
This document describes the mechanisms and programs used by DLXS for accessing and viewing images of pages that correspond to pages in TextClass documents (and possibly other classes). It also explains the particular metadata requirements that exist for this functionality to be possible and shows a sample pageview.dat file. The pageview.dat mechanisms are now deprecated, but still useful for importing information into the Pageview table (see instructions below in Populating the Pageview Table). Also discussed is the mechanism for linking to ImageClass to view Pageviewer images in more detail.
 
-
 
-
* [#GeneralInfo General Information]
 
-
** [ivimageconversion.html Itemviewer Image Conversion]
 
-
* [#pb The Page Break (PB) Element and Page Image Metadata]
 
-
* [#pageviewTable Populating the Pageview Table]
 
-
* [#pageviewDat Creating pageview.dat Files]
 
-
* [#pageviewFig Linking from Pageviewer to ImageClass]
 
-
 
-
= General Information =
 
-
 
-
For collections where the middleware delivers page images rather than or in addition to the text content of the pages, the main mechanism for viewing the pages is the pageviewer-idx CGI program. In order to link from the XML text to the corresponding image, this CGI expects that page images are stored in directories based on (1) the DLXSROOT value, (2) the object directory recorded in the collection manager, and (3) the unique identifier assigned to the XML text and stored in the IDNO element, and that there are page break elements in the document referencing the images. In addition, there must be a Pageview table in the [../../collmeta/colldatabases.html dlxs metadata database] that should contain a column for each page image, listing the image file name, its sequence in the XML text, the page number (if any) specified on the page, the OCR confidence value (if available), and a three-letter code for any special features of the page (the default value for no special feature is UNS; see below for more information).
 
-
 
-
<span class="command">pageviewer-idx</span> connects to the database and retrieves the name and location of the page image file from the Pageview table. <span class="command">pageviewer-idx</span> then decides how to deliver the page. If the stored file format is different from the requested format as recorded in the collection manager (e.g., stored as tiff and requested as gif), a separate program, tif2web, is started to convert the image on the fly. ''For more information about how pageviewer-idx does its work in deciding how to deliver the page image, see [ivimageconversion.html Itemviewer Image Conversion ]. ''
 
-
 
-
= The Page Break Element and Page Image Metadata =
 
-
 
-
The PB tag in the XML data, representing a page break, has this form in Text Class:
 
-
 
-
&lt;PB REF="00000009.tif" SEQ="00000009" RES="600dpi" FMT="TIFF6.0" FTR="TPG" CNF="856" N=iiii"/&gt;
 
-
 
-
 
-
The attributes are:
 
-
 
-
* REF: file name of page image
 
-
* SEQ: the sequence number of the page in the series, from start to finish, of all the pages in the document.
 
-
* RES: the resolution of the page image.
 
-
* FMT: the file format of the page image.
 
-
* FTR: the feature of the page, given as a three letter code. Possible values are listed below.
 
-
* CNF: the confidence value of the OCR for the page, given by the OCR software.
 
-
* N: the page number, not as a sequence, but rather the number as printed on the page (e.g., 3, 96, ix, etc.). This may be left blank, but the attribute cannot be omitted.
 
-
 
-
The information in this PB tag allows the Text Class middleware to create a URL to call the pageviewer-idx program with the parameters necessary to retrieve and display the corresponding page image. pageviewer-idx uses the Pageview table of the dlxs metadata database to do so.
 
-
 
-
The following are examples of feature codes and their expanded definitions that have been used in various collections mounted by DLPS. You may redefine these or use other codes, but will need to add or change the values in the PageView.cfg file found in the Text Class cgi directory. To see an example of these codes in use in the Text Class interface, go to [http://www.hti.umich.edu/cgi/t/text/pageviewer-idx?c=moa;cc=moa;seq=00000019;idno=AGH8649.0001.001 this page from ''The Use of the Barometer''] in the Making of America and note the variety of features in the upper right hand pull down menu labeled "go to." If UNS is the sole feature recorded, no special features will be listed in this menu. Note that the FTR value becomes part of a lookup key in the langmap.en.xml to be referred to from the the pageviewer XSL.
 
-
 
-
Here is an example:
 
-
 
-
<blockquote>
 
-
 
-
 
-
        &lt;Lookup id="viewer"&gt;
 
-
          &lt;Item key="viewer.ftr.1stpg"&gt;First Page&lt;/Item&gt;
 
-
          &lt;Item key="viewer.ftr.ack"&gt;Acknowledgement&lt;/Item&gt;
 
-
          &lt;Item key="viewer.ftr.adv"&gt;Advertisement&lt;/Item&gt;
 
-
          &lt;Item key="viewer.ftr.app"&gt;Appendix&lt;/Item&gt;
 
-
          &lt;Item key="viewer.ftr.bib"&gt;Bibliography&lt;/Item&gt;
 
-
          &lt;Item key="viewer.ftr.blp"&gt;Blank Page&lt;/Item&gt;
 
-
          &lt;Item key="viewer.ftr.ctp"&gt;Cover Title Page&lt;/Item&gt;
 
-
          &lt;Item key="viewer.ftr.dig"&gt;Digest&lt;/Item&gt;
 
-
          &lt;Item key="viewer.ftr.err"&gt;Errata&lt;/Item&gt;
 
-
          &lt;Item key="viewer.ftr.fnt"&gt;Front Matter&lt;/Item&gt;
 
-
          &lt;Item key="viewer.ftr.his"&gt;History&lt;/Item&gt;
 
-
          &lt;Item key="viewer.ftr.ind"&gt;Comprehensive Index&lt;/Item&gt;
 
-
          &lt;Item key="viewer.ftr.loi"&gt;List of Illustrations&lt;/Item&gt;
 
-
          &lt;Item key="viewer.ftr.lot"&gt;List of Tables&lt;/Item&gt;
 
-
          &lt;Item key="viewer.ftr.map"&gt;Map&lt;/Item&gt;
 
-
          &lt;Item key="viewer.ftr.mis"&gt;Miscellaneous&lt;/Item&gt;
 
-
          &lt;Item key="viewer.ftr.mss"&gt;Manuscript&lt;/Item&gt;
 
-
          &lt;Item key="viewer.ftr.not"&gt;Notes&lt;/Item&gt;
 
-
          &lt;Item key="viewer.ftr.npn"&gt;[n/a]&lt;/Item&gt;
 
-
          &lt;Item key="viewer.ftr.ord"&gt;Ordinances&lt;/Item&gt;
 
-
          &lt;Item key="viewer.ftr.pni"&gt;Author or Name Index&lt;/Item&gt;
 
-
          &lt;Item key="viewer.ftr.pnt"&gt;Production Note&lt;/Item&gt;
 
-
          &lt;Item key="viewer.ftr.pre"&gt;Preface&lt;/Item&gt;
 
-
          &lt;Item key="viewer.ftr.ref"&gt;References&lt;/Item&gt;
 
-
          &lt;Item key="viewer.ftr.reg"&gt;Regulations&lt;/Item&gt;
 
-
          &lt;Item key="viewer.ftr.rul"&gt;Rules&lt;/Item&gt;
 
-
          &lt;Item key="viewer.ftr.spi"&gt;Special Index&lt;/Item&gt;
 
-
          &lt;Item key="viewer.ftr.sui"&gt;Subject Index&lt;/Item&gt;
 
-
          &lt;Item key="viewer.ftr.sup"&gt;Supplement&lt;/Item&gt;
 
-
          &lt;Item key="viewer.ftr.tab"&gt;Table&lt;/Item&gt;
 
-
          &lt;Item key="viewer.ftr.toc"&gt;Table of Contents&lt;/Item&gt;
 
-
          &lt;Item key="viewer.ftr.tpg"&gt;Title Page&lt;/Item&gt;
 
-
          &lt;Item key="viewer.ftr.uns"&gt;&lt;/Item&gt;
 
-
          &lt;Item key="viewer.ftr.ves"&gt;Volume End Sheets&lt;/Item&gt;
 
-
          &lt;Item key="viewer.ftr.vli"&gt;Volume List of Illus&lt;/Item&gt;
 
-
          &lt;Item key="viewer.ftr.voi"&gt;Volume Index&lt;/Item&gt;
 
-
          &lt;Item key="viewer.ftr.vpg"&gt;Various Pagination&lt;/Item&gt;
 
-
          &lt;Item key="viewer.ftr.vtp"&gt;Volume Title Page&lt;/Item&gt;
 
-
          &lt;Item key="viewer.ftr.vtv"&gt;Volume Title Page Verso&lt;/Item&gt;
 
-
          &lt;Item key="viewer.nopagenum"&gt;[n/a]&lt;/Item&gt;
 
-
        &lt;/Lookup&gt;
 
-
 
-
 
-
</blockquote>
 
-
 
-
= Populating the Pageview Table =
 
-
 
-
In DLXS releases prior to CD-ROM 8, the pageview.dat, a tab-delimited ASCII file used to locate page images associated with a text, was stored in the directory with the page images for a particular collection. If you have created pageview.dat files and would like to migrate them to the Pageview table, [../../data/pageview.html instructions can be found here.] Otherwise, metadata about page images for a collection should be entered directly into the Pageview table.
 
-
 
-
= Creating pageview.dat Files (For Information Only) =
 
-
 
-
The pageview.dat file for a particular XML text can be automatically generated provided that the metadata required is stored as attributes in the page break (PB) elements in the text. On the distribution CD-ROM, in the directory /l1/bin/t/text/, you will find a perl script named makepageviewdata.pl. When run with a directory path as its sole argument, it will work through the subdirectories, creating pageview.dat files for all files with a .sgm* extension. (For XML files, you will need to edit lines 27 and 51 to point the script to files with the extension .xml.) For example,
 
-
 
-
/l1/bin/t/text/makepageviewdata.pl /l1/obj/a/
 
-
 
-
will run through '''all''' the subdirectories below /l1/obj/a/ and report on the files it finds and work it is doing:
 
-
 
-
Working on xml files in directory: /l1/obj/a/j/l/ajl7777.0001.001
 
-
Working on file: /l1/obj/a/j/l/ajl7777.0001.001/ajl7777.0001.001.xml
 
-
Working on PB tag for sequence: 0001
 
-
Working on PB tag for sequence: 0002
 
-
Working on PB tag for sequence: 0003
 
-
Working on PB tag for sequence: 0004
 
-
 
-
Working with a document containing these four page break tags:
 
-
 
-
&lt;PB REF="00000001.tif" SEQ="00000001" RES="600dpi" FMT="TIFF6.0" FTR="TPG" CNF="852" N="1"/&gt;
 
-
&lt;PB REF="00000002.tif" SEQ="00000002" RES="600dpi" FMT="TIFF6.0" FTR="UNSPEC" CNF="100" N="2"/&gt;
 
-
&lt;PB REF="00000003.tif" SEQ="00000003" RES="600dpi" FMT="TIFF6.0" FTR="UNSPEC" CNF="884" N="3"/&gt;
 
-
&lt;PB REF="00000004.tif" SEQ="00000004" RES="600dpi" FMT="TIFF6.0" FTR="UNSPEC" CNF="872" N="4"/&gt;
 
-
 
-
would result in a pageview.dat file that contained this data:
 
-
 
-
<nowiki>## File:        /l1/obj/b/a/b/bab3633.0001.001/pageview.dat
 
-
## Created:    Mon Aug  6 11:32:55 EDT 2001
 
-
##
 
-
#filename      seq      pagenum confid  feature
 
-
00000001.tif    00000001        00000001        852    TPG
 
-
00000002.tif    00000002        00000002        100    UNS
 
-
00000003.tif    00000003        00000003        884    UNS
 
-
00000004.tif    00000004        00000004        872    UNS</nowiki>
 
-
 
-
= Linking from Pageviewer to ImageClass =
 
-
 
-
Pageviewer can handle both bitonal and continuous tone image formats. The output of a page capture process might be a bitonal scan of a page containing a continuous tone plate and a second continuous scan of the plate. Or the entire page might be captured as a continuous tone image. So there are two cases.
 
-
 
-
* In the bitonal with continuous plate case, Pageviewer can display the bitonal page image and provide a link into ImageClass for the continuous tone plate. The link may be simple text or a thumbnail of the continuous tone plate (if thumbnail images are available).
 
-
* If the entire page is a continuous tone image it may still be desirable to link to ImageClass for enhanced viewing opportunities provided there.
 
-
 
-
Pageviewer can provide a links to the image if it is placed under ImageCLass management. The bitonal page with thumbnail link is shown in the following illustration.
 
-
 
-
<center>[[Image:pageviewerlink.png]]</center>
 
-
 
-
== Detailed Description of the Mechanism ==
 
-
 
-
=== ImageClass ===
 
-
 
-
The continuous tone images and optional thumb images are placed under ImageClass management ase described in [../image/imageloading.html Image Class Image Loading for Online Delivery].
 
-
 
-
=== DLXS idresolver CGI and DLXS database idresolver database table ===
 
-
 
-
The DLXS IdResolver CGI script is delivered in <tt>DLXSROOT/cgi/i/idresolver/idresolver</tt>. It depends on a table in the DLXS database called <tt>idresolver</tt>. Check <tt>DLXSROOT/misc/db/MySQL-dump.sql</tt> for the schema. A row consists of an ID and a URL to ImageClass for that ID.
 
-
 
-
The default format of the ID is <tt>ic-''collid-idno''-''ENTITY''-[th|rec]</tt> where '''th''' is for the thumb and '''''rec''''' is for the full sized image. '''''ENTITY''''' is the value of the '''ENTITY''' attribute in the &lt;FIGURE&gt; tag in the XML. Simple bitonal page collections use the &lt;PB&gt; tag to indicate pages. If the page has a continuous tone plate or is itself a continuous tone image for which ImageClass viewing is desirable, the &lt;FIGURE&gt; tag is used in addition to the &lt;PB&gt; tag.
 
-
 
-
The virtual host running the idresolver CGI is assigned to <tt>$LibGlobals::gIdResolverHost</tt> via an install-time question. It can be changed later by editing DLXSROOT/lib/LibGlobals.cfg.
 
-
 
-
Population of the <tt>idresolver</tt> table is accomplished by a variety of local mechanisms which will vary depending upon your data preparation processes.
 
-
 
-
=== DLXS databse PageviewFig table ===
 
-
 
-
The final component to this mechanism is the DLXS database <tt>PageviewFig</tt> table.
 
-
This table is consulted by Pageviewer to determine whether to present links to ImageClass when a given page is displayed. Check <tt>DLXSROOT/misc/db/MySQL-dump.sql</tt> for the schema. A row consists of (1) the <tt>idno</tt> of the item, (2) the page sequence number (<tt>seq</tt>) from the '''SEQ''' attribute of the &lt;PB&gt; tag for the page being viewed and (3) the value of the '''ENTITY''' attribute from the &lt;FIGURE&gt; tag accompanying the &lt;PB&gt; tag in the XML.
+
==[[Working with the Text Class User Interface]]==
-
The <tt>PageviewFig</tt> table can be populated by the <tt>DLXSROOT/bin/t/text/processPageviewFig.pl</tt> Perl script. It accepts a single command-line option with values of '''-a''' to insert rows and '''-d''' to delete rows. The script will prompt for DLXSROOT and for a path relative to DLXSROOT where the XPAT Data Dictionary for the collection resides, e.g. <tt>/idx/c/coll/coll.dd</tt>. The script will perform XPAT queries and parse the '''ENTITY''' attribute value from the &lt;FIGURE&gt; tag and the '''SEQ''' attribute of the containing &lt;PB&gt; tag and the <tt>idno</tt> of the containing item. The result is inserted into the table. Deletion is accomplished by deleting all rows with <tt>idno</tt> values for the collection in question.
+
==[[Creating Text Class Wordwheels]]==
-
=== Putting it all together ===
+
==[[Newspaper Clip Image Access Mechanisms]]==
-
When Pageviewer displays the image for a given page ('''seq''') for a given item ('''idno''') in some collection it uses these keys to look up the corresponding '''ENTITY''' value in the PageviewFig table. It constructs two IDs from the '''ENTITY''' for lookup in the <tt>idresolver</tt> table. It then builds links using the URLs to ImageClass that come back in the query result. If the thumbnail image URL is empty it builds a text link instead of a thumb link.
+
==[[Workflows for delivering PDF files as page images through pageturner]]==
-
==[[Working with the User Interface]]==
+
==[[Picklist customization for serial articles]]==  
-
===[[Specifying Individual Collection Characteristics]]===
+
<span class="release_15">'''Release_15'''</span>
-
Could we re-label this topic with a friendlier title? Perhaps "Customizing Text Class UI Layout"or something.
+
-
===[[Text Class XML Template Files]]===
+
-
===[[Text Class Processing Instructions]]===
+
-
===[[Text Class Graphics Files]]===
+
==[[Text Class Troubleshooting]]==
==[[Text Class Troubleshooting]]==
 +
This section should include Text Class problems and solutions--user-contributed material is welcome!
[[#top|Top]]
[[#top|Top]]

Current revision

Main Page > Mounting Collections: Class-specific Steps > Mounting a Text Class Collection

Contents

[edit] Text Class Overview

[edit] Examples of Text Class Implementations and Practices

Making of America, Twentieth Century American Poetry.

The decision process for inclusion of content in the Text Class is roughly based on the following:

  • The text is a monograph or journal issue available in SGML or XML.
  • Each text has a unique ID.
  • The campus and/or world community would benefit from access to the information.

These are, at best, general guidelines for decision making based on the current state of the Text Class implementation at the University of Michigan.

[edit] Working with Text Class Markup

This section describes the the Text Class DTD and mechanisms and programs used by DLXS for accessing and displaying images that are defined by the FIGURE element in TextClass document markup.

[edit] Working with Text Class Data and Directories

[edit] Data Preparation and Conversion: Unicode, XML, and Normalization

[edit] Working with Fabricated Regions

[edit] Building the Text Class Index

[edit] Mounting the Text Class Collection Online

[edit] Adding Text Class Search Functionality

[edit] Working with Page Image Access Mechanisms in Text Class

[edit] Working with the Text Class User Interface

[edit] Creating Text Class Wordwheels

[edit] Newspaper Clip Image Access Mechanisms

[edit] Workflows for delivering PDF files as page images through pageturner

[edit] Picklist customization for serial articles

Release_15

[edit] Text Class Troubleshooting

This section should include Text Class problems and solutions--user-contributed material is welcome!


Top

Retrieved from "http://webservices.itcs.umich.edu/mediawiki/dlxs15/index.php/Mounting_a_Text_Class_Collection"
Personal tools