OAI Harvester

From DLXS Documentation

(Difference between revisions)
Jump to: navigation, search
(→Harvester (UMHarvester))
Current revision (13:33, 20 October 2008) (edit) (undo)
m (→Overview)
 
(28 intermediate revisions not shown.)
Line 1: Line 1:
 +
[[DLXS Wiki|Main Page]] > [[Ancillary Resources]] > OAI Harvester
 +
==Overview==
==Overview==
 +
This document details how to run the harvester used for harvesting OAI records from data providers.
-
<p>DLXS hosts the [http://www.oaister.org/ OAIster] project, as well as other OAI test portals ([http://quod.lib.umich.edu/m/mods/ MODS],
+
This tool, along with the [[OAI Provider]], is open source and available for download from [http://sourceforge.net/projects/umoaitoolkit Source Forge (UMich OAI Toolkit)] as well as included in DLXS release 14.
-
[http://quod.lib.umich.edu/a/aquifer/ Aquifer], [http://quod.lib.umich.edu/i/imls/ DLF]). This document details how to run the harvester used for gathering records we use in these portals, and how to transform the harvested records into [[Mounting a Bib Class Collection|Bibliographic Class]].</p>
+
-
For more information on OAI and the Protocol for Metadata Harvesting (OAI-PMH), see the [http://www.openarchives.org/pmh/ official site]. For best practices related to OAI, see the
+
Also available in the [http://sourceforge.net/projects/umoaitoolkit UMich OAI Toolkit] is a simple script (OaiList.pl) that can be used for harvesting OAI data. Details on how to use that can be found [[OaiList|here]].
-
[http://oai-best.comm.nsdl.org/cgi-bin/wiki.pl?TableOfContents Best Practices wiki].
+
-
 
+
-
 
+
-
To download just the harvester and transform engine, see the  [http://www.dlxs.org/products/release13_OAI.html OAI Tools Package].
+
==Harvester (UMHarvester)==
==Harvester (UMHarvester)==
-
<p><strong>N.B.</strong>: To use the harvester in your system, you may have to make changes to the Global Parameters located at the beginning of the UMHarvester script.</p>
 
-
<p>To start the harvester use <code>./UMHarvester</code> from within /$DLXSROOT/bin/o/oaister/scripts/</p>
+
'''N.B.''': To use the harvester in your system, you may have to make changes to the Global Parameters located at the beginning of the UMHarvester script.
-
<p>These flags let you perform harvesting:</p>
+
To start the harvester use <code>./UMHarvester</code> from within <code>$DLXSROOT/bin/o/oaister/scripts/</code>
-
<ul>
+
These flags let you perform harvesting:
-
<li><code>-i</code><br/> The id of the repository. This is pulled from /$DLXSROOT/bin/o/oaister/scripts/id_URL_table.txt, a text file that lists the id and the baseURL of repositories, e.g., cogprints=http://cogprints.ecs.soton.ac.uk/perl/oai2. (Use id_URL_table.sample.txt to get started.) You can run multiple harvests at the same time by separating ids with commas.</li>
+
-
<li><code>-v</code><br/> The verbs. You can specify:
+
*'''<code>-i</code>''': The id of the repository. This is pulled from $DLXSROOT/bin/o/oaister/scripts/id_URL_table.txt, a text file that lists the id and the baseURL of repositories, e.g., cogprints=http://cogprints.ecs.soton.ac.uk/perl/oai2. (Use id_URL_table.sample.txt to get started.) You can run multiple harvests at the same time by separating ids with commas.
-
<ul>
+
-
* <code>lr</code> = ListRecords
+
-
* <code>id</code> = Identify
+
-
* <code>ls</code> = ListSets
+
-
* <code>lf</code> = ListMetadataFormats
+
-
</ul>
+
-
When running ListRecords, if the folder that will contain the repository's record already exists, it will place the original folder in a backup place. Currently, this is set up as such:
+
-
<ul>
+
-
* Records are placed in <code>/$DLXSROOT/prep/h/harvester/[repository_id]</code>>
+
-
* Backup is in <code>/$DLXSROOT/prep/h/harvester_other/backup/[repository_id]</code>
+
-
</ul>
+
-
Logs of ListRecords output are placed in /$DLXSROOT/bin/o/oaister/scripts/log/. The active.log is overwritten for each (set of) harvests run. The [repository_id].log is appended with successful harvests or harvesting errors.</li>
+
-
<li><code>-s</code><br/> To harvest sets for a particular repository. Use the <code>setSpec</code> name for a particular set.</li>
+
*'''<code>-v</code>''': The verbs. You can specify:<br><code>lr</code> = ListRecords<br><code>id</code> = Identify<br><code>ls</code> = ListSets<br><code>lf</code> = ListMetadataFormats<br>When running ListRecords, if the folder that will contain the repository's record already exists, it will place the original folder in a backup place. Currently, this is set up as such:
 +
**Records are placed in <code>$DLXSROOT/prep/h/harvester/[repository_id]</code>
 +
**Backup is in <code>$DLXSROOT/prep/h/harvester_other/backup/[repository_id]</code>
 +
**Logs of ListRecords output are placed in <code>$DLXSROOT/bin/o/oaister/scripts/log/</code>. The active.log is overwritten for each (set of) harvests run. The <code>[repository_id].log</code> is appended with successful harvests or harvesting errors.
-
<li><code>-f</code><br/> To harvest records in the following metadata formats:
+
*'''<code>-s</code>''': To harvest sets for a particular repository. Use the <code>setSpec</code> name for a particular set.
-
<ul>
+
-
* mods
+
-
* oai_marc
+
-
* marc21
+
-
* marc21a
+
-
* marc21b
+
-
* marcxml
+
-
</ul>
+
-
The harvester will harvest in oai_dc format, unless a metadata format is specified.</li>
+
-
<li><code>-n</code><br/> To harvest records from the last harvest date for a repository (i.e., incremental harvesting). This flag checks the datestamp granularity from the Identify response and starts harvesting either the next second after the last harvest finished (e.g., last harvest finished 2005-03-15T10:45:46Z, incremental harvest starts at 2005-03-15T10:45:47Z) or the next day after the last harvest finished (e.g., last harvest finished 2005-03-15, incremental harvest starts at 205-03-16). The flag creates a tar.gz copy of the repository's records and puts that in the backup directory. As the incremental harvest runs, it checks the contents of the repository's directory and replaces those records that have been modified. If it finds no replacement for a harvested record, it places this in a directory specific to the incremental harvest date (e.g., cogprints1-1000_2007-03-28).</li>
+
*'''<code>-f</code>''': To harvest records in the following metadata formats:<br>mods<br>oai_marc<br>marc21<br>marc21a<br>marc21b<br>marcxml<br>The harvester will harvest in oai_dc format unless a metadata format is specified.
-
<li><code>-a</code><br/> To retry a harvest following a timeout. This flag will wait 5 minutes to retry a timeout on a particular resumptionToken. The limit is 10 retries for a particular harvest.
+
*'''<code>-n</code>''': To harvest records from the last harvest date for a repository (i.e., incremental harvesting). This flag checks the datestamp granularity from the Identify response and starts harvesting either the next second after the last harvest finished (e.g., last harvest finished 2005-03-15T10:45:46Z, incremental harvest starts at 2005-03-15T10:45:47Z) or the next day after the last harvest finished (e.g., last harvest finished 2005-03-15, incremental harvest starts at 205-03-16). The flag creates a tar.gz copy of the repository's records and puts that in the backup directory. As the incremental harvest runs, it checks the contents of the repository's directory and replaces those records that have been modified. If it finds no replacement for a harvested record, it places this in a directory specific to the incremental harvest date (e.g., cogprints1-1000_2007-03-28).
-
<li><i>Examples</i>:<br/>
+
*'''<code>-a</code>''': To retry a harvest following a timeout. This flag will wait 5 minutes to retry a timeout on a particular resumptionToken. The limit is 10 retries for a particular harvest.
-
<ul>
+
-
* <code>./UMHarvester -i auburn,epsilondiss,rdn -v ls</code>
+
-
* <code>./UMHarvester -i cogprints -v id</code>
+
-
* <code>./UMHarvester -i cogprints -v id</code>
+
-
* <code>./UMHarvester -i uiucimages -v lr -s ALA</code>
+
-
* <code>./UMHarvester -i lcoa1 -v lr -f mods</code>
+
-
* <code>./UMHarvester -i forex -v lr -n</code>
+
-
* <code>./UMHarvester -i CCSDthesis -v lr -a</code>
+
-
</ul>
+
-
<p>The Batch_UMHarvest file is used to run automated incremental harvests on repositories. See the /$DLXSROOT/bin/o/oaister/scripts/Batch_UMHarvest_sample file for an example.</p>
+
Examples:
-
<blockquote><font size="-1">
 
<pre>
<pre>
-
my @Monday =
+
./UMHarvester -i auburn,epsilondiss,rdn -v ls
-
(['uiucimages', 'ALA', 'oai_dc', 'dr'],
+
./UMHarvester -i cogprints -v id
-
);
+
./UMHarvester -i cogprints -v id
-
</pre></font></blockquote>
+
./UMHarvester -i uiucimages -v lr -s ALA
 +
./UMHarvester -i lcoa1 -v lr -f mods
 +
./UMHarvester -i forex -v lr -n
 +
./UMHarvester -i CCSDthesis -v lr -a
 +
</pre>
-
<p>Add your own repository id, set, metadata format, and run specification (<code>r</code> to run, <code>dr</code> to not run OAITransform) for each repository you wish to batch harvest. Batch_UMHarvest will perform an incremental harvest from the last time you harvested, based on the .log file for that repository id.</p>
+
The Batch_UMHarvest file is used to run automated incremental harvests on repositories. See the <code>$DLXSROOT/bin/o/oaister/scripts/Batch_UMHarvest_sample</code> file for an example.
-
<p>Rename Batch_UMHarvest_sample to Batch_UMHarvest to use. To start the Batch_UMHarvest run <br/>
+
<pre>
-
<code>./Batch_UMHarvest -d M &</code><br/>
+
my @Monday =
-
from within /$DLXSROOT/bin/o/oaister/scripts/. This will run all the repository ids within the "M" (or Monday) batch harvest group.</p>
+
(['uiucimages', 'ALA', 'oai_dc', 'dr', 's'],
 +
);
 +
</pre>
-
==Transform engine (OAITransform)==
+
Add your own repository id, set, metadata format, run specification (<code>r</code> to run, <code>dr</code> to not run OAITransform), and flag to skip HTML removal for each repository you wish to batch harvest. Batch_UMHarvest will perform an incremental harvest from the last time you harvested, based on the .log file for that repository id.
-
<p>OAITransform creates concatenated BibClass file of all oai_dc records, per repository. To start the transform tool use <code>./oaitransform/OAITransform [repository_id]</code> from within /$DLXSROOT/bin/o/oaister/oaitransform/</p>
+
Rename Batch_UMHarvest_sample to Batch_UMHarvest to use. To start the Batch_UMHarvest run
-
 
+
-
<p>Add the repository id you want to transform. This id is taken from repository_table.txt, which you will build using repository_table.sample.txt as your starting point.<br/>
+
-
e.g.,<code>./oaitransform/OAITransform celebration</code></p>
+
-
 
+
-
<p>The transform program will process your oai_dc harvested files, first by concatenating them into raw files and then by transforming them into BibClass files. The /$DLXSROOT/bin/o/oaister/oaitransform/oai-bibclass3.xsl file is used to perform the mapping from oai_dc to BibClass.</p>
+
-
 
+
-
<p>The repository report at the end of the transform will provide a number of statistics.</p>
+
-
 
+
-
<blockquote><font size="-1">
+
<pre>
<pre>
-
Repository Report: bristol
+
      ./Batch_UMHarvest -d M &
-
        records with URLs      = 818
+
</pre>
-
        records without URLs    = 5
+
-
        repository records      = 823
+
-
        success rate            = 99.39%
+
-
        ------------------------
+
-
        data conditioning msgs? = YES!
+
-
        deleted records (.del)  = 0
+
-
        normalization errors    = 2
+
-
        raw parse failures      = 0
+
-
</pre></font></blockquote>
+
-
 
+
-
<ul>
+
-
<li>records with URLs: OAIster is only interested in oai_dc records with a dc:identifier beginning with http or ftp, so the transform engines only transforms those records with those dc:identifiers.</li>
+
-
<li>records without URLs: The remainder of the records.</li>
+
-
<li>repository records: All the oai_dc records harvested.</li>
+
-
<li>data condtioning msgs: If there are character errors during transformation, these are fixed by OAITransform. See <a href="condition.html">an explanation of the data conditioning</a> that can be performed. To see character errors that have been fixed after a transformation, see /$DLXSROOT/bin/o/oaister/errors/utf8_status_log.txt</li>
+
-
<li>deleted records (.del): Not used unless you want to re-write the harvester to mark deleted records with a .del extension.</li>
+
-
<li>normalization errors: The transform tool uses the /$DLXSROOT/bin/o/oaister/oaitransform/normal_types.txt file to normalize the dc:type field values into five distinct BibClass TYPE values, i.e., text, image, audio, video, dataset. If the values in the dc:type fields can't be normalized because there are not mappings for them, these will be logged to an error file located at /$DLXSROOT/bin/o/oaister/errors/normalization_errors.txt</li>
+
-
<li>raw parse failures: If there are encoding errors which the transform tool cannot fix, these will be indicated during the transform.</li>
+
from within $DLXSROOT/bin/o/oaister/scripts/. This will run all the repository ids within the "M" (or Monday) batch harvest group.
-
</ul>
+
-
<p>For questions on how to transform MODS records, please contact Kat Hagedorn at khage at umich dot edu.</p>
+
[[#top|Top]]

Current revision

Main Page > Ancillary Resources > OAI Harvester

[edit] Overview

This document details how to run the harvester used for harvesting OAI records from data providers.

This tool, along with the OAI Provider, is open source and available for download from Source Forge (UMich OAI Toolkit) as well as included in DLXS release 14.

Also available in the UMich OAI Toolkit is a simple script (OaiList.pl) that can be used for harvesting OAI data. Details on how to use that can be found here.

[edit] Harvester (UMHarvester)

N.B.: To use the harvester in your system, you may have to make changes to the Global Parameters located at the beginning of the UMHarvester script.

To start the harvester use ./UMHarvester from within $DLXSROOT/bin/o/oaister/scripts/

These flags let you perform harvesting:

  • -i: The id of the repository. This is pulled from $DLXSROOT/bin/o/oaister/scripts/id_URL_table.txt, a text file that lists the id and the baseURL of repositories, e.g., cogprints=http://cogprints.ecs.soton.ac.uk/perl/oai2. (Use id_URL_table.sample.txt to get started.) You can run multiple harvests at the same time by separating ids with commas.
  • -v: The verbs. You can specify:
    lr = ListRecords
    id = Identify
    ls = ListSets
    lf = ListMetadataFormats
    When running ListRecords, if the folder that will contain the repository's record already exists, it will place the original folder in a backup place. Currently, this is set up as such:
    • Records are placed in $DLXSROOT/prep/h/harvester/[repository_id]
    • Backup is in $DLXSROOT/prep/h/harvester_other/backup/[repository_id]
    • Logs of ListRecords output are placed in $DLXSROOT/bin/o/oaister/scripts/log/. The active.log is overwritten for each (set of) harvests run. The [repository_id].log is appended with successful harvests or harvesting errors.
  • -s: To harvest sets for a particular repository. Use the setSpec name for a particular set.
  • -f: To harvest records in the following metadata formats:
    mods
    oai_marc
    marc21
    marc21a
    marc21b
    marcxml
    The harvester will harvest in oai_dc format unless a metadata format is specified.
  • -n: To harvest records from the last harvest date for a repository (i.e., incremental harvesting). This flag checks the datestamp granularity from the Identify response and starts harvesting either the next second after the last harvest finished (e.g., last harvest finished 2005-03-15T10:45:46Z, incremental harvest starts at 2005-03-15T10:45:47Z) or the next day after the last harvest finished (e.g., last harvest finished 2005-03-15, incremental harvest starts at 205-03-16). The flag creates a tar.gz copy of the repository's records and puts that in the backup directory. As the incremental harvest runs, it checks the contents of the repository's directory and replaces those records that have been modified. If it finds no replacement for a harvested record, it places this in a directory specific to the incremental harvest date (e.g., cogprints1-1000_2007-03-28).
  • -a: To retry a harvest following a timeout. This flag will wait 5 minutes to retry a timeout on a particular resumptionToken. The limit is 10 retries for a particular harvest.

Examples: 

	./UMHarvester -i auburn,epsilondiss,rdn -v ls
	./UMHarvester -i cogprints -v id
	./UMHarvester -i cogprints -v id
	./UMHarvester -i uiucimages -v lr -s ALA
	./UMHarvester -i lcoa1 -v lr -f mods
	./UMHarvester -i forex -v lr -n
	./UMHarvester -i CCSDthesis -v lr -a

The Batch_UMHarvest file is used to run automated incremental harvests on repositories. See the $DLXSROOT/bin/o/oaister/scripts/Batch_UMHarvest_sample file for an example. 

	my @Monday = 
	(['uiucimages', 'ALA', 'oai_dc', 'dr', 's'],
	);

Add your own repository id, set, metadata format, run specification (r to run, dr to not run OAITransform), and flag to skip HTML removal for each repository you wish to batch harvest. Batch_UMHarvest will perform an incremental harvest from the last time you harvested, based on the .log file for that repository id.

Rename Batch_UMHarvest_sample to Batch_UMHarvest to use. To start the Batch_UMHarvest run 

       ./Batch_UMHarvest -d M &

from within $DLXSROOT/bin/o/oaister/scripts/. This will run all the repository ids within the "M" (or Monday) batch harvest group.

Top

Retrieved from "http://webservices.itcs.umich.edu/mediawiki/dlxs15/index.php/OAI_Harvester"
Personal tools