From DLXS Documentation

Revision as of 14:47, 8 August 2008

Overview

This document explains the process for creating or patching a single DLXS Release.

The release process is conducted on the DLXS development server (dev-linux). The release process is largely governed by CVS. Files for release are tagged and the tagged files are exported to a staging area. The staging area contains both a tarred-gzipped form of the files and the file sources. These staged files are archived to a second directory structure that maintains them by DLXS middleware class, by release number and by ISO image. They are then copied from the archive to the DLXS document root on dev-linux. Finally they are rdisted to the dlxs.org web site on production servers.

Documentation Edits

The current, evolving documentation is maintained in the DLXS wiki. Wiki edits to the documentation either correct/supplement the documentation for the current release or document the next release. If they document the next release, they are inserted using a special style.

For release number N:

• Update the install steps at Installing_DLXS
• Update the system requirements at System_Requirements
• Integrate the specially styled wiki edits into the current documentation
• Make a snapshot of the wiki and save it to /l1/web/d/dlxs/docs/N/
• Run /l1/bin/d/dlxsdocs/swishe-index.csh N to index the documentation.
• Edit http://dlxs.org/docs/index.html and point to the snapshot.

Directories, Scripts and Files

Directories

There are several directory structures involved in the release process as mentioned in the overview. This structure may seem a bit complex but its maintenance and population is totally automated by the release build scripts. No human intervention into this structure should be required.

• /l1/{bin,cgi,lib,idx,obj,prep,web} - These are the top-level directories where CVS-controlled source files are stored. These are usually referred to as the "release" directories to differentiate them from the development directories where developers (/l1/dev/uniqname) do their work. CVS tags are applied to the files in the release directories.
• /l1/stage{text,image,bib,findaid,lib, ...} - These directories are the staging areas into which CVS tagged files are exported. There is one directory for each DLXS class of software.
• /l1/dlxs_release - is the archive for DLXS releases. It has two important subdirectories:
  • archive-software - the archive with subdirectories by class, image and release number:
    • archive-by-CDROM - contains the archive by release number. Actually each release number under subdirectory under this directory contains symlinks to all of the gzipped tar files for the classes that make up a given release. For example the BibClass symlink might be: BibClass_v3-1-0_rel14.tar.gz -> ../../archive-by-class/BibClass/BibClass_v3-1-0_rel14.tar.gz
    • archive-by-class - contains the archive of all of the gzipped tarfiles for all releases divided up by DLXS class name. These are pointed to from the archive-by-CDROM directories as shown in the last item.
    • archive-by-image - contains the archive of the ISO images of the release. These are complete mountable filesystem files containing all of the gzipped tarfiles that constitute a given release. This archive directory is also divided by release number. Each release has two ISO images. For example, the release 14 subdirectory contains dlxs_open_14.iso and dlxs_xpat_14.iso
  • cdrom-software - a symlink to archive-software/archive-by-image/RELEASE_NUMBER where RELEASE_NUMBER is the current release. This symlink is a convenience to the person who buns the ISO images to cdrom for a given release.
• /l1/web/d/dlxs/products - this is the product directory under the DLXS DocumentRoot on dev-linux. The entire DocumentRoot is mirrored on the production server(s). The archive is copied to the products directory here as a staging area for deployment to production via rdist. The DocumentRoot parent of the products directory contain the rdist file for the software (but not the documentation) for a given release.
• /l1/web/d/dlxs/docs - this is the documentation directory under the DLXS DocumentRoot on dev-linux. It contains archived snapshots of the wiki documentation, indexed, by release number and, in the releasenotes subdirectory, the release note splash pages e.g. releasenotes10.html by release number and the documentation rdist file. The documentation is rdisted as a separate step from the software so it can be updated for accuracy for past releases.

Scripts

Database prep scripts

There are a number of Perl scripts that semi-automate the database preparation and testing phase of the release process. Database preparation consists of dumping a subset of the current DLXS production database for delivery with the release software. An upgrade script upgrade_N_N+1 (e.g. upgrade_5_6) has to be written if the database schema has changed since the last release N to the current release N+1.

• /l1/bin/db/dbdump - This utility connects to the production database and creates dump files that end up in /l1/misc/db named db-dump-vN-release.sql and db-dump-vN-workshop.sql where N is the release number, e.g. 6.0.
• /l1/bin/db/dbcmp - This utility performs a sanity check on the upgrade_N_N+1 script to ensure that the result of upgrading the database delivered with the previous release is the same as the dump of the current database.
• upgrade_N_N+1 - This Perl script is written for each release and delivered with the release to allow sites with previous releases of DLXS to upgrade their database to work with the new release software.
• /l1/bin/db/droptables.sql - A convenience file containing the list of table to drop to revert a DLXS database to an empty state.

Release building scripts

There are a number of Perl scripts that semi-automate the release building phase of the process. These are the main work horses of the release process. These are usually executed in the order listed here.

• /l/local/bin/build-release-package - This is a driver script that performs several related steps. It does these steps for each DLXS software class, e.g. "text", "image", etc. It generates uniform Tag names based on the release number and any secondary release numbers like "a", "b" etc. In detail it:
• • CVS updates the release directory using /l1/bin/updaterelease.full
  • Removes all files from /l1/stageclass
  • Prompts for a CVS tag for the class and applys it using /l1/bin/c/class/cvstag.class
  • CVS exports files with that tag to /l1/stageclass
  • Removes passwords and some UM related information from software files exported to /l1/stageclass
  • Tars and gzips the files exported to /l1/stageclass.
• /l/local/bin/build-cdrom-package - This driver runs the process of gathering up all the gzipped tarfiles, creating the directory structures describe <a href="#complexdirstruct">above</a> and populating them with the gzipped tarfiles now existing in /l1/stageclass. It then refers to this complex directory structure and makes ISO images it storing them in /l1/dlxs_release/archive-software/archive-by-image/RELEASE_NUMBER
• /l/local/bin/web-release-middleware - This driver copies the files placed and created by build-cdrom-package into the DLXS web DocumentRoot by release number. This it it copies to: /l1/web/d/dlxs/products/archive-by-CDROM/RELEASE_NUMBER
• /l/local/bin/web-release-xpatl - This driver runs a process identical to /l/local/bin/web-release-middleware except it only handles the files related to the XPAT Lite Toolset. These files rarely change and so this driver is not ususally run.

Staging New Binaries

We deliver a number of auxilliary programs as binaries for Solaris and Linux. Some of them are delivered with source files included. These programs should be staged in /l/local/src/programname by the name of the binary. Under this directory should be the various releases of the program and a dist directory. The dist is the storage location for different versioned directories of the binary and is used by the release build scripts, e.g. programname-1.0.1. In this location should be found the compressed binaries for Solaris/Sparc and Linux/x86 with the following naming convention: programname-linux-x86.gz and programname-solaris-sparc.gz. If the source is delivered it sholuld be named programname-src.tar.gz. It may be necessary to build the binaries. This step is not discussed in this document.

Database preparation

New Database Prepration

Dump the current DLXS production database subsets for the release and for the DLXS workshop using the /l1/bin/db/dbdump program. In /l1/bin/db/dbdump.cfg: make sure the $UseDevelopmentDbname and $UseDevelopmentServer flags are set to ZERO (0). This will create/l1/misc/db/db-dump-v6.0-release.sql and /l1/misc/db/db-dump-v6.0-workshop.sql (assuming the current release's database version is 6. CVS add and commit both these sql dump files.

In /l1/bin/db/dbdump.cfg: if there are any new tables or if tables have been dropped from the current production database or fields have been added or deleted since the last time this file was edited, paste those commands into the %gDlxsTablesHash. Adjust the other hashes as needed. Make sure the value in the Version.version column in the current database was advanced following the previous release. Check the /l1/cgi/c/collmgr/collmgr.cfg flags $gCollmgrDocumentationRelease and $gCollmgrVersion are set to the right versions.

• Add any tables that were dropped from the database for this release to /l1/bin/db/droptables.sql and CVS commit.

Database Upgrade Prepration

Write the upgrade script:

Write upgrade_N_N+1 and upgrade_N_N+1.cfg where N+1 is the release you are going to build. These files may only need cloning as they are framework-based. The business end of the upgrade is in upgrade_N_N+1.cfg. The %gCommandsHash should contain, in order, the SQL commands needed to convert a version N database to version N+1. Note that the collmgr records alterations to tables in the dlxs.CollmgrChangeLog table. Consult this table for changes. Newly added whole tables are not recorded as this function would occur outside of collmgr.

Load and perform the upgrade for testing:

Use the development dev.mysql.umdl.umich.edu server to test the upgrade script. Run with $DLXSROOT set to the release directory /l1. You will use the scratch database named dlxs_previous to hold the dump of the previous release and the scratch database named dlxs_current to hold the dump of the current release. If either the database is not empty, drop its tables using /l1/bin/db/droptables.sql via the mysql command line client.

• Load the previous release's database dump into dlxs_previous. For example, if the previous release was 12 (which used version 5 of the database), load /l1/misc/db/db-dump-v5.0-release.sql into dlxs_previous.
• Set $UseDevelopmentServer and $UseDevelopmentDbname to 1 in /l1/bin/db/upgrade_N_N+1.cfg
• Run /l1/bin/db/upgrade_N_N+1. This will upgrade the contents of dlxs_previous in place.

Test the result of the upgrade:

• Load the current release's database dump into dlxs_current. The dbdump script (which you ran already to produce the dump of the current release) configures itself from lib/LibGlobals.cfg so it dumped a subset of the current production database. For example, if the current release is 13 and it uses verion 6 of the database, load the release 13 dump from /l1/misc/db/db-dump-v6.0-release.sql into dlxs_current.
• Run /l1/bin/db/dbcmp -q -o dlxs_previous -n dlxs_current recalling that you have just upgraded dlxs_previous so the hope is that it will match the release dump of the current database. /l1/bin/db/dbcmp will compare numbers of table, names of tables, names of fields in tables and datatypes of fields. If the old upgraded database compares with a release dump of the current database all should be well, the upgrade script works and you have created the workshop and release dumps for the current release.

• Make sure that the $UseDevelopmentServer variable in /l1/bin/db/upgrade_N_N+1.cfg is set to ZERO (0) in the version committed to CVS.

Script Configuration

Release build scripts

This section lists and explains the edits to script configuration variables that define the content of the release.

• /l/local/bin/BuildFuncs.pm
  • Add any new classes to %gClasses
  • Add names for the new classes to %gClassNames
  • Add a coresponding line to %gClassTag
  • Add a line to %gClassDirs. The fields in each line specify which directories (cgi, bin, web, etc.) contain files that will be tagged and so should be exported. The directories specified in ths line MUST match the directories tagged by /l1/bin/c/class/cvstag.class script. Check this!
  • Add any special commands that should be applied to files after they have been exported to /l1/stageclass to %gClassCmds. An example of this is symlinking.
  • Add any password removal lines to %gRemovePasswordCmds

• /l/local/bin/build-cdrom-package
  • Edit %gClasses and %gClassNames in this file in the same way as above.
  • Edit %gBinaries by creating a new block of values for the upcoming release number. The values in the block should reflect newer versions of existing binaries and any new binaries. Obsolete binaries should be removed. Here's an release 13 example:

'13' => {
         'tif2web'         => 'tif2web-1.0.4',
         'mrsid_retrieve'  => 'mrsid_retrieve-1.3.1',
         'dlxsd'           => 'dlxsd-1.0.1',
         'XPAT'            => 'xpat-5.3.2',
         'dlps_auth'       => 'dlps_auth-1.5',
         'kakadu'          => 'kakadu-4.0.2',
         'cjpeg'           => 'cjpeg-6b',
         'xpatutf8check'   => 'xpatutf8check-1.0',
         'utf8conditioner' => 'utf8conditioner-1.0.1',
         }, 

• • Add the next new release number to the prompt in $cdromNumber

• /l/local/bin/web-release-middleware

• • Add a new block to %releaseDefs. This block specifies for each class name as listed in %gClassNames in BuildFuncs.pm whether the class is a gzipped tarfile, a binary distribution or a ISO image.
  • Add the next new release number to the prompt in $releaseNumber

• /l/local/bin/web-release-xpatl Set the XPAT Lite release number in $releaseNumber

• /l1/web/d/dlxs/release_versions.txt Set the values to correspond to the release versions in /l1/web/d/dlxs/versions.html for batch processing using the a)ll option in /l/local/bin/build-release-package

DLXS Installer script configuration

• /l1/bin/installer/Installer.pl is the program that installs a DLXS release. /l1/bin/installer/main.cfg is its configuration file. Change $VERSION to the release number, e.g. 13 and $DB_VERSION to the the database version, e.g. 6.0 in main.cfg
• Add any new Perl modules that DLXS depends on to @perlModules in main.cfg
• Run /l1/dev/uniqname/bin/installer/Installer.pl -u to update the versions in @perlModules in main.cfg to the latest values.
• Run /l1/bin/installer/FindConfigFiles.pl to get a complete list of .cfg files in DLXSROOT. For each file in this list that contains an Installer comment (#> OPEN INSTALLATION BLOCK) make sure it is listed in @gConfigFiles in main.cfg.
• Edit the lists of scripts in which the Perl hashbang and DLXSROOT values need to be fixed up. Add new one and delete obsolete ones main.cfg
• Edit the list of calls to make_symlink in main.cfg to make any necessary symlinks to new binaries that the Middleware reaches from DLXSROOT/bin/symlinks
• Update the post install notes block at the end of main.cfg
• CVS commit main.cfg

Preparing to Build the Release

There are a number of small tasks to to complete before building the release:

• Use build-production-package to put all classes into production before the release. The code to be delivered should have been running in production for 4 weeks without problems before release.
• Survey all developers involved with base class development to make sure that all modified files that are part of the release are committed to CVS. This includes yourself!
• Declare a code freeze to dlxs-staff and spo-staff via email. Base class middleware and XML/XSL template changes are affected. No one should commit changes to these files. Subclasses are not affected by the freeze and can continue to be committed.
• Increment the CVS version numbers on all DLXS classes in /l1/web/d/dlxs/versions.html. Version numbers are of the form I.J.K where I is the major version, J is the minor version and K is the tertiary version number. Use you judgment as to the increments. Usually the tertiary version is set to 0 and the minor version is incremented by 1. If there have been virtually no changes, just increment the tertiary version by 1.
• Change the $::VERSION Perl variable in all the class main program files class-idx to match the I.J.K value. CVS commit them.
• Add any tables that were dropped from the database for this release to /l1/bin/db/droptables.sql and CVS commit.
• Increment the version variables in lib/LibVersion.pm and CVS commit. $LibVersion::VERSION should be the release I.J.K /l1/web/d/dlxs/versions.html value for the Lib middleware class. $LibVersion::DLXS_CDROM is the release number, e.g. 13;
• If BibClass, TextClass or FindaidClass sample XML data has changed since the last release. re-index it using /l1/bin/s/sampleXX/Makefile and CVS commit the changes to the class sample obj, idx and prep directories.
• Edit /l1/web/d/dlxs/Distfile to add an except_pat for the previous release number so rdist does not spend time checking all previous release to determine the set to update..

Building the Release

The build is actually the most automatic part of the process. In order perform the following steps:

1. Run /l/local/bin/build-release-package: Select "a)ll" classes. For each class, you will be prompted for the version number for each class (use the values in /l1/web/d/dlxs/versions.html), the release number, and an optional "a", "b", etc. version. The script will then CVS update the release directory. You will then be prompted to allow the tagging to go ahead. Answer "Y" (yes). The script will perform tagging followed by a CVS export of the class to its staging area /l1/stageclass.
2. Run /l/local/bin/build-cdrom-package: You will be prompted to enter the release number. When prompted to "Jump directly to build images?" select "n)o". The script will set up the structure described in Directories above.
3. Run /l/local/bin/web-release-middleware: You will be prompted to enter the release number. Then select "a)ll" classes. This will populate /l1/web/d/dlxs/products while will be deployed to the DLXS web site in later steps.
4. Optionally, run /l/local/bin/web-release-xpatl

Installing and Testing the Release

This step involves installing the release and loading the database dump. It will test the installer, and the sanity of the release configuration and build process. Substitute the release number for N in the steps below. If you want to redo the install, use /tmp/main.cfg to re-install, recalling that it is created by Installer.pl and is read by default and contains all you previous answers if present.

Run the Installer

• As root, mount the ISO image of the release N middleware using /l1/release/cd as the mount point:
  % cd /l1/release
% mount -o loop /l1/dlxs_release/cdrom-software/dlxs_open_N.iso cd
  
• Change directory to /l1/release/cd and run the installer answering all the prompts:  % perl ./Installer.pl
• For the database name use dlxs_N which you created as described in New database prepration above. Passwords will be those for the administrative and middleware userids for DLPS development database administrative user and production databases, respectively.
• For the DLXSROOT value use /l1/release/N

Load the Database

Load the sample database dump file /l1/release/N/misc/db/db-dump-vN.0-release.sql using the following command.

Test the Install

• Have system administration create the virtual host releaseN.umdl.umich.edu on the development web server.

Test functionality using the following URLs:

Collmgr:	http://releaseN.dev.umdl.umich.edu/cgi/c/collmgr/collmgr
BibClass:	http://releaseN.dev.umdl.umich.edu/cgi/b/bib/bib-idx?c=samplebc
FindaidClass:	http://releaseN.dev.umdl.umich.edu/cgi/f/findaid/findaid-idx?c=samplefa
ImageClass:	http://releaseN.dev.umdl.umich.edu/cgi/i/image/image-idx?c=sampleic
TextClass:	http://releaseN.dev.mdl.umich.edu/cgi/t/text/text-idx?c=sampletc_utf8

Deploying the Release

In this step the release middleware and documentation is copied to the DLXS website and CDROMs are burned.

• Edit /l1/web/d/dlxs/products/index.html for the new release.
• Edit /l1/web/d/dlxs/products/index.html for the new release.
• Create /l1/web/d/dlxs/products/releaseN.html for the new release.
• Create /l1/web/d/dlxs/products/releaseN_OAI.html for the new release.
• Change directory to /l1/web/d/dlxs and run rdist making sure you have added an except_pat for previous releases:
  % rdist -f Distfile
• Change directory to /l1/web/d/dlxs/docs and run rdist making sure you have indexed the documentation as described in <a href="#documentchanges">Documentation Edits</a>:
  % rdist -f Distfile
  This step can be run repeatedly as contributors add material to the documentation for the current release (or previous ones, for that matter).
• Make sure the /l1/dlxs_release/cdrom-software symlink is pointing to the ISO images for the release you have just built and notify the person responsible for burning those images to CDROM.
• Send out the announcement by email to the DLXS list.

Post-Release Operations

• Edit /l1/web/c/collmgr/collmgr.cfg to change $gCollmgrVersion to the next release number N+1 and CVS commit the change.
• Change the value in the Version.version column in the production (mysqlserv) database dlxs to N+1.
• Change the value in the Version.version column in the development (dev.mysql.umdl.umich.edu) database dlxs to N+1.

Deploying Post-Release Patches

Patches may be created after the release. These should consist of wholesale replacements for the affected files.

• Place the files in a subdirectory of /l1/dlxs_release/archive-software/archive-by-class/Patches/N where N is the number of the release to be patched. If there are several files, use zip to capture their entire subdirectory tree and place the zip file in the patch subdirectory. For example, if you are delivering obj/s/samplefa/samplefa.xml and idx/s/samplefa/samplefa.dd include both directories in the zip file. Create the patch directory by hand if it does not exist. The patch subdirectory should be named by date using the format DDMonthYYYY, e.g. 17July2007. The patch files should be accompanied by a README file explaining the patch.
• Run /l/local/bin/web-release-middleware -p: (Note the -p switch) You will be prompted to enter the release number. This will populate the correct subdirectory under /l1/web/d/dlxs/products with the patch files, creating directories as needed.
• Deploy the patch to www.dlxs.org by via rdist:

  cd /l1/web/d/dlxs
rdist -f Distfile

• Announce the patch to the DLXS listserv.