DLXS Simple Regression Testing System

From DLXS Documentation

Revision as of 18:55, 26 July 2007 (edit) Cboulay (Talk | contribs) (→</a>Operation) ← Previous diff		Revision as of 19:11, 26 July 2007 (edit) (undo) Cboulay (Talk | contribs) Next diff →
Line 52:		Line 52:
	<p>The operation of the system consists of running the system script from the shell command line. There are four basic functions supported by the system. These are:		<p>The operation of the system consists of running the system script from the shell command line. There are four basic functions supported by the system. These are:
	<ul>		<ul>
-	<li>[[Executing Test Cases]]</li>	+	<li>[[#Executing_Test_Cases]]</li>
-	<li>[[Executing Failed Test Cases]]</li>	+	<li>[[#Executing_Failed_Test_Cases]]</li>
-		+	<li>[[#Importing_Test_Cases_to_the_Database]]</li>
-	<li>[[Importing Test Cases to the Database]]</li>	+	<li>[[#Listing_Test_Cases]]</li>
-	<li>[[Listing Test Cases]]</li>	+
	</ul>		</ul>

---

Revision as of 19:11, 26 July 2007

Contents 1 Functional Overview 2 Technical Overview 3 Operation 3.1 Executing Test Cases 3.2 Executing Failed Test Cases 3.3 Importing Test Cases to the Database 3.4 Listing Test Cases 4 Test Case and Database Design and Management

Functional Overview

The DLXS Simple Regression Testing System consists of script to drive the testing process and a collection of files representing the test case database. Database entries are URL strings. The system can test for assertion failures and runtime errors in the DLXS middleware. It is not a system for verifying the correctness of search results or of the visual appearance of web pages.

The system has the following general capabilities:

• Read the test case database, submit URLs to the middleware, receive HTTP status results returning from the middleware.
• Report test case results.
• Save failed test cases to a separate database which can be subsequently used for re-testing.
• Import user-defined test cases consisting of URLs categorized into groups by DLXS class and by user defined sets within and across DLXS classes.
• List the contents of the test case database(s) completely or by class or user-defined set.

While not designed to verify correctness of search results or the visual appearance of web pages the system does provide a powerful check on the basic operation of the DLXS middleware. By supporting the handling of many hundreds of URLs, the system can perform a more comprehensive test than actual users are able to do by random interaction through a web browser.

The system can grow in completeness over time as URLs that have generated errors are added to the database to be checked against subsequent coding changes. The fact that the tests are automated means that they are more likely to be run, resulting a more stable system.

Technical Overview

The test system script and associated files that make up the database are located in DLXSROOT/bin/regression. Within this directory are the following components:

• simple: regression testing and database management program.
• simple.cfg: configuration data including default filenames and virtual host.
• Directory: testcases/: storage for the test case database files.
• Directory: failures/: storage for the database files containing failed test cases.
• Directory: imports/: storage for user-created test case data to be imported into the test case database.

The test case database has a primitive schema consisting of the following fields:

CLASS

SETID

URL

• CLASS is one of mdp, text, image, findaid, bib or all
• SET is a user-defined set ID or "set:default". Set ID syntax is: set:name where name is user defined.
• URL is a full URL string to be submitted as one test case. It must include the http:// protocol specifier.

The test system supports cookies. DLXS stores the session ID in the cookie. Therefore, during the execution of a sequence of test cases, the state of the DLXS middleware, as maintained in the session, is preserved. This allows testing of search history, bookbag functions, portfolios and various cache mechanisms.

Operation

The operation of the system consists of running the system script from the shell command line. There are four basic functions supported by the system. These are:

• #Executing_Test_Cases
• #Executing_Failed_Test_Cases
• #Importing_Test_Cases_to_the_Database
• #Listing_Test_Cases

Executing Test Cases

       <p>The regression test script (simple) has the following command line arguments related to executing test cases. Square braces indicate optional arguments.  Note that for test execution, all arguments are optional.  In that case, all existing test cases are executed on the default virtual host, using the default database (testcases/default.tcd).

• [ -f TESTCASEFILE | "all" ]
  If this argument is omitted, only the default database is used. The default database is stored in testcases/default.tcd. If TESTCASEFILE is supplied, the testcases/TESTCASEFILE is used. If all is supplied, all databases in testcases/ are used. TESTCASEFILE must have a .tcd extension.
• [ -v VIRTUALHOST ]
  If this argument is omitted, the default virtual host is used. In the University of Michigan DLPS environment this is dev-linux.umdl.umich.edu. Note that by supplying this argument it is possible to conduct tests on a variety of servers.
• [ -k CLASS | "all" ]
  If this argument is omitted, tests related to all classes for all sets (unless a particular set is specified) in the indicated database will be executed . Otherwise, tests related to only CLASS are executed (subject to any limitation on sets). See above for allowed values of CLASS.
• [ -s SETID | "all" ]
  If this argument is omitted, all set tests related to the CLASS argument (or all classes if CLASS argument is omitted) and for the indicated database will be executed. If this argument is supplied, tests related only to SETID (subject to any limitations on sets) are executed.

Test execution progress and results are written to the terminal. They can also be captured by redirecting the output of the test script to a file. Here is an example of a results report.

               Overall Test Result Summary
               ---------------------------------------
                       Total test cases executed: 4
                               Total elapsed time: 8 seconds
                               Total successful tests: 4
                               Total failed test cases: 0
                       Class="text" had:
                               4 successes
                               0 failures
                                       Setid="set:default" had:
                                               1 successes
                                               0 failures
                                       Setid="set:groups" had:
                                               1 successes
                                               0 failures
                                       Setid="set:tcgroups" had:
                                               2 successes
                                               0 failures

               Failed Test Result Summary
                       No failures
            

Executing Failed Test Cases

A test case originates from a test case database identified by some filename testcases/TESTCASEFILE. If that test case fails, it is appended to a database file in the failures/ directory also named TESTCASEFILE. The failures/TESTCASEFILE is initialized before the test cases from a given database are executed, wiping out any cases saved in the failures/TESTCASEFILE database from previous runs.

The test system can be run against these failed test case databases. Only one database at a time can be used as a source of failed test cases to be re-executed. These database files can be renamed to preserve them so that they are not overwritten, if desired. The following arguments to the test system are supported.

• -R
  If this argument is used the tests found in the indicated database under failures/ are executed.
• [ -f TESTCASEFILE ]
  If this argument is omitted, the default database is used. The default failure database is stored in failures/default.tcd. Otherwise, the named database is the source of the failed test cases.
• [ -v VIRTUALHOST ]
  If this argument is supplied the indicated virtual host is used. Otherwise, the default virtual host is used. This man be a different host that was used when the failure cases were generated. This may be useful for comparison off the code running on different hosts.
• [ -k CLASS | "all" ]
  If this argument is omitted, tests related to all classes for all sets (unless a particular set is specified) in the indicated database will be executed . Otherwise, tests related to only CLASS are executed (subject to any limitation on sets). See above for allowed values of CLASS.
• [ -s SETID | "all" ]
  If this argument is omitted, all set tests related to the CLASS argument (or all classes if CLASS argument is omitted) and for the indicated database will be executed. If this argument is supplied, tests related only to SETID (subject to any limitations on sets) are executed.

Importing Test Cases to the Database

The test system supports the importation of new test cases. The new test cases may be imported into a new database or incorporated into an existing database. The following command line arguments control the import function.

The format of the test case definition file is exhibited in this example:

                    [text]
                    [set:default]

                    http://__VHOST__/cgi/t/text/text-idx
                    http://__VHOST__/cgi/t/text/text-idx?page=newmaterial
                    http://__VHOST__/t/text/accesspolicy.html
                    http://__VHOST__/cgi/t/text/text-idx?page=browsecolls

                    [set:tcgroups]

                    http://__VHOST__/cgi/t/text/text-idx?xg=1;page=simple;g=dentist
                    http://__VHOST__/cgi/t/text/text-idx?xg=1;page=simple;g=amer19c
                    http://__VHOST__/cgi/t/text/text-idx?xg=1;page=simple;g=llmc
                    http://__VHOST__/cgi/t/text/text-idx?xg=1;page=simple;g=moagrp
                    http://__VHOST__/cgi/t/text/text-idx?xg=1;page=simple;g=mqrg
            

         Comments begin with '#'.  Blank lines are skipped.  The set specifer is optional and defaults to [set:default].  Use __VHOST__ to be replaced by the virtual host argument or the default virtual host when the test is executed.

The name of the import file determines the name of the database file that will be the destination of the import. So, for example, if the import file is imports/specialcases.txt, those test cases will be created in the testcases/specialcases.txt database. Omitting the import file name means to use imports/default.txt as source and testcases/default.tcd as destination.

This convention supports the addition of cases to the default database and the creation of additional databases for special purposes. Note that the run options support the execution of test from all databases in one run or just cases from a single selected database.

• -I
  This argument is required to trigger the import function.
• [ -i IMPORTFILE ]
  If this argument is omitted, the test case definitions in imports/default.txt are imported and saved totestcases/default.tcd. Otherwise, imports/IMPORTFILE is imported and the test cases are created in testcases/IMPORTFILE.

The import process will not add a test case that duplicates an existing case for a given class and set within that class. Duplicates are allowed in different sets and in different databases.

The import/ directory is basically a scratch directory to hold test case definition files. Once in the database, there is no reason to keep them around. Similarly, the error database files saved in the failures/ directory do not need to be kept after the problem has been resolved since the case is also present in the test case database. The test case databases should be maintained under source control so that should any corruption occur the last know good database can be recovered.

Listing Test Cases

The test system supports the ability to dump a list of test cases for a given setid or class and from a given database. When both CLASS and SETID are specified, CLASS takes precedence. The following command line arguments control the listing function. The format of the CLASS dump is identical to the format of the test case definition file. Therefore the dump can be saved to a file, edited and used to import testcases to a database. The format of the SETID dump is importable as well if [class] specifiers are added.

• -l
  This argument is required to trigger the list function using the importable format.
• -L
  This argument is required to trigger the list function in a numbered format for reference to test cases identified by number as in failures..
• [ -f TESTCASEFILE ]
  If this argument is omitted, the default database is used. Otherwise, the named database is listed subject to the CLASS or SETID argument. TESTCASEFILE must have a .tcd extension.
• [ -k CLASS | "all" ]
  If this argument is omitted, tests related to all classes in the indicated database will be executed . Otherwise, tests related to only CLASS are executed. See above for allowed values of CLASS.
• [ -s SETID | "all" ]
  If this argument is omitted, all set tests related to the CLASS argument (or all classes if CLASS argument is omitted) and for the indicated database will be executed. If this argument is supplied, tests related only to SETID subject to the class specification are executed.

Test Case and Database Design and Management

Following are some points to consider when designing test cases and organizing them into one or more databases

• Test cases in the default database should cover the complete set of basic functionality for each class.
• For basic functionality, a minimal set of collections should be included to cover the types of material commonly encountered.
• Test cases that are specialized to a particular function across all collections should probably be placed in a separate database for that function.
• Test cases that are specialized to a particular collection should probably be placed in a separate database for that collection.
• When the middleware fails in production or development and the failure can be captured by a URL, add that URL to one of the test case databases for retesting.
• Use sets to group tests into logical blocks for a given class or even across classes.
• Other things you will probably think of.