VSI Testlib

testlib.bsh

Simple shell command language test library

Copyright:

License: MIT

## Writing unit tests

Testlib gives you a number of basic unit test functionality from bash, including:

Running tests in subshells to prevent environment variable pollution.
An automatically self deleting TRASHDIR for all the tests
An automatically self deleting TESTDIR for each individual test (in the TRASHDIR)
A tally of successfully run and failed tests, in addition to expected failures, unexpected successes, required failures, and skipped tests
Individual est times: TESTLIB_SHOW_TIMING
A user defined setup function run before the first test in a file
A user defined teardown function run after the last test in a file
Keep temporary directories for debugging: TESTLIB_KEEP_TEMP_DIRS
Pause before deleting temporary directories if there is a failure for inspection: TESTLIB_KEEP_PAUSE_AFTER_ERROR
Run only a single test by its description: TESTLIB_RUN_SINGLE_TEST
Regular expression to skip tests by description: TESTLIB_SKIP_TESTS
Controlled stderr/stdout redirection TESTLIB_REDIRECT_OUTPUT
Stop testing after N failures TESTLIB_STOP_AFTER_FAILS
Custom PS4 in trace using TESTLIB_PS4
Ability to conditionally skip a test by calling skip_next_test in any condition check
Track files outside the TRASHDIR with ttouch so that they will be automatically deleted during cleanup.
Other helper functions like test_utils.bsh not, test_utils.bsh not_s, test_utils.bsh assert_array_values, test_utils.bsh assert_array_regex_values, test_utils.bsh assert_array_contiguous
Auto discover and run tests script: run_tests

Example

Tests must follow the basic form:

source testlib.bsh

begin_test "the thing"
(
     set -e
     echo "hello"
     [ 1 == 1 ] # this is ok
     # However, the following needs "|| false" because on bash 3.2 and 4.0
     # there is a bug where [[ ]] will fail, and bash knows it fails ($?),
     # but "set -e" does not count this as an error. This is fixed in bash 4.1
     [[ 1 == 1 ]] || false
     false
)
end_test

Any command that evaluates to false "fails" the test. When a test fails, its stdout, stderr, and call trace are printed out.

Bugs

On darling: when debugging a unit test error, sometimes the printout is cut off, making it difficult to do “printf debugging.” While the cause and scope of this is unknown, a work around that sometimes works is

runtests 2>&1 | less -R

## Test status

Every test will print out a line with its name, and the status of the test run.

Possible results of a test:

OK - The test passed!
FAILED - The test did not pass. A trace is printed out for debugging.
SETUP FAILURE - Each test must call setup_test, and it was not detected for this test.
SKIPPED - The test was not run.
FAIL REQUIRED - A test successfully failed as it is required to. If the test succeeds, it is treated as a failed test.
FAIL EXPECTED - A test successfully failed as it is expected to. If the test succeeds, it is treated as an unexpected success.
SHOULD HAVE FAILED ELSEWHERE - A required or expected to fail test failed in an the wrong area of the code. There is something wrong with the test, and a trace is printed out for debugging.
SHOULD HAVE FAILED - A required to fail test did not fail. There is something wrong with the test, and a trace is printed out for debugging.
UNEXPECTED SUCCESS - An expected to fail test never actually failed. This doesn’t count as a failure, but a middle ground in its own category.
REQUIRED FAILURE SETUP ERROR - A required to fail test did not call begin_fail_zone and is considered setup incorrectly.
EXPECTED FAILURE SETUP ERROR - An expected to fail test did not call begin_fail_zone and is considered setup incorrectly.

setup

Function run before the first test. Must be declared before the first test is called in the test file, or else it will not be discovered in time.

Note

A directory TRASHDIR is created for setup, right before running setup ().

Setup is not run if no tests are ever run

TRASHDIR

Temporary directory where everything for the test file is stored

Automatically generated and removed (unless TESTLIB_KEEP_TEMP_DIRS is changed)

See also

TESTDIR

TESTDIR

Unique temporary directory for a single test (in TRASHDIR)

Automatically generated and removed (unless TESTLIB_KEEP_TEMP_DIRS is changed)

See also

TRASHDIR

TESTLIB_DIR

Directory of testlib’s source code

teardown

Function run after the last test

Note

Teardown is not run if no tests are ever run

TESTLIB_KEEP_TEMP_DIRS

Keep the trashdir/setup dir

Debug flag to keep the temporary directories generated when testing. Set to 1 to keep directories.

Default:: 0

TESTLIB_KEEP_PAUSE_AFTER_ERROR

Pauses before allowing testlib to cleanup and delete all temporary files.

A better alternative to TESTLIB_KEEP_TEMP_DIRS, so that you are given time to investigate a problem, and can then press any key to cleanup the temporary directory

Default:: 0

TESTLIB_SHOW_TIMING

Display test time after each test

Debug flag to display time elapsed for each test. Set to 1 to enable.

Default:: 0

TESTLIB_RUN_SINGLE_TEST

Run a single test

Instead of running all the tests in a test file, only the tests with a description exactly matching the value of TESTLIB_RUN_SINGLE_TEST will be run. Useful for debugging a specific test/piece of code

Default:: unset

TESTLIB_SKIP_TESTS

A bash regex expressions that designates tests to be skipped.

Default:: unset

Examples

TESTLIB_SKIP_TESTS='^New Just$|foo'
# Skip "New Just" and anything with "foo" it is, e.g. "food"

TESTLIB_REDIRECT_OUTPUT

Redirects stdout and stderr to temporary files

By default, all tests are run with set -xv for debugging purposes when a tests fails. This output is stored in a out/err/xtrace file temporarily and only displayed if a tests fails. You can set this variable to control the streams to always output.

Values:

3 Redirect stdout, stderr, and xtrace
2 Redirect stderr, and xtrace, but let stdout through
1 Redirect xtrace, but let stdout and stderr through. On bash 4.0 and older, this will let xtrace through too
0 Let everything through

Default:

3

TESTLIB_PS4

Optionally set a custom PS4 output for trace output on test errors. If unset, the testlib default is use: +${BASH_SOURCE[0]##*/}:${LINENO})\t

Default:: unset

TESTLIB_STOP_AFTER_FAILS

If set, stops after this many tests have fails in a single file. Instead of running the rest of the tests in a file, they are skipped.

Default:: 0 - Unlimited

atexit

Function that runs at process exit

Usage

Automatically called on exit by trap.

Checks to see if teardown is defined, and calls it. teardown is typically a function, alias, or something that makes sense to call.

begin_test

Beginning of test demarcation

Usage

Mark the beginning of a test. A subshell should immediately follow this statement.

See also

end_test

begin_expected_fail_test

Beginning of expected to fail test demarcation

Usage

Define the beginning of a test that is expected to fail. Failures may only occur in “fail zones” denoted by begin_fail_zone and end_fail_zone. When a test fails in a fail zone, it is counted as a success. If a test that was expected to fail never fails, it counts as an “unexpected success” rather than a normal success. If the test fails outside a fail zone, it is marked as a failure.

The typical use case for expecting a failure is when a known bug is being tested and has not or cannot be fixed yet. For this reason, a success is counted as an “unexpected success” rather than a normal success. While unexpected successes do not cause a non-zero exit code, they can easily be noticed as something that should be checked out.

Note

end_fail_zone is not typically needed.

begin_required_fail_test

Beginning of required to fail test demarcation

Usage

Define the beginning of a test that is required to fail. Failures may only occur in “fail zones” denoted by begin_fail_zone and end_fail_zone. When a test fails in a fail zone, it is counted as a success. If a test that was required to fail never fails, it counts as a failure. If the test fails outside a fail zone, it is marked as a failure.

The typical use case for requiring a failure is testing that an exception is raise under proper circumstances.

Note

end_fail_zone is not typically needed.

begin_fail_zone

Start a fail zone

In practice, having a test that is expected or required to fail leads to the possibility of a test failing somewhere you don’t expect it to. For this reason, fail zones must be denoted in order for begin_expected_fail_test and begin_required_fail_test tests to succeed, and the failures must only occur in a fail zone. If a failure happens outside the fail zone, the test will be marked as a failure with the message SHOULD HAVE FAILED ELSEWHERE.

Typically this will be called right before the last last line of a test

Example

begin_expected_fail_test "Some test"
(
  setup_test
  # Failing here would result in failure
  begin_fail_zone
  false is ok here
)

end_fail_zone

End a fail zone

While not common, it might be possible to have a test that is likely to fail in one of many places; for this reason a fail zone can be turned off, before being turned on again.

Example

begin_required_fail_test "Some test"
(
  setup_test
  true something

  begin_fail_zone
  maybe_false
  end_fail_zone

  true again

  begin_fail_zone
  false_if_other_was_not
  # end_fail_zone # not required at the end of the test, but won't hurt
)
end_test

setup_test

Sets up the test

Once inside the () subshell, typically set -eu needs to be run, then other things such as checking to see if a test should be skipped, etc. need to be done. This is all encapsulated into setup_test. This is required; without it, end_test will know you forgot to call this and fail.

This is also the second part of creating a skippable test.

You are free to change “set -eu” after setup_test, should you wish.

Usage

Place at the beginning of a test

Example

skip_next_test
begin_test "Skipping test"
(
  setup_test
  #test code here
)

See also

skip_next_test

end_test

End of a test demarcation

Usage

Mark the end of a test. Must be the first command after the test group, or else the return value will not be captured successfully.

See also

begin_test

skip_next_test

Function to indicate the next test should be skipped

This is the first part of creating a skippable test, used in conjunction with setup_test

Example

For example, skip if docker command not found

  if ! command -v docker &> /dev/null; then
    skip_next_test
  fi
  begin_test "My test"
  (
    setup_test
    [ "$(docker run -it --rm ubuntu:14.04 echo hi)" = "hi" ]
  )

See also

setup_test

Note

This must be done outside of the test, or else the skip variable will not be set and detected by end_test

track_touched_files

Start tracking touched files

After running track_touched_files, any call to ttouch will cause that file to be added to the internal list (touched_files). Just prior to the teardown phase, all of these files will be automatically removed for your convenience.

ttouch should be used in cases where a file cannot be redirected to TESTDIR or TRASHDIR

Example

setup()
{
  track_touched_files
}
begin_test Testing
(
  ttouch /tmp/hiya
)
end_test

Usage

Should be called before the begin_test block, not inside. Inside a () subshell block will not work. Setup is the logical place to put it.

Bugs

Does not work in sh, only bash. Uses array, and I didn’t want to make this use a string instead.

Not thread safe. Use a different file for each thread

See also

cleanup_touched_files

ttouch

Touch function that should behave like the original touch command

See also

track_touched_files

cleanup_touched_files

Delete all the touched files

At the end of the last test, delete all the files in the array