===========
VSI Testlib
===========

.. default-domain:: bash

.. file:: testlib.bsh

Simple shell command language test library

:Copyright: Original version: (c) 2011-13 by Ryan Tomayko <http://tomayko.com>

            License: MIT

## Writing unit tests

Testlib gives you a number of basic unit test functionality from bash, including:

- Running tests in subshells to prevent environment variable pollution.
- An automatically self deleting :envvar:`TRASHDIR` for all the tests
- An automatically self deleting :envvar:`TESTDIR` for each individual test (in the :envvar:`TRASHDIR`)
- A tally of successfully run and failed tests, in addition to expected failures, unexpected successes, required failures, and skipped tests
- Individual est times: :envvar:`TESTLIB_SHOW_TIMING`
- A user defined :func:`setup` function run before the first test in a file
- A user defined :func:`teardown` function run after the last test in a file
- Keep temporary directories for debugging: :envvar:`TESTLIB_KEEP_TEMP_DIRS`
- Pause before deleting temporary directories if there is a failure for inspection: :envvar:`TESTLIB_KEEP_PAUSE_AFTER_ERROR`
- Run only a single test by its description: :envvar:`TESTLIB_RUN_SINGLE_TEST`
- Regular expression to skip tests by description: :envvar:`TESTLIB_SKIP_TESTS`
- Controlled stderr/stdout redirection :envvar:`TESTLIB_REDIRECT_OUTPUT`
- Stop testing after ``N`` failures :envvar:`TESTLIB_STOP_AFTER_FAILS`
- Custom PS4 in trace using :envvar:`TESTLIB_PS4`
- Ability to conditionally skip a test by calling :func:`skip_next_test` in any condition check
- Track files outside the :envvar:`TRASHDIR` with :func:`ttouch` so that they will be automatically deleted during cleanup.
- Other helper functions like :func:`test_utils.bsh not`, :func:`test_utils.bsh not_s`, :func:`test_utils.bsh assert_array_values`, :func:`test_utils.bsh assert_array_regex_values`, :func:`test_utils.bsh assert_array_contiguous`
- Auto discover and run tests script: :file:`run_tests`

.. rubric:: Example

.. code-block:: bash
  :caption: Tests must follow the basic form:

  source testlib.bsh

  begin_test "the thing"
  (
       set -e
       echo "hello"
       [ 1 == 1 ] # this is ok
       # However, the following needs "|| false" because on bash 3.2 and 4.0
       # there is a bug where [[ ]] will fail, and bash knows it fails ($?),
       # but "set -e" does not count this as an error. This is fixed in bash 4.1
       [[ 1 == 1 ]] || false
       false
  )
  end_test

  Any command that evaluates to false "fails" the test. When a test fails, its stdout, stderr, and call trace are printed out.

.. rubric:: Bugs

On darling: when debugging a unit test error, sometimes the printout is cut off, making it difficult to do "printf debugging." While the cause and scope of this is unknown, a work around that sometimes works is

.. code-block:: bash

    runtests 2>&1 | less -R

## Test status

Every test will print out a line with its name, and the status of the test run.

Possible results of a test:

- ``OK`` - The test passed!
- ``FAILED`` - The test did not pass. A trace is printed out for debugging.
- ``SETUP FAILURE`` - Each test must call :func:`setup_test`, and it was not detected for this test.
- ``SKIPPED`` - The test was not run.
- ``FAIL REQUIRED`` - A test successfully failed as it is required to. If the test succeeds, it is treated as a failed test.
- ``FAIL EXPECTED`` - A test successfully failed as it is expected to. If the test succeeds, it is treated as an unexpected success.
- ``SHOULD HAVE FAILED ELSEWHERE`` - A required or expected to fail test failed in an the wrong area of the code. There is something wrong with the test, and a trace is printed out for debugging.
- ``SHOULD HAVE FAILED`` - A required to fail test did not fail. There is something wrong with the test, and a trace is printed out for debugging.
- ``UNEXPECTED SUCCESS`` - An expected to fail test never actually failed. This doesn't count as a failure, but a middle ground in its own category.
- ``REQUIRED FAILURE SETUP ERROR`` - A required to fail test did not call :func:`begin_fail_zone` and is considered setup incorrectly.
- ``EXPECTED FAILURE SETUP ERROR`` - An expected to fail test did not call :func:`begin_fail_zone` and is considered setup incorrectly.

.. function:: setup

Function run before the first test. Must be declared before the first test is called in the test file, or else it will not be discovered in time.

.. note::
  A directory :envvar:`TRASHDIR` is created for setup, right before running :func:`setup` ().

  Setup is not run if no tests are ever run

.. envvar:: TRASHDIR

Temporary directory where everything for the test file is stored

Automatically generated and removed (unless :envvar:`TESTLIB_KEEP_TEMP_DIRS` is changed)

.. seealso::
  :envvar:`TESTDIR`

.. envvar:: TESTDIR

Unique temporary directory for a single test (in :envvar:`TRASHDIR`)

Automatically generated and removed (unless :envvar:`TESTLIB_KEEP_TEMP_DIRS` is changed)

.. seealso::
  :envvar:`TRASHDIR`

.. envvar:: TESTLIB_DIR

Directory of testlib's source code

.. function:: teardown

Function run after the last test

.. note::
  Teardown is not run if no tests are ever run

.. envvar:: TESTLIB_KEEP_TEMP_DIRS

Keep the trashdir/setup dir

Debug flag to keep the temporary directories generated when testing. Set to ``1`` to keep directories.

:Default: ``0``

.. envvar:: TESTLIB_KEEP_PAUSE_AFTER_ERROR

Pauses before allowing testlib to cleanup and delete all temporary files.

A better alternative to :envvar:`TESTLIB_KEEP_TEMP_DIRS`, so that you are given time to investigate a problem, and can then press any key to cleanup the temporary directory

:Default: ``0``

.. envvar:: TESTLIB_SHOW_TIMING

Display test time after each test

Debug flag to display time elapsed for each test. Set to ``1`` to enable.

:Default: ``0``

.. envvar:: TESTLIB_RUN_SINGLE_TEST

Run a single test

Instead of running all the tests in a test file, only the tests with a description exactly matching the value of :envvar:`TESTLIB_RUN_SINGLE_TEST` will be run. Useful for debugging a specific test/piece of code

:Default: *unset*

.. envvar:: TESTLIB_SKIP_TESTS

A bash regex expressions that designates tests to be skipped.

:Default: *unset*

.. rubric:: Examples

.. code-block:: bash

   TESTLIB_SKIP_TESTS='^New Just$|foo'
   # Skip "New Just" and anything with "foo" it is, e.g. "food"

.. envvar:: TESTLIB_REDIRECT_OUTPUT

Redirects stdout and stderr to temporary files

By default, all tests are run with ``set -xv`` for debugging purposes when a tests fails. This output is stored in a out/err/xtrace file temporarily and only displayed if a tests fails. You can set this variable to control the streams to always output.

:Values: * ``3`` Redirect stdout, stderr, and xtrace
         * ``2`` Redirect stderr, and xtrace, but let stdout through
         * ``1`` Redirect xtrace, but let stdout and stderr through. On bash 4.0 and older, this will let xtrace through too
         * ``0`` Let everything through

:Default: ``3``

.. envvar:: TESTLIB_PS4

Optionally set a custom PS4 output for trace output on test errors. If unset, the testlib default is use: ``+${BASH_SOURCE[0]##*/}:${LINENO})\t``

:Default: *unset*

.. envvar:: TESTLIB_STOP_AFTER_FAILS

If set, stops after this many tests have fails in a single file. Instead of running the rest of the tests in a file, they are skipped.

:Default: ``0`` - Unlimited

.. function:: atexit

Function that runs at process exit

.. rubric:: Usage

Automatically called on exit by trap.

Checks to see if :func:`teardown` is defined, and calls it. :func:`teardown` is typically a function, alias, or something that makes sense to call.

.. function:: begin_test

Beginning of test demarcation

.. rubric:: Usage

Mark the beginning of a test. A subshell should immediately follow this statement.

.. seealso::
  :func:`end_test`

.. function:: begin_expected_fail_test

Beginning of expected to fail test demarcation

.. rubric:: Usage

Define the beginning of a test that is expected to fail. Failures may only occur in "fail zones" denoted by :func:`begin_fail_zone` and :func:`end_fail_zone`. When a test fails in a fail zone, it is counted as a success. If a test that was expected to fail never fails, it counts as an "unexpected success" rather than a normal success. If the test fails outside a fail zone, it is marked as a failure.

The typical use case for expecting a failure is when a known bug is being tested and has not or cannot be fixed yet. For this reason, a success is counted as an "unexpected success" rather than a normal success. While unexpected successes do not cause a non-zero exit code, they can easily be noticed as something that should be checked out.

.. note::
  :func:`end_fail_zone` is not typically needed.

.. function:: begin_required_fail_test

Beginning of required to fail test demarcation

.. rubric:: Usage

Define the beginning of a test that is required to fail. Failures may only occur in "fail zones" denoted by :func:`begin_fail_zone` and :func:`end_fail_zone`. When a test fails in a fail zone, it is counted as a success. If a test that was required to fail never fails, it counts as a failure. If the test fails outside a fail zone, it is marked as a failure.

The typical use case for requiring a failure is testing that an exception is raise under proper circumstances.

.. note::
  :func:`end_fail_zone` is not typically needed.

.. function:: begin_fail_zone

Start a fail zone

In practice, having a test that is expected or required to fail leads to the possibility of a test failing somewhere you don't expect it to. For this reason, fail zones must be denoted in order for :func:`begin_expected_fail_test` and :func:`begin_required_fail_test` tests to succeed, and the failures must only occur in a fail zone. If a failure happens outside the fail zone, the test will be marked as a failure with the message ``SHOULD HAVE FAILED ELSEWHERE``.

Typically this will be called right before the last last line of a test

.. rubric:: Example

.. code-block:: bash

  begin_expected_fail_test "Some test"
  (
    setup_test
    # Failing here would result in failure
    begin_fail_zone
    false is ok here
  )

.. function:: end_fail_zone

End a fail zone

While not common, it might be possible to have a test that is likely to fail in one of many places; for this reason a fail zone can be turned off, before being turned on again.

.. rubric:: Example

.. code-block:: bash

  begin_required_fail_test "Some test"
  (
    setup_test
    true something

    begin_fail_zone
    maybe_false
    end_fail_zone

    true again

    begin_fail_zone
    false_if_other_was_not
    # end_fail_zone # not required at the end of the test, but won't hurt
  )
  end_test

.. function:: setup_test

Sets up the test

Once inside the () subshell, typically set -eu needs to be run, then other things such as checking to see if a test should be skipped, etc. need to be done. This is all encapsulated into :func:`setup_test`. This is required; without it, :func:`end_test` will know you forgot to call this and fail.

This is also the second part of creating a skippable test.

You are free to change "set -eu" after :func:`setup_test`, should you wish.

.. rubric:: Usage

Place at the beginning of a test

.. rubric:: Example

.. code-block:: bash

  skip_next_test
  begin_test "Skipping test"
  (
    setup_test
    #test code here
  )

.. seealso::
  :func:`skip_next_test`

.. function:: end_test

End of a test demarcation

.. rubric:: Usage

Mark the end of a test. Must be the first command after the test group, or else the return value will not be captured successfully.

.. seealso::
  :func:`begin_test`

.. function:: skip_next_test

Function to indicate the next test should be skipped

This is the first part of creating a skippable test, used in conjunction with :func:`setup_test`

.. rubric:: Example

.. code-block:: bash

  For example, skip if docker command not found

    if ! command -v docker &> /dev/null; then
      skip_next_test
    fi
    begin_test "My test"
    (
      setup_test
      [ "$(docker run -it --rm ubuntu:14.04 echo hi)" = "hi" ]
    )

.. seealso::
  :func:`setup_test`

.. note::
  This must be done outside of the test, or else the skip variable will not be set and detected by :func:`end_test`

.. function:: track_touched_files

Start tracking touched files

After running :func:`track_touched_files`, any call to :func:`ttouch` will cause that file to be added to the internal list (touched_files). Just prior to the teardown phase, all of these files will be automatically removed for your convenience.

:func:`ttouch` should be used in cases where a file cannot be redirected to :envvar:`TESTDIR` or :envvar:`TRASHDIR`

.. rubric:: Example

.. code-block:: bash

  setup()
  {
    track_touched_files
  }
  begin_test Testing
  (
    ttouch /tmp/hiya
  )
  end_test

.. rubric:: Usage

Should be called before the :func:`begin_test` block, not inside. Inside a () subshell block will not work. Setup is the logical place to put it.

.. rubric:: Bugs

Does not work in sh, only ``bash``. Uses array, and I didn't want to make this use a string instead.

Not thread safe. Use a different file for each thread

.. seealso::
  :func:`cleanup_touched_files`

.. function:: ttouch

Touch function that should behave like the original touch command

.. seealso::
  :func:`track_touched_files`

.. function:: cleanup_touched_files

Delete all the touched files

At the end of the last test, delete all the files in the array