======================= Compatibility Variables ======================= .. default-domain:: bash .. file:: compat.bsh A collection of variables to help cope with the differences between different operating systems and versions. .. function:: load_vsi_compat Typically, this does not need to be called directly. :func:`load_vsi_compat` is run automatically when :file:`compat.bsh` is sourced. However, if you need to reload the variables, e.g. :envvar:`VSI_SED_COMPAT` is set and you want to get the forced compatibility flags, then you could call :func:`load_vsi_compat`, but this would be extremely rare. Sed flags ========= While most flags between GNU and BSD (macOS) sed are fairly compatible, a few create issues in corner cases. .. envvar:: VSI_SED_COMPAT Force ``bsd`` or ``gnu`` mode when loading the ``sed`` flags .. var:: sed_flag_rE Flag to enable extended regex support in sed. The GNU flag is ``-r``, while the BSD flag is ``-E``. macOS does not support ``-r`` and GNU sed prior to 4.2 does not accept ``-E``. Using this variable should always yield the correct flag. .. rubric:: Example .. code-block:: bash sed "-${sed_flag_rE}" 's|foo(.*)|bar\1.|' .. var:: sed_flags_i When using inplace replacement on sed, the macOS version requires an argument to the ``-i`` flag, the extension added to the output file. The GNU version does not support this. Using this flag will perform an inplace replacement with no added extension. The :var:`sed_flags_i` cannot be combined with other flags (see example below). .. rubric:: Example .. code-block:: bash sed "${sed_flags_i[@]}" 's|foo|bar|' some_file.txt # Not ok sed "${sed_flags_i}" ... # Not ok sed -"${sed_flags_i[@]}" ... # Not ok sed "${sed_flags_i[@]}"n ... # This is ok sed "${sed_flags_i[@]}" -n ... Date ==== The following variables will help cope with the difference in the ``date`` command as seamlessly as possible. .. var:: date_feature_nanoseconds Non-GNU versions of ``date`` do not support the nanosecond sequence for a date format (e.g. BSD and busybox). :Value: * ``0`` - Supports nanosecond precision ``%N`` * ``1`` - Does not support nanoseconds Bash ==== The following variables will help cope with the difference in bash versions as seamlessly as possible. .. var:: bash_feature_posix_process_substitution In bash 5.0 and older, process substitution isn't allowed when in posix mode. This issue can arise if a function contains non-posix commands and is exported before calling ``bash --posix``. Bash will complain about each and every function that is non-posix. :Value: * ``0`` - Bash does not allow process substitution in posix mode * ``1`` - Bash allows process substitution in posix mode .. code-block:: bash function foo() { cat <(echo hi) } export -f foo bash --posix One solution is to unexport any functions that might be causing an issue. Another is to ignore the error messages, and start bash without ``set -e`` and then enable ``set -e`` after starting the new shell. .. var:: bash_behavior_declare_array_quote In bash version 4.3 and older, ``declare -p`` of an array adds an extra ``'`` around the array. This variable stores the state of that variable :Value: * **(null)** - Bash 4.4 or newer * ``'`` - Bash 4.3 or older .. var:: bash_feature_parameter_transformation Does bash support Parameter Transformations, e.g. ``${var@a}`` :Value: * ``0`` - Bash does support parameter transformations (Bash 4.4 or newer) * ``1`` - Bash does not support parameter transformations .. var:: bash_feature_declare_array_escape_special_characters Starting in bash 4.4, ``declare -p`` of an array, will escape special characters such as tab (``\t``) and newline (``\n``) :Value: * ``0`` - Declare will escape special characters for arrays * ``1`` - Declare uses literal versions of special characters, but still escapes quotes as ``\"`` .. var:: bash_bug_exported_function_corrupt_bash_source Older versions of bash have a bug where exported functions can corrupt the ``BASH_SOURCE`` environment variable, and possibly ``BASH_LINENO`` too. A workaround is not always possible. :Value: * ``0`` - Bash source bug is present * ``1`` - Bash source bug is not present .. rubric:: Example .. code-block:: bash for x in 3.2 4.0 4.1 4.2 4.3 4.4 5.0 5.1; do docker run -it --rm "bash:${x}" bash -c " echo 'blah(){ declare -p BASH_SOURCE; } export -f blah bash /tmp/bar' > /tmp/foo echo 'declare -p BASH_SOURCE blah declare -p BASH_SOURCE' > /tmp/bar bash /tmp/foo"; done # Bash 4.3 # declare -a BASH_SOURCE='([0]="/tmp/bar")' # declare -a BASH_SOURCE='([1]="/tmp/bar")' # declare -a BASH_SOURCE='()' # Bash 4.4 # declare -a BASH_SOURCE=([0]="/tmp/bar") # declare -a BASH_SOURCE=([0]="environment" [1]="/tmp/bar") # declare -a BASH_SOURCE=([0]="/tmp/bar") .. var:: bash_feature_declare_name_reffing Do bash variables support name reffing :Value: * ``0`` - Bash does support name reffing (Bash 4.3 or newer) * ``1`` - Bash does not support name reffing .. var:: bash_feature_declare_print_function Does ``declare`` support printing a function via ``declare -pf function_name``. Work around: use ``declare -F`` to list all functions and ``type function_name`` or ``command -V function_name``, however the first line needs to be removed. :Value: * ``0`` - ``declare`` supports printing a function * ``1`` - ``declare`` does not support printing a specific function. .. var:: bash_bug_local_shadow_exported_variable In bash 4.2 and older, a local variable can shadow an exported variable, obscuring it from children processes. This is a confusing behavior that has to occasionally be corrected. :Value: * ``0`` - Local variables shadow exported variable in children processes * ``1`` - Local variables do not shadow exported variable .. rubric:: Example .. code-block:: bash export x=12 function foo() { local x declare -p x # in the x=11 case, this is 11?! # export x # Doesn't make a difference. x is already exported compgen -A export x # compgen always sees x, shadowed or not. bash -c "declare -p x" } foo # x will be undefined when bash_bug_local_shadow_exported_variable is 0 # x will be 12 when bash_bug_local_shadow_exported_variable is 1 export -f foo; x=11 bash -c foo # Same result as above x=13 foo # x is always 13 and in the bash child too. No explanation for this behavior .. var:: bash_bug_allocate_xtracefd In Bash 4.1 and 4.2, there is a bug where the following command does not work right: .. code-block:: bash exec {BASH_XTRACEFD}>"somefile" ``BASH_XTRACEFD``'s value is set, but whatever triggers ``BASH_XTRACEFD`` to work, is not triggered. .. note:: ``BASH_XTRACEFD`` and automatic file descriptor allocation are not available features in bash 4.0 or older, but :var:`bash_bug_allocate_xtracefd` will still be set to ``0`` .. seealso:: https://github.com/cylc/cylc-flow/pull/3572#pullrequestreview-398453037 .. function:: bash_behavior_pattern_substitution_slash_escape_with_single_quote A pattern substitution with single quotes and backslashes is hard to get right. In bash 4.2 and older, if the string contains backslashes, the results of pattern substitution changes when quotes are added around the expression. In bash 4.3, surrounding quotes don't affect the substitution, however, extra slashes are needed if the string contains a single quote. When surrounding quotes are always used, :func:`bash_behavior_pattern_substitution_slash_escape_with_single_quote` gives a way to determine if extra back slashes are needed. :Value: * '0' - On bash 4.3 and newer * '1' - On bash 4.2 and older .. code-block:: bash # Bash 4.2 $ (FOOBAR="o'o"; x="${FOOBAR//\'/\\\'}"; declare -p x) declare -- x="o\\\\'o" $ (FOOBAR="o'o"; x=${FOOBAR//\'/\\\'}; declare -p x) # Notice the no quotes after = declare -- x="o\\'o" # Bash 4.3 $ (FOOBAR="o'o"; x=${FOOBAR//\'/\\\\\'}; declare -p x) # Notice the no quotes after = declare -- x="o\\\\'o" $ (FOOBAR="o'o"; x="${FOOBAR//\'/\\\\\'}"; declare -p x) declare -- x="o\\\\'o" # Compatible way $ FOOBAR="o'o" $ if bash_behavior_pattern_substitution_slash_escape_with_single_quote; then x="${FOOBAR//\'/\\\\\'}" y="${FOOBAR//\'/\\\'}" else x="${FOOBAR//\'/\\\'}" y="${FOOBAR//\'/\'}" fi $ declare -p FOOBAR x y declare -- FOOBAR="o'o" declare -- x="o\\\\'o" declare -- y="o\\'o" .. var:: bash_feature_declare_global Does bash support declaring global variables with the ``declare`` command. :Value: * ``0`` - Bash supports ``declare -g`` * ``1`` - Bash does not support ``declare -g`` .. var:: bash_behavior_all_traps_display_inherited When a subshell is spawned, the ERR/RETURN/DEBUG traps always show up in ``trap -p`` even if they are not actually inherited. (``set -E`` and ``set -T`` control whether these traps are actually inherited or not). This is an annoying behavior bash has and it makes determining which traps are actually active impossible to determine. Starting in bash 4.2, all signals started observing this behavior. This was done on purpose, but really makes tracking traps harder. .. var:: bash_feature_printf_array_assignment Does ``printf -v`` support storing to array variables :Value: * ``0`` - ``printf`` does support storing to arrays (Bash 4.1 or newer) * ``1`` - ``printf`` only supports storing to non-array variables .. var:: bash_feature_allocate_file_descriptor Does ``exec {varname}>&1`` work. :Value: * ``0`` - ``exec`` can auto allocate an unused file descriptor * ``1`` - File descriptors must be manually allocated. .. seealso:: :func:`file_tools.bsh find_open_fd` .. var:: bash_feature_xtracefd Does bash support the ``BASH_XTRACEFD`` environment variable :Value: * ``0`` - Bash supports ``BASH_XTRACEFD`` * ``1`` - Bash does not support ``BASH_XTRACEFD`` .. var:: bash_feature_associative_array Does bash have associative arrays :Value: * ``0`` Bash does have associative arrays (4.0 or newer) * ``1`` Bash does not have associative arrays (3.2) .. var:: bash_feature_bashpid Does bash have a ``BASHPID`` variable. This is more accurate than ``$$`` when it comes to subshells. :Value: * ``0`` Bash sets ``BASHPID`` * ``1`` Only ``$$`` is set, see :func:`signal_tools.bsh set_bashpid` .. var:: bash_feature_case_modification Does bash support ``${x^}/${x^^}/${x,}/${x,,}`` parameter substitution :Value: * ``0`` Bash supports case modification * ``1`` Bash does not supports case modification .. seealso:: :func:`string_tools.bsh lowercase`, :func:`string_tools.bsh uppercase` .. var:: bash_bug_unassigned_variable_set_to_null There is a bug, fixed in bash 4.0, that causes an unassigned variable to be set to null :Value: * ``0`` Bash has the bug * ``1`` An unassigned variable works as expected .. rubric:: Example .. code-block:: bash $ for x in 3.2 4.0 4.1 4.2 4.3 4.4 5.0; do docker run -it --rm bash:${x} bash -c \ 'function foo() { \ local y; \ echo "y=${y+y_set}"; \ }; \ foo; \ echo $?'; done # In bash 3.2: # y=y_set # 0 # In bash >= 4: # y= # 0 .. var :: bash_bug_uncaught_unbound_on_exit_trap There is a bug in bash 3.2, where ``set -u`` will not return non-zero for an unbound variable if an ``EXIT`` trap is set. Unfortunately, there is no way to detect this happens in the ``EXIT`` trap code, nor is there any way to cope with it. This means an unbound variable could potentially go undetected. This is probably related to a similar phenomena where and error code is lost by the trap being called. :Value: * ``0`` Bash has the bug * ``1`` Unbound variables error as expected .. var :: bash_bug_uncaught_error_compound_subshell There is a bug in bash 3.2, where ``set -e`` will not trigger an error in the parent process when a subshell command fails. The subshell does trigger the error as expected, but it is not propagated to the parent, even though the return code is. :Value: * ``0`` Bash has the bug * ``1`` Error from subshell statements are thrown as expected .. rubric:: Example .. code-block:: bash set -e echo hi ( echo inner false echo "you won't ever see this" ) echo "You will only see this on some bash 3.2: $?" .. function:: bash_bug_bash_env_process_substitution With most bash 3.2's, using process substitution for the BASH_ENV env variable does not work as expected (for some reason.) A work around is to write to a temporary file instead. Since this can be expensive to test, it is not evaluated into a variable on load. Instead, this is a function that is called. The result is cached so that is does not need to be tested again. The feature has less to do with the exact version of bash, but rather how it is compiled or patched. :Return Value: * ``0`` - ``BASH_ENV`` does not support process substitution * ``1`` - ``BASH_ENV`` works normally .. function:: bash_feature_rcfile Depending on how bash is compiled (``-DSYS_BASHRC``), bash will use or ignore the ``--rcfile`` argument. This has to be tested by actually seeing if this works, so it is a function that is called on demand, instead of on :file:`compat.bsh` load. :Return Value: * ``0`` - Bash supports ``--rcfile`` * ``1`` - Bash ignored ``--rcfile`` .. function:: __bash_bug_at_expansion_null_ifs There is a bug, fixed in bash 4.4, that causes ${@+"${@}"} not to expand to multiple words when IFS is set to the empty string. :Return Value: * ``0`` - ``${@}`` does not expand as expected when the IFS is set to '' * ``1`` - ``${@}`` expansion works as expected .. seealso:: https://tiswww.case.edu/php/chet/bash/CHANGES Section 'bash-4.4-rc1' Change w .. var:: bash_feature_bashpid_read_only The ``BASHPID`` variable is only only protected as read only in bash 4. Bash 5 releases this restriction (but does not release the restriction for any other read only variable) :Value: * ``0`` ``BASHPID`` is read only * ``1`` ``BASHPID`` is not read only .. var:: bash_behavior_regex_special_characters_non_literal Most instances of bash require special characters to be literal in regular expressions (e.g. ``$'\n'``). However, on operating systems such as alpine, a special character will be matched based in its non-literal form (e.g. ``"\n"``). This has nothing to do with the version of bash, but rather how it was compiled (regex might come from g++'s implementation). :var:`bash_behavior_regex_special_characters_non_literal` will be set if the non-literal form works. Currently, the literal form always appears to work. :Value: * ``0`` - Non literal form works (e.g. ``"\t"``) * ``1`` - Non literal form does not work .. var:: bash_bug_regex_null_string Most instances of bash allow a null string to be a valid regular expression, basically any string matches. However, on operating systems such as macOS, it fails to compile the regular expression, and does not match correctly, giving the opposite answer as all other operating systems. This has nothing to do with the version of bash, but rather how it was compiled (regex might come from g++'s implementation). :Value: * ``0`` - The null string for a pattern does not match correctly * ``1`` - The null string for a pattern does match correctly .. var:: bash_bug_declare_fails_local_declared_unset_variable Bash 4.3 has a bug where ``declare -p var`` fails if a local variable is declared, but unset. The workaround is to find the variable in ``declare -p`` :Value: * ``0`` - Has declare fails bug * ``1`` - Does not have the declare fails bug .. var:: bash_bug_declare_fails_global_declared_unset_variable Bash 4.0 to 4.3 has a bug where ``declare -p var`` fails if a global variable is declared, but has no set value. The workaround is to find the variable in ``declare -p`` :Value: * ``0`` - Has declare fails bug * ``1`` - Does not have the declare fails bug .. var:: bash_bug_declared_unset_value Bash 3.2 through 4.1 has a confusing behavior where unset declared variables appear to be set to null ``""`` instead of unset, in the output of ``declare -p``. However these versions of bash do indeed have an "unset but declared state", it just can't be queried using the output of ``declare -p``. :Value: * ``=""`` - The characters seen after ``declare varname`` * **(null)** .. rubric:: Example .. code-block:: bash declare x if declare -p x &> /dev/null; then assert_str_eq "$(declare -p x)" "declare -p x${bash_bug_declared_unset_value}" fi # How to detect if variable is unset but declared? .. note:: In Bash 3.2, ``declare -p var`` doesn't look the same as ``declare -p`` Detecting declared unset variables ---------------------------------- Unfortunately, there is no fool proof simple way to detect a declared unset variable. The ``${var+set}`` parameter substitution will let you know if a a variable is unset, and using if :var:`bash_bug_declare_fails_local_declared_unset_variable`/:var:`bash_bug_declare_fails_global_declared_unset_variable` is ``1``, then ``declare -p var_name`` will let you know it is declared. If either of those bugs are set to ``0``, then you have to fall back on ``declare -p`` and match the regex ``$'(^|\n)declare -- '"${var_name}${bash_bug_declared_unset_value}"$'(\n|$)'`` to determine if it was declared. Unfortunately, in Bash 4.3 it is possible for another variable to be set to a value that can falsely match the regex. This is highly unlikely to happen by accident, but can be worked around using some sed (specific to bash 4.3's ``declare -p`` output) .. rubric:: Bash 4.3 Example .. code-block:: bash $ declare x_15 $ x_13=$'foo\ndeclare -- x_14\nbar' $ declare -p | grep x_15 declare -- x_15 $ declare -p | grep x_14 declare -- x_14 $ compgen -A variables x_1 x_13 $ declare -p | sed -E ':combine $bdone N bcombine :done # Remove all escaped quotes s|\\"||g # Remove all values, so everything looks like a null value (and is guaranteed to be one line now) s|"[^"]*"|""|g' | grep x_ declare -- x_13="" declare -- x_15 .. var:: bash_bug_substitute_empty_funcname ``FUNCNAME`` is the only builtin array in bash that is likely to be declared but unset (when stack depth is zero). In bash 4.3 and 4.4, there is a bug where an using parameter expansion on ``FUNCNAME`` when unset will incorrectly expand (e.g. ``${FUNCNAME[@]+example}``). This could be due to funcname not really being empty in global scope, but just doesn't show up as populated until in its first function scope .. note:: Typing this into an interactive bash session will not show this bug. It has to be in a file that is sourced or run for this bug to show up. .. var:: bash_bug_ifs_array_slice_expansion In some versions of bash (3.2 is the only known version), ``@`` expanding an array and slicing it when IFS is not space will fail to expand into multiple arguments. :Value: * ``0`` - Bug present * ``1`` - Not bugged .. rubric:: Example .. code-block:: bash $ x=(11 22 33 44) $ IFS=x $ z=("${x[@]:2}") # @ expansion should be unaffected by IFS, unlike * expansion declare -a z='([0]="33 44")' # Wrong declare -a z='([0]="33" [1]="44")' # Right Git === The following variables will help cope with the difference in git versions as seamlessly as possible. .. function:: git_bug_submodule_path_with_special_characters There is a bug in git, fixed in git 1.8.3.3, that caused many ``git submodule`` operations not to work on a submodule at a path whose name contains a tab or " (double quote) (and presumably other characters). See the fourth entry in `Fixes since v1.8.3.2 `_ (I think). :Return Value: * ``0`` - ``git`` does not support submodule paths containing special characters * ``1`` - ``git`` works as expected .. function:: git_feature_support_tls Parts of git (e.g. git-remote-http) link to libcurl, so the version of TLS it can support depends on what is in the OS :Return Value: * ``0`` - Supports TLS Version 1.0 * ``1`` - Supports TLS Version 1.1 * ``2`` - Supports TLS Version 1.2 * ``3`` - Supports TLS Version 1.3 (Not testable yet) * ``125`` - Unable to determine .. function:: git_feature_submodule_update_with_custom_command In Git 1.8.4, support was added to ``git submodule update`` to run a custom command via the config key submodule..update. See the 11th entry in `UI, Workflows & Features `_ :Return Value: * ``0`` - Supports submodule updates with a custom command * ``1`` - Does not support submodule updates with a custom command .. function:: git_feature_cli_config_option In Git 1.8.2, the "-c name=value" option was added to ``git`` to override configuration variable from the command line. See the seventh entry in `Updates since v1.7.1 `_ :Return Value: * ``0`` - Supports the -c option * ``1`` - Does not support the -c option Tar === The following variables will help cope with the difference in tar versions as seamlessly as possible. .. function:: tar_feature_incremental_backup BSD versions of ``tar`` do not support incremental backups. :Return Value: * ``0`` - ``tar`` supports incremental backups * ``1`` - ``tar`` does not support incremental backups readlink ======== The following variables will help cope with the difference in readlink versions as seamlessly as possible. .. function:: readlink_behavior_nonexistent_path Darwin versions of ``readlink`` refuse to work when part of the path does not exist. :Return Value: * ``0`` - ``readlink`` supports non-existing paths * ``1`` - ``readlink`` does not support non-existing paths realpath ======== The following variables will help cope with the difference in realpath versions as seamlessly as possible. .. function:: readpath_behavior_nonexistent_path Darwin versions of ``readpath`` refuse to work when part of the path does not exist. :Return Value: * ``0`` - ``readpath`` supports non-existing paths * ``1`` - ``readpath`` does not support non-existing paths singularity =========== The following variables will help cope with the difference in singularity versions as seamlessly as possible. .. function:: singularity_feature_nvccli Determines if singularity has the nvccli flag feature. This does not guarantee nvidia-container-cli_path is set in singularity.conf :Return Value: * ``0`` - ``singularity`` supports the --nvccli feature * ``1`` - ``singularity`` does not support the --nvccli feature .. function:: singularity_feature_nvccli_wsl2_gpu Determines if the singularity ``--nvccli`` flag supports WSL2 GPUs. This does not guarantee ``nvidia-container-cli_path`` is set in ``singularity.conf`` :Return Value: * ``0`` - ``singularity`` ``--nvccli`` supports WSL2 GPUs * ``1`` - ``singularity`` does not support WSL2 GPUS .. seealso:: :func:`singularity_bug_writable_tmpfs_without_root`: While WSL2 GPUs may work, this bug may prevent WSL from running successfully without root .. function:: singularity_bug_writable_tmpfs_without_root In Apptainer 1.1.0, they stared using ``fuse-overlay`` filesystems for non-root users. This introduced a bug that made using ``--writable-tmpfs`` as non-root error out (at least on WSL). This was fixed finally in 1.1.3. Singularity-CE is unaffected by this bug, as they only ever use ``overlay``. :Value: * ``0`` Apptainer has the bug * ``1`` Singularity works as expected .. seealso:: `Apptainer bug `_ Avoid failures with ``--writable-tmpfs`` in non-setuid mode when using fuse-overlayfs versions 1.8 or greater by adding the fuse-overlayfs noacl mount option to disable support for POSIX Access Control Lists.