Compatibility Variables

compat.bsh

A collection of variables to help cope with the differences between different operating systems and versions.

load_vsi_compat

Typically, this does not need to be called directly. load_vsi_compat is run automatically when compat.bsh is sourced. However, if you need to reload the variables, e.g. VSI_SED_COMPAT is set and you want to get the forced compatibility flags, then you could call load_vsi_compat, but this would be extremely rare.

Sed flags

While most flags between GNU and BSD (macOS) sed are fairly compatible, a few create issues in corner cases.

VSI_SED_COMPAT: Force bsd or gnu mode when loading the sed flags

sed_flag_rE

Flag to enable extended regex support in sed. The GNU flag is -r, while the BSD flag is -E. macOS does not support -r and GNU sed prior to 4.2 does not accept -E. Using this variable should always yield the correct flag.

Example

sed "-${sed_flag_rE}" 's|foo(.*)|bar\1.|'

sed_flags_i

When using inplace replacement on sed, the macOS version requires an argument to the -i flag, the extension added to the output file. The GNU version does not support this. Using this flag will perform an inplace replacement with no added extension. The sed_flags_i cannot be combined with other flags (see example below).

Example

sed "${sed_flags_i[@]}" 's|foo|bar|' some_file.txt

# Not ok
sed "${sed_flags_i}" ...
# Not ok
sed -"${sed_flags_i[@]}" ...
# Not ok
sed "${sed_flags_i[@]}"n ...
# This is ok
sed "${sed_flags_i[@]}" -n ...

Date

The following variables will help cope with the difference in the date command as seamlessly as possible.

date_feature_nanoseconds

Non-GNU versions of date do not support the nanosecond sequence for a date format (e.g. BSD and busybox).

Value:

0 - Supports nanosecond precision %N
1 - Does not support nanoseconds

Bash

The following variables will help cope with the difference in bash versions as seamlessly as possible.

bash_feature_posix_process_substitution

In bash 5.0 and older, process substitution isn’t allowed when in posix mode. This issue can arise if a function contains non-posix commands and is exported before calling bash --posix. Bash will complain about each and every function that is non-posix.

Value:

0 - Bash does not allow process substitution in posix mode
1 - Bash allows process substitution in posix mode

function foo()
{
  cat <(echo hi)
}
export -f foo
bash --posix

One solution is to unexport any functions that might be causing an issue. Another is to ignore the error messages, and start bash without set -e and then enable set -e after starting the new shell.

bash_behavior_declare_array_quote

In bash version 4.3 and older, declare -p of an array adds an extra ' around the array. This variable stores the state of that variable

Value:

(null) - Bash 4.4 or newer
' - Bash 4.3 or older

bash_feature_parameter_transformation

Does bash support Parameter Transformations, e.g. ${var@a}

Value:

0 - Bash does support parameter transformations (Bash 4.4 or newer)
1 - Bash does not support parameter transformations

bash_feature_declare_array_escape_special_characters

Starting in bash 4.4, declare -p of an array, will escape special characters such as tab (\t) and newline (\n)

Value:

0 - Declare will escape special characters for arrays
1 - Declare uses literal versions of special characters, but still escapes quotes as \"

bash_bug_exported_function_corrupt_bash_source

Older versions of bash have a bug where exported functions can corrupt the BASH_SOURCE environment variable, and possibly BASH_LINENO too. A workaround is not always possible.

Value:

0 - Bash source bug is present
1 - Bash source bug is not present

Example

for x in 3.2 4.0 4.1 4.2 4.3 4.4 5.0 5.1; do
  docker run -it --rm "bash:${x}" bash -c "
      echo 'blah(){ declare -p BASH_SOURCE; }
            export -f blah
            bash /tmp/bar' > /tmp/foo
      echo 'declare -p BASH_SOURCE
            blah
            declare -p BASH_SOURCE' > /tmp/bar
      bash /tmp/foo"; done

# Bash 4.3
# declare -a BASH_SOURCE='([0]="/tmp/bar")'
# declare -a BASH_SOURCE='([1]="/tmp/bar")'
# declare -a BASH_SOURCE='()'
# Bash 4.4
# declare -a BASH_SOURCE=([0]="/tmp/bar")
# declare -a BASH_SOURCE=([0]="environment" [1]="/tmp/bar")
# declare -a BASH_SOURCE=([0]="/tmp/bar")

bash_feature_declare_name_reffing

Do bash variables support name reffing

Value:

0 - Bash does support name reffing (Bash 4.3 or newer)
1 - Bash does not support name reffing

bash_feature_declare_print_function

Does declare support printing a function via declare -pf function_name. Work around: use declare -F to list all functions and type function_name or command -V function_name, however the first line needs to be removed.

Value:

0 - declare supports printing a function
1 - declare does not support printing a specific function.

bash_bug_local_shadow_exported_variable

In bash 4.2 and older, a local variable can shadow an exported variable, obscuring it from children processes. This is a confusing behavior that has to occasionally be corrected.

Value:

0 - Local variables shadow exported variable in children processes
1 - Local variables do not shadow exported variable

Example

export x=12
function foo()
{
  local x
  declare -p x        # in the x=11 case, this is 11?!
  # export x          # Doesn't make a difference. x is already exported
  compgen -A export x # compgen always sees x, shadowed or not.
  bash -c "declare -p x"
}
foo
# x will be undefined when bash_bug_local_shadow_exported_variable is 0
# x will be 12 when bash_bug_local_shadow_exported_variable is 1
export -f foo; x=11 bash -c foo # Same result as above
x=13 foo # x is always 13 and in the bash child too. No explanation for this behavior

bash_bug_allocate_xtracefd

In Bash 4.1 and 4.2, there is a bug where the following command does not work right:

exec {BASH_XTRACEFD}>"somefile"

BASH_XTRACEFD’s value is set, but whatever triggers BASH_XTRACEFD to work, is not triggered.

Note

BASH_XTRACEFD and automatic file descriptor allocation are not available features in bash 4.0 or older, but bash_bug_allocate_xtracefd will still be set to 0

bash_behavior_pattern_substitution_slash_escape_with_single_quote

A pattern substitution with single quotes and backslashes is hard to get right. In bash 4.2 and older, if the string contains backslashes, the results of pattern substitution changes when quotes are added around the expression. In bash 4.3, surrounding quotes don’t affect the substitution, however, extra slashes are needed if the string contains a single quote.

When surrounding quotes are always used, bash_behavior_pattern_substitution_slash_escape_with_single_quote gives a way to determine if extra back slashes are needed.

Value:

‘0’ - On bash 4.3 and newer
‘1’ - On bash 4.2 and older

# Bash 4.2
$ (FOOBAR="o'o"; x="${FOOBAR//\'/\\\'}"; declare -p x)
declare -- x="o\\\\'o"
$ (FOOBAR="o'o"; x=${FOOBAR//\'/\\\'}; declare -p x) # Notice the no quotes after =
declare -- x="o\\'o"

# Bash 4.3
$ (FOOBAR="o'o"; x=${FOOBAR//\'/\\\\\'}; declare -p x) # Notice the no quotes after =
declare -- x="o\\\\'o"
$ (FOOBAR="o'o"; x="${FOOBAR//\'/\\\\\'}"; declare -p x)
declare -- x="o\\\\'o"

# Compatible way
$ FOOBAR="o'o"
$ if bash_behavior_pattern_substitution_slash_escape_with_single_quote; then
    x="${FOOBAR//\'/\\\\\'}"
    y="${FOOBAR//\'/\\\'}"
  else
    x="${FOOBAR//\'/\\\'}"
    y="${FOOBAR//\'/\'}"
  fi
$ declare -p FOOBAR x y
declare -- FOOBAR="o'o"
declare -- x="o\\\\'o"
declare -- y="o\\'o"

bash_feature_declare_global

Does bash support declaring global variables with the declare command.

Value:

0 - Bash supports declare -g
1 - Bash does not support declare -g

bash_behavior_all_traps_display_inherited

When a subshell is spawned, the ERR/RETURN/DEBUG traps always show up in trap -p even if they are not actually inherited. (set -E and set -T control whether these traps are actually inherited or not). This is an annoying behavior bash has and it makes determining which traps are actually active impossible to determine. Starting in bash 4.2, all signals started observing this behavior. This was done on purpose, but really makes tracking traps harder.

bash_feature_printf_array_assignment

Does printf -v support storing to array variables

Value:

0 - printf does support storing to arrays (Bash 4.1 or newer)
1 - printf only supports storing to non-array variables

bash_feature_allocate_file_descriptor

Does exec {varname}>&1 work.

Value:

0 - exec can auto allocate an unused file descriptor
1 - File descriptors must be manually allocated.

See also

file_tools.bsh find_open_fd

bash_feature_xtracefd

Does bash support the BASH_XTRACEFD environment variable

Value:

0 - Bash supports BASH_XTRACEFD
1 - Bash does not support BASH_XTRACEFD

bash_feature_associative_array

Does bash have associative arrays

Value:

0 Bash does have associative arrays (4.0 or newer)
1 Bash does not have associative arrays (3.2)

bash_feature_bashpid

Does bash have a BASHPID variable. This is more accurate than $$ when it comes to subshells.

Value:

0 Bash sets BASHPID
1 Only $$ is set, see signal_tools.bsh set_bashpid

bash_feature_case_modification

Does bash support ${x^}/${x^^}/${x,}/${x,,} parameter substitution

Value:

0 Bash supports case modification
1 Bash does not supports case modification

bash_bug_unassigned_variable_set_to_null

There is a bug, fixed in bash 4.0, that causes an unassigned variable to be set to null

Value:

0 Bash has the bug
1 An unassigned variable works as expected

Example

$ for x in 3.2 4.0 4.1 4.2 4.3 4.4 5.0; do
    docker run -it --rm bash:${x} bash -c \
      'function foo() { \
         local y; \
         echo "y=${y+y_set}"; \
       }; \
       foo; \
       echo $?';
  done
# In bash 3.2:
# y=y_set
# 0
# In bash >= 4:
# y=
# 0

bash_bug_uncaught_unbound_on_exit_trap

There is a bug in bash 3.2, where set -u will not return non-zero for an unbound variable if an EXIT trap is set. Unfortunately, there is no way to detect this happens in the EXIT trap code, nor is there any way to cope with it. This means an unbound variable could potentially go undetected.

This is probably related to a similar phenomena where and error code is lost by the trap being called.

Value:

0 Bash has the bug
1 Unbound variables error as expected

There is a bug in bash 3.2, where set -e will not trigger an error in the parent process when a subshell command fails. The subshell does trigger the error as expected, but it is not propagated to the parent, even though the return code is.

Value:

0 Bash has the bug
1 Error from subshell statements are thrown as expected

Example

set -e
echo hi
(
  echo inner
  false
  echo "you won't ever see this"
)
echo "You will only see this on some bash 3.2: $?"

bash_bug_bash_env_process_substitution

With most bash 3.2’s, using process substitution for the BASH_ENV env variable does not work as expected (for some reason.) A work around is to write to a temporary file instead. Since this can be expensive to test, it is not evaluated into a variable on load. Instead, this is a function that is called. The result is cached so that is does not need to be tested again. The feature has less to do with the exact version of bash, but rather how it is compiled or patched.

Return Value:

0 - BASH_ENV does not support process substitution
1 - BASH_ENV works normally

bash_feature_rcfile

Depending on how bash is compiled (-DSYS_BASHRC), bash will use or ignore the --rcfile argument. This has to be tested by actually seeing if this works, so it is a function that is called on demand, instead of on compat.bsh load.

Return Value:

0 - Bash supports --rcfile
1 - Bash ignored --rcfile

__bash_bug_at_expansion_null_ifs

There is a bug, fixed in bash 4.4, that causes ${@+”${@}”} not to expand to multiple words when IFS is set to the empty string.

Return Value:

0 - ${@} does not expand as expected when the IFS is set to ‘’
1 - ${@} expansion works as expected

See also

https://tiswww.case.edu/php/chet/bash/CHANGES Section ‘bash-4.4-rc1’ Change w

bash_feature_bashpid_read_only

The BASHPID variable is only only protected as read only in bash 4. Bash 5 releases this restriction (but does not release the restriction for any other read only variable)

Value:

0 BASHPID is read only
1 BASHPID is not read only

bash_behavior_regex_special_characters_non_literal

Most instances of bash require special characters to be literal in regular expressions (e.g. $'\n'). However, on operating systems such as alpine, a special character will be matched based in its non-literal form (e.g. "\n"). This has nothing to do with the version of bash, but rather how it was compiled (regex might come from g++’s implementation).

bash_behavior_regex_special_characters_non_literal will be set if the non-literal form works. Currently, the literal form always appears to work.

Value:

0 - Non literal form works (e.g. "\t")
1 - Non literal form does not work

bash_bug_regex_null_string

Most instances of bash allow a null string to be a valid regular expression, basically any string matches. However, on operating systems such as macOS, it fails to compile the regular expression, and does not match correctly, giving the opposite answer as all other operating systems. This has nothing to do with the version of bash, but rather how it was compiled (regex might come from g++’s implementation).

Value:

0 - The null string for a pattern does not match correctly
1 - The null string for a pattern does match correctly

bash_bug_declare_fails_local_declared_unset_variable

Bash 4.3 has a bug where declare -p var fails if a local variable is declared, but unset. The workaround is to find the variable in declare -p

Value:

0 - Has declare fails bug
1 - Does not have the declare fails bug

bash_bug_declare_fails_global_declared_unset_variable

Bash 4.0 to 4.3 has a bug where declare -p var fails if a global variable is declared, but has no set value. The workaround is to find the variable in declare -p

Value:

0 - Has declare fails bug
1 - Does not have the declare fails bug

bash_bug_declared_unset_value

Bash 3.2 through 4.1 has a confusing behavior where unset declared variables appear to be set to null "" instead of unset, in the output of declare -p. However these versions of bash do indeed have an “unset but declared state”, it just can’t be queried using the output of declare -p.

Value:

="" - The characters seen after declare varname
(null)

Example

declare x
if declare -p x &> /dev/null; then
  assert_str_eq "$(declare -p x)" "declare -p x${bash_bug_declared_unset_value}"
fi

# How to detect if variable is unset but declared?

Note

In Bash 3.2, declare -p var doesn’t look the same as declare -p

Detecting declared unset variables

Unfortunately, there is no fool proof simple way to detect a declared unset variable. The ${var+set} parameter substitution will let you know if a a variable is unset, and using if bash_bug_declare_fails_local_declared_unset_variable/bash_bug_declare_fails_global_declared_unset_variable is 1, then declare -p var_name will let you know it is declared. If either of those bugs are set to 0, then you have to fall back on declare -p and match the regex $'(^|\n)declare -- '"${var_name}${bash_bug_declared_unset_value}"$'(\n|$)' to determine if it was declared. Unfortunately, in Bash 4.3 it is possible for another variable to be set to a value that can falsely match the regex. This is highly unlikely to happen by accident, but can be worked around using some sed (specific to bash 4.3’s declare -p output)

Bash 4.3 Example

$ declare x_15
$ x_13=$'foo\ndeclare -- x_14\nbar'
$ declare -p | grep x_15
declare -- x_15
$ declare -p | grep x_14
declare -- x_14
$ compgen -A variables x_1
x_13
$ declare -p | sed -E ':combine
                       $bdone
                       N
                       bcombine
                       :done
                       # Remove all escaped quotes
                       s|\\"||g
                       # Remove all values, so everything looks like a null value (and is guaranteed to be one line now)
                       s|"[^"]*"|""|g' | grep x_
declare -- x_13=""
declare -- x_15

bash_bug_substitute_empty_funcname

FUNCNAME is the only builtin array in bash that is likely to be declared but unset (when stack depth is zero). In bash 4.3 and 4.4, there is a bug where an using parameter expansion on FUNCNAME when unset will incorrectly expand (e.g. ${FUNCNAME[@]+example}). This could be due to funcname not really being empty in global scope, but just doesn’t show up as populated until in its first function scope

Note

Typing this into an interactive bash session will not show this bug. It has to be in a file that is sourced or run for this bug to show up.

bash_bug_ifs_array_slice_expansion

In some versions of bash (3.2 is the only known version), @ expanding an array and slicing it when IFS is not space will fail to expand into multiple arguments.

Value:

0 - Bug present
1 - Not bugged

Example

$ x=(11 22 33 44)
$ IFS=x
$ z=("${x[@]:2}")
# @ expansion should be unaffected by IFS, unlike * expansion
declare -a z='([0]="33 44")'       # Wrong
declare -a z='([0]="33" [1]="44")' # Right

Git

The following variables will help cope with the difference in git versions as seamlessly as possible.

git_bug_submodule_path_with_special_characters

There is a bug in git, fixed in git 1.8.3.3, that caused many git submodule operations not to work on a submodule at a path whose name contains a tab or ” (double quote) (and presumably other characters). See the fourth entry in Fixes since v1.8.3.2 (I think).

Return Value:

0 - git does not support submodule paths containing special characters
1 - git works as expected

git_feature_support_tls

Parts of git (e.g. git-remote-http) link to libcurl, so the version of TLS it can support depends on what is in the OS

Return Value:

0 - Supports TLS Version 1.0
1 - Supports TLS Version 1.1
2 - Supports TLS Version 1.2
3 - Supports TLS Version 1.3 (Not testable yet)
125 - Unable to determine

git_feature_submodule_update_with_custom_command

In Git 1.8.4, support was added to git submodule update to run a custom command via the config key submodule.<name>.update. See the 11th entry in UI, Workflows & Features

Return Value:

0 - Supports submodule updates with a custom command
1 - Does not support submodule updates with a custom command

git_feature_cli_config_option

In Git 1.8.2, the “-c name=value” option was added to git to override configuration variable from the command line. See the seventh entry in Updates since v1.7.1

Return Value:

0 - Supports the -c option
1 - Does not support the -c option

Tar

The following variables will help cope with the difference in tar versions as seamlessly as possible.

tar_feature_incremental_backup

BSD versions of tar do not support incremental backups.

Return Value:

0 - tar supports incremental backups
1 - tar does not support incremental backups

readlink

The following variables will help cope with the difference in readlink versions as seamlessly as possible.

readlink_behavior_nonexistent_path

Darwin versions of readlink refuse to work when part of the path does not exist.

Return Value:

0 - readlink supports non-existing paths
1 - readlink does not support non-existing paths

realpath

The following variables will help cope with the difference in realpath versions as seamlessly as possible.

readpath_behavior_nonexistent_path

Darwin versions of readpath refuse to work when part of the path does not exist.

Return Value:

0 - readpath supports non-existing paths
1 - readpath does not support non-existing paths

singularity

The following variables will help cope with the difference in singularity versions as seamlessly as possible.

singularity_feature_nvccli

Determines if singularity has the nvccli flag feature. This does not guarantee nvidia-container-cli_path is set in singularity.conf

Return Value:

0 - singularity supports the –nvccli feature
1 - singularity does not support the –nvccli feature

singularity_feature_nvccli_wsl2_gpu

Determines if the singularity --nvccli flag supports WSL2 GPUs. This does not guarantee nvidia-container-cli_path is set in singularity.conf

Return Value:

0 - singularity --nvccli supports WSL2 GPUs
1 - singularity does not support WSL2 GPUS

See also

singularity_bug_writable_tmpfs_without_root:: While WSL2 GPUs may work, this bug may prevent WSL from running successfully without root

singularity_bug_writable_tmpfs_without_root

In Apptainer 1.1.0, they stared using fuse-overlay filesystems for non-root users. This introduced a bug that made using --writable-tmpfs as non-root error out (at least on WSL). This was fixed finally in 1.1.3. Singularity-CE is unaffected by this bug, as they only ever use overlay.

Value:

0 Apptainer has the bug
1 Singularity works as expected

See also

Apptainer bug: Avoid failures with --writable-tmpfs in non-setuid mode when using fuse-overlayfs versions 1.8 or greater by adding the fuse-overlayfs noacl mount option to disable support for POSIX Access Control Lists.