J.U.S.T. Mirror to an Air-gapped Repository

just_git_airgap_repo.bsh

While creating a git mirror is as simple as git clone --mirror, unfortunately this command does not support git submodules or lfs. These functions help in creating and subsequently cloning a mirror of a project with submodules and/or git lfs.

Example

Assume we have a just project called project_A stored at https://git-server.com/projectA/project_A.git and vsi_common is a submodule of it.

This repository is recursively cloned to /src; it looks like

project_A/
  .gitmodules
  Justfile
  setup.env
  external/vsi_common/               # submodule
  external/vsi_common/docker/recipes # sub-submodule

Before this repository can be mirrored (which is not the same thing as a clone) and pushed to a new air-gapped git server, first, a little setup is necessary.

The need for this setup is due to a chicken-and-egg problem. chicken) If a developer tried to simply clone the mirrored repository in the air-gapped environment, the submodules would fail to update because their URLs cannot be reached in that environment. Instead, the submodules must first be re-configured; then, they can be updated (recursively). This is handled by the git_airgap-submodule-update just target. egg) However, this obviously cannot be called until the vsi_common submodule is itself initialized and updated.

To accomplish this, a small function, git_airgap_submodule_helper.bsh git_airgap_submodule_update, is (orphan) committed to the air-gapped repository so that it is available for this task.

As no just functions can be called yet, this function should be called in the file specified by JUST_SETUP_SCRIPT (typically called setup.env). For example, the setup.env script, which typically looks like:

export JUST_SETUP_SCRIPT="$(basename "${BASH_SOURCE[0]}")"
source "$(dirname "${BASH_SOURCE[0]}")/external/vsi_common/env.bsh"

would become:

export JUST_SETUP_SCRIPT="$(basename "${BASH_SOURCE[0]}")"
if [ ! -f "$(dirname "${BASH_SOURCE[0]}")/external/vsi_common/env.bsh" ]; then
  echo "'just' could not be loaded. Trying to setup the repository as an"
  echo "air-gapped repository"
  # source the contents of repo_map.env (in a bash 3.2 compatible way)
  source /dev/stdin <<< \
      "$(git show origin/__just_git_mirror_info_file:repo_map.env 2>/dev/null || :)"
  if ! declare -Fx git_airgap_submodule_update; then
    echo "ERROR the vsi_common submodule could not be found!"
    return 1
  fi
  git_airgap_submodule_update external/vsi_common
fi
source "$(dirname "${BASH_SOURCE[0]}")/external/vsi_common/env.bsh"

This repository is now setup and can be mirrored and pushed to a new air-gapped git server:

just git export-repo-guided - This target asks a series of questions and then mirrors the repository and its submodules (recursively). For this example, when prompted, we will, “Create a new mirror from a remote’s URL”, and save the the mirrored repositories to the output directory, {output_dir}. In this case, because there is only a single remote, origin, and a single branch, main, they are chosen automatically.

Note

The mirror is created from (the URL of) the remote—not directly from this clone itself.

Transfer the archive at {output_dir}/transfer_{date}.tgz to your destination.
On the destination, create a directory, e.g., {transfer_dir}, and move the archive into it
Extract the archive (the archive will extract directly into this directory)

In {transfer_dir}, edit repo_map.env and set the just_git_airgap_repo.bsh create_repo_map JUST_GIT_AIRGAP_MIRROR_URL environment variable. For example,

JUST_GIT_AIRGAP_MIRROR_URL=https://git-airgap.com/projectA

Initialize bare repositories on the air-gapped git server for project_A and its submodules. The list of all submodules can be found in the repo_map.env file, which maps between the submodule’s path and its new URL. In this example, these should be located at

https://git-airgap.com/projectA/project_A.git

https://git-airgap.com/projectA/vsi_common.git

https://git-airgap.com/projectA/recipes.git

Note

Depending on your permission level and the git client on the air-gapped server, you may be allowed to create these repositories on demand when pushing them. This is, for example, possible with GitLab.

source setup.env
just git import-repo - Push the mirrored repository and all its submodules to the new git server as defined by repo_map.env

Note

You must have permissions on the server to (force) push to any branch; for example, in GitLab, no branches can be protected against the user doing the transfer.

Note

A tag is left on all branches when they are transferred so if a branch is force pushed, the old branch can be recovered if necessary.

Subsequent updates can be pushed to the repositories using much the same process, although with a few variations:

In step 1, when prompted, choose “Base the archive off an existing airgap mirror”; in this example, {output_dir}.
- In addition to another full archive, an incremental archive, transfer_{date}_transfer_{previous_date}.tgz, is also created (if supported). This incremental archive may be significantly smaller than the full archive.
- If the incremental archive is transferred to the destination in step 2, then
  
  In step 3, move the archive into the same directory as before; in this example, {transfer_dir}.
  
  In step 4, extract this incremental archive on top of the existing mirror.
- If instead the full archive is transferred in step 2, then
  
  In step 3, the archive can be moved into the same directory as before, {transfer_dir}, although it doesn’t have to be.
In step 5, unless there is a new submodule being mirrored, the repositories are already configured.

A developer can then run:

git clone https://git-airgap.com/projectA/project_A.git - Note: non-recursive
source setup.env
just git airgap-submodule-update - Clone submodules recursively from the new mirror

Note

If a new submodule is added to the repository then just git airgap-submodule-update must be re-run instead of the standard git command, git submodule update, which will fail because the location of the submodules as defined in the .gitmodules file is not correct here—instead, the submodule must first be re-configured to point to the URL specified by repo_map.env. (This command is essentially doing a custom git submodule sync and git submodule update. Accordingly, it must also be run if the URL of the submodule is changed, which could happen, e.g., if the location of the airgap’ed server changed. Note that in this case, an updated repo_map.env file would need to be committed to the old airgap’ed server.)

Limitations - There are a few limitations with a mirrored repository:

While the mirrored repository is a proper git repository, care must be taken to ensure subsequent (incremental) mirrors are successful: specifically, the transferred branches must remain read-only. However, additional branches/tags can be created as long as their names won’t clash with those from the host repository.
git_mirror git_mirror_main, and by extension this plugin, does not mirror all submodules that have ever been part of the repo, only those from a specific branch/SHA/tag you specify (git’s init.defaultBranch by default). Consequently, checking out another version of the repository with a different version of the .gitmodules file in which a submodule has been deleted or renamed may cause the git_airgap-submodule-update to fail because the submodule’s remote URL could not be re-configured to point to the mirror.

create_repo_map

Create the contents of the repo_map.env file

Create the repository mapping such that, once the JUST_GIT_AIRGAP_MIRROR_URL variable is defined, it can be sourced by git_mirror git_push_main and git_mirror git_clone_main.

Argument:: $1 - The project’s repository name (e.g., vsi_common)
Parameters:: [ASSOCIATIVE_REPO_MAP] - Set to a value to create the repo map as an associative-array, repos, as opposed to two partitioned arrays (which is bash 3.2 compatible): repo_urls and repo_paths. (Default: unset; i.e., partitioned)
Output:: stdout - The contents of the repo_map.env file, the file that maps between the submodule’s path and its new URL. For example,

# The urls are specified with the variable JUST_GIT_AIRGAP_MIRROR_URL,
# which must be set to the mirrored repositories' new location on the
# air-gapped git server. Delay setting this variable until the archive has
# been moved to the destination in case the information must be controlled
JUST_GIT_AIRGAP_MIRROR_URL=

repo_paths=(
  ./
  ./docker/recipes
)
repo_urls=(
  "${JUST_GIT_AIRGAP_MIRROR_URL}/vsi_common.git"
  "${JUST_GIT_AIRGAP_MIRROR_URL}/recipes.git"
)

If, for example, JUST_GIT_AIRGAP_MIRROR_URL was set to https://git-server/projectA, these urls would expand to:

https://git-server/projectA/vsi_common.git

https://git-server/projectA/recipes.git

Note

One limitation of this function occurs when two dependencies are only differentiated by their organization, e.g., projectC/dependency.git and projectD/dependency.git; then, these dependencies will both be mirrored to the same URL: “${JUST_GIT_AIRGAP_MIRROR_URL}/dependency.git”. One option is to override (or extend) this function to fixup the repository map as needed. Alternatively, if significant customization of the repository map is required, it can be created manually and used directly by git_mirror git_mirror_main and family.

JUST_GIT_AIRGAP_MIRROR_URL

A variable used in the repo_map.env file created by create_repo_map to specify the mirrored repositories’ new location on the air-gapped git server

orphan_commit_repo_map

Orphan commit the repo_map.env file

Given a repository mapping like the one produced by create_repo_map, make an orphan commit in the repository (on a branch named __just_git_mirror_info_file by default) to a file named repo_map.env.

Arguments:

$1 - The contents of the repo_map.env file; i.e., the file that maps between the submodule’s path and its new URL
[$2] - The name of the branch on which to make the orphan commit. Default: __just_git_mirror_info_file

add_import-repo_just_project

Create a simple just project in the prep_dir

This function creates a simple just project (a README.md, setup.env, and Justfile that includes this plugin) in the air-gapped mirror cache (the prep_dir) created by the git_export-repo just target. This just project can be used to push (aka import) the mirrored repositories to their respective air-gapped git server by using the git_import-repo just target.

Arguments:: $1 - The output directory (prep_dir) that caches the mirrored repositories and archive to be transferred
Output:: ${1}/{README.md,setup.env,Justfile}

relocate_git_defaultify

Git relocate plugin for just