10.4. Edit a Tool

• Edit a Tool description
• Edit a Tool version
  • Run/Delete a Tool version
  • Set a default version
  • Commit a Tool
    • Committing features
      • Pre/post-commit hooks
      • Container size limits
• Edit a Tool settings
• Delete a Tool

To view and edit tool's attributes see 17. CP objects tagging by additional attributes.

Edit a Tool description

Select a Tool and click its name. Navigate to the Description tab of a Tool. Click the Edit buttons on the right side of the screen to modify the Short description and Full description of the Tool.

Edit a Tool version

In this tab, you can run a Tool version with custom settings or delete a Tool version. To see an example of a launching a Tool with custom settings see here.

Run/Delete a Tool version

1. Navigate to the Versions tab of a Tool.
2. Click on the arrow near the Run button.
3. Select Custom settings option to configure run parameters.
4. If you want to delete a Tool version, click the Delete button.

Set a default version

The default version of a Tool is the version, which is used if a version is not specified:

• the Run button at the top of the Tool page launches it
• the Settings tab shows and saves its settings
• a version, which has no own settings, is launched from the GUI with the settings of the default version
• a Tool image specified without a version - e.g. in pipe run -di library/ubuntu, in a pipeline configuration or in an API request - is launched with it

Until a default version is set, the latest version is used. The Run button at the top of the Tool page launches the only version of a Tool, if it has no latest one. In the Versions tab, only a version set as the default one is marked with the default label.

Users with the WRITE permission for a Tool can set another version as the default one:

1. Navigate to the Versions tab of a Tool.
2. Click the "check" button (Set as default) opposite the version, and confirm it. The version gets the default label, and its button is filled.
3. To use the latest version again, click the "filled check" button (Unset default) opposite the default version, and confirm it.

Note: if the default version is deleted, the Tool has no default version anymore, and the latest version (or the only one left) is used again, if the Tool has it. A Docker registry deletes all the versions of the same image together, so deleting another version, which has the same digest as the default one, removes the default version as well. In both cases, the deletion confirmation warns about it.

Note: if the default version is removed from the Docker registry outside of Cloud Pipeline, the Versions tab warns that it does not exist. Until it is unset by the Unset button of the warning, or another version is set as the default one, a Tool image specified without a version is launched with the missing version, which cannot be pulled.

Note: a "symlinked" Tool uses the default version of the original Tool, it can be changed only for the original Tool (see 10.8. Symlinks between tools).

Commit a Tool

Commit function allows modifying existing Tools.

1. Launch a Tool in the sleep infinity mode. See an example here.
2. SSH into it via the SSH button at the "Run logs" page.
   
3. Change something.
   Example: here we install a biopython package into this Docker image. Wait for the installation to complete!
   
4. Go back to the Logs tab and click the COMMIT button.
   • Choose a Docker registry and a Tool group.
   • Change a name for the modified image or add a new version to the current Tool by typing the version name in a separate box.
     Note: to add a new version to the existing Tool don't change the original name of the Tool!
     Note: image name and a version name should be written according to the following rules:
     • May contain lowercase letters, digits, and separators.
     • A separator is defined as a period, one or two underscores, or one or more dashes.
     • A name component may not start or end with a separator.
   • Tick boxes if needed:
     • Delete runtime files box - to delete all files from /runs/[pipeline name] folder before committing.
     • Stop pipeline box - to stop the current run after committing.
       In this example, we will change "base-generic-centos7" to "base-generic-centos7-biopython".
       
5. Committing may take some time:
   
   When it is complete COMMITING status on the right side of the screen will change to COMMIT SUCCEEDED. In round brackets the date/time of the latest commit is shown:
   
6. Find a modified Tool in the registry.
   

Note: the new version gets the settings of the default version of the Tool, which the run was launched from. A commit into the version, which the run was launched with, or into that default version, keeps the settings of that version.

Note: if the Tool, which the run is committed into, has no latest version and no default version set, the commit also pushes the new version as the latest one, so that the Tool can be launched without a version specified. If the Tool has a default version set, it is launched with that version, and the commit doesn't create the latest version.

Committing features

Pre/post-commit hooks

In certain use-cases, extra steps shall be executed before/after running the commit command in the container. For example, to avoid warning messages about terminating the previous session (which was committed) of the tool application in a non-graceful manner. Some applications may require extra cleanup to be performed before the termination.

To workaround such issues in the Cloud Pipeline an approach of "pre/post-commit hooks" is implemented. That allows to perform some graceful cleanup/restore before/after performing the commit itself.

Note: Those hooks are valid only for the specific images and therefore shall be contained within those images. Cloud Pipeline itself only performs the calls to the hooks if they exist.

There are two preferences from system-level settings that determine a behavior of the "pre/post-commit hooks" approach:

• commit.pre.command.path: specified a path to a script within a docker image, that will be executed in a currently running container, before docker commit occurs (default value: /root/pre_commit.sh).
  • This option is useful if any operations shall be performed with the running processes (e.g. send a signal), because in the subsequent post operation - only filesystem operations will be available.
    Note: any changes done at this stage will affect the running container.
• commit.post.command.path: specified a path to a script within a docker image, that will be executed in a committed image, after docker commit occurs (default value: /root/post_commit.sh).
  • This hook can be used to perform any filesystem cleanup or other operations, that shall not affect the currently running processes.

Note: User shall have ROLE_ADMIN to read and update system-level settings.

So, when you try to commit some tool, the Cloud Pipeline will check preferences described above and execute scripts with the specified names, if that scripts files exist in a docker image. If a corresponding pre/post script is not found in the docker image - it will not be executed.

Consider an example with RStudio tool, that Cloud Pipeline provides "out of the box". RStudio Docker image contains a post-commit script that cleans the session after the commit:

1. Open the Tools page, launch RStudio.
2. SSH to the launched run via the SSH control in the upper-right corner of the run logs page.
3. Check that post_commit.sh script exists in the Docker image - use the command ls /root/:
   
4. View the contents of that script. It looks like that:
   
   Note: /etc/cp_env.sh is a special Cloud Pipeline script, that sets all environment variables of the current docker container.
   So, as you can see from the code, as the result of that script launch will be a cleanup (removing) of the RStudio active session from the user home directory.
5. Close SSH tab and try to do a commit - click the COMMIT control in the upper-right corner of the run logs page. Specify a new name for the committed tool, e.g. rstudio-test:
   
6. While image is being committed, you can see the follow-up log for the CommitPipelineRun task:
   
   Here:
   • (1) pre-commit script is not found in the docker image - nothing will be executed before commit
   • (2) post-commit script is found at the specified path in the docker image - and it is being executed
   • (3) post-commit script was performed successfully

Container size limits

Docker images can be extremely large. Therefore, Cloud Pipeline supports a mechanism to warn/reject users from creation of such big images.

To configure that mechanism, there is the special system preference - commit.container.size.limits.
This preference has a format:

{
    "soft": <soft_limit_size>,
    "hard": <hard_limit_size>
}

This preference defines "soft" and "hard" limits for a container size in bytes:

• if user tries to perform a commit operation and the container size exceeds "soft" limit - user will get warning notification, but can proceed the commit at their own risk
• if the container size exceeds "hard" limit - commit operation in this case will be unavailable
• if the size of any limit is set as 0 - this means there is no limitation of that type

Example of the preference configuration:

In this example, "soft" limit is set as 1 GB and there is no "hard" limit.

In case, when "soft" limit is set more than 0 and user tries to commit some tool which container exceeds this limit, the following warning will appear:

At the same time, commit operation is available.

In case, when "hard" limit is set more than 0 and user tries to commit some tool which container exceeds this limit, the following error will appear:

Commit operation is unavailable.

Login records

Just before the commit, and before a run is paused, the login records of the container are emptied - /var/log/lastlog, /var/log/faillog and /var/log/tallylog, the ones that exist.

These files keep a record of each user at an offset that grows with the user's UID. For a large UID - e.g. one mapped from Active Directory - they are sparse files of up to hundreds of GB. In the running container such a file takes almost no disk, but a commit stores it in full, and every pull of the image would write it out to the node's disk. So lastlog and similar commands show no earlier logins in a container of a committed image.

Note: an image, which was committed with such a file before these files were emptied, keeps it in one of its layers. A new commit from a run of that image only hides the file, and every pull still writes it out. To get a small image again, commit from a run of the image it was committed from - e.g. the original tool - not from a run of the large image.

Note: the container size limits are checked before the commit, so they count these files at their full size. For a user with a large UID, a container may exceed a limit, while the committed image is small.

Edit a Tool settings

Settings in this tab are applied to all Tool versions (i.e. these settings will be a default for all Tool version). They are saved as the settings of the default version of the Tool.

1. Navigate to the Settings tab.
   
2. Specify Endpoints for a Tool by click "Add endpoint" button:
   
   In the example below: the port is 8788 and the endpoint name is rstudio-s3fs:
   
   Let's look at the endpoint closer:
   • "nginx" - type of the endpoints (only nginx is currently supported)
   • "port":XXXX - an application will be deployed on this port on the pipeline node.
   • You can specify additional nginx configuration for that endpoint in the text field bottom in JSON format.
     Note: optional path parameter may be required in case your application starts on <host:port:>/path.
     Note: optional additional parameter may be required in case you need to specify nginx location settings. See more information here.
   • "name" - this value will be visible as a hyperlink in the UI. It is especially convenient when user sets more than one endpoint configuration for an interactive tool (learn more about interactive services - 15. Interactive services).
     In the example below, we name one endpoint as "rstudio-s3fs" and another one as "shiny".
     
     This is how everything will look in the Run log window:
     
3. Click the + New Label button and add a label to the Tool (e.g. "Awesome Tool").
   
4. Specify "Execution defaults":
   • Instance type (e.g. "m4.large")
   • Price type (e.g. "Spot")
   • Capacity retries, how many times to try to start a compute node (optional; shown if the ui.launch.allow.nodeup.count preference is true)
   • Disk size in Gb (e.g. "20")
   • select available storages
   • configure cluster, if it's necessary for your Tool
   • write the Default command for the Tool execution (e.g. echo "Hello world!")
     
5. If it's necessary for your Tool - add system or custom parameters. For more details see 6.1. Create and configure pipeline.
6. Click Save button to save these settings.
   

Delete a Tool

To detach a Tool from the Cloud Pipeline:

1. Open the Tool you wish to detach
2. Hover over the gear icon in the right upper corner and click the "Delete tool" item in the appeared list:
   
3. Click the Delete button to confirm:
   

Note: if you try to delete a tool/version, but there is/are active job(s) using it - the corresponding warning notification will be shown, e.g.: