3. Overview

User Journey in a nutshell

Cloud Pipeline (CP) is a cloud-based web application which allows users to solve a wide range of analytical tasks and includes:

  • Data processing: you can create data processing pipelines and run them in the cloud in an automated way.
  • Data storing: create your data storage, download or upload data from it or edit text files within CP UI. File version control is supported.
  • Tool management: you can create and deploy your own calculation environment using Docker's container concept.

This Manual is mostly around data processing lifecycle which, in a nutshell, can be described in these several steps:

  1. To run a user's calculation script it shall be registered in CP as a pipeline. The script could be created in CP environment or uploaded from the local machine. See more details 6. Manage Pipeline.
    Note: If you need to run a pipeline in different environments simultaneously or set a specific type of data, you can use detach configuration object. See more 7. Manage Detached configuration.
  2. To store pipeline's inputs and outputs data files the Data Storage shall be determined in CP. Learn more 8. Manage Data Storage.
  3. Almost every pipeline requires a specific package of software to run it, which is defined in a docker image. So when a user starts a pipeline, CP starts a new cloud instance (nodes) and runs a docker image at it. See more details 9. Manage Cluster nodes and 10. Manage Tools.
  4. When the environment is set, pipeline starts execution. A user in CP can change and save configurations of the run. Learn more 6.2. Launch a pipeline.
  5. A user can monitor the status of active and completed runs and usage of active instances in CP. Learn more 11. Manage Runs and 9. Manage Cluster nodes.

Note: CP can run a docker image at instance without any pipeline at all if needed. There just will be an instance with some installed and running software. A user can SSH to it or use it in the interactive mode.

Also, CP supports CLI, which duplicates some of GUI features and has extra features unavailable via GUI, such as automation of interaction with CP during the pipeline script running, or uploading considerable amount of data (more than 5 Gb), etc. The basics of CLI you can learn here.

CP_Overview

GUI menu tab bar

There are several menu tabs at the left edge of CP window.
CP_Overview

Home

Home space is opened by default when Cloud Pipeline is loaded. It provides "personal" and often-used information for a user.
By default, this dashboard shows 3 widgets:

  • Active Runs (see picture below, 1) displays a list of user's active runs. See more details 11. Manage Runs.
    Note: press Explore all active runs to see a list of all runs.
  • Tools (see picture below, 2) shows a list of Tools in a user's personal repository and Tools available to your group. Learn more 10. Manage Tools and 13. Permissions.
    Note: group-level Tools will be shown on the top of the Tools list.
    Note: user can search for a particular Tool by using the Search tools text box. System will suggest Tools based on a user's query.
  • Data (see picture below, 3) displays available Data Storages for a user - user shall have OWNER rights or READ/WRITE access to a Data Storage. See more 8. Manage Data Storage and 13. Permissions.
    Note: personal Data storages (i.e. a user is an OWNER of this Storage) will be shown on top, WRITE - second priority, READ - third priority.

CP_Overview

Each widget has the Help icon CP_Overview. Hover this icon to get a brief description of a widget. Learn more about Home tab 18. Home page.

Library

Library space supports a hierarchical view of its content:

The tab consists of two panels:

  • "Hierarchy" view (see the picture below, 2) displays a hierarchical-structured list of pipelines, folders, storages and machine configurations, etc.
    Use the "Search" field to find a CP object by a name. "Collapse/Expand" button (see the picture below, 1) at the bottom-left corner of the screen: use it to collapse or expand "Hierarchy" view.
  • "Details" view (see the picture below, 3) shows details of a selected item in hierarchy panel. Depends on a selected type of object it has a very different view. You can learn about each on respective pages of the manual.

CP_Overview

Note: Each time "Hierarchy" view loads, the first Folder in the hierarchy that has more than 1 child object (Folder, Pipeline, Data Storage, etc) is automatically selected. Contents of that Folder are automatically expanded.

Cluster Nodes

This space provides a list of working nodes. You can get information on their usage and terminate them in this tab. See more details 9. Manage Cluster nodes.

Tools

Tools space displays available Docker images and their tools, organized in tool groups and docker registries. You can edit information about them and run nodes with any Docker image in this tab. See more details 10. Manage Tools.

Runs

This space helps you monitor the state of your run instances. You can get parameters and logs of a specific run, stop a run, rerun a completed run. Learn more 11. Manage Runs.

Settings

This tab opens a Settings window which allows:

  • generate a CLI installation and configuration commands to set CLI for CP,
  • manage system events notifications,
  • manage roles and permissions.

See more details 12. Manage Settings.

It's a search field which allows searching specific runs by its "Run ID" or "Parameters".

Logout

This is a Logout button which logs you out.
Note: if automatic logging-in is configured, you will be re-logged at once.
Note: if any changes occur in the CP application during an authorized session, the changes are applied after re-login.