8.7. Create shared file system

User shall have ROLE_ADMIN to mount FS to the Cloud Pipeline. For more information see 13. Permissions.

A shared file system is a data storage based on network file system. It has several advantages over regular data storages and local file system:

  • While regular data storage is a great option for a long-term storage, it cannot be used as a shared file system for high-performance computing jobs as it does not support network-like interface.
  • A local disk cannot be shared across several nodes.
  • A user needs to specify local disk size when scheduling a run. If a user specifies a size that is not enough to finish a job - it will fail. Cloud-based shared file system could be used to workaround this issue.

Create FS mount

  1. Navigate to a desired folder and click + Create → Storages → Create new FS mount.
    CP_CreateSharedFileSystem
    Note: For FS mounts - "Add existing" option is not available.
    Note: For the correct FS mount creation, at least one mount point shall be registered in the System Preferences for any Cloud Region. If no - the corresponding button of the FS mount creation becomes invisible:
    CP_CreateSharedFileSystem
  2. Specify Storage path and other optional parameters.
    CP_CreateSharedFileSystem
    Note: Storage path parameter contains FS mount path and the name of the storage to be created.
    Note: FS storages are just subdirectories of the mounted FS. One FS mount can have multiple FS storages.
  3. When deleted from GUI, FS storage is unmounted from the Cloud Pipeline.

FS storage features

  1. For FS storages GUI doesn't display the following features typical for regular data storages:
    • STS
    • LTS
    • Versioning and Backup duration.
      CP_CreateSharedFileSystem
  2. When a user selects Input/Common/Output path parameter for a pipeline run - it is impossible to set FS storage:
    • FS storages aren't displayed in the "Browse..." dialog for Input/Common/Output path parameters;
    • Value of Input/Common/Output path parameters is validated so that user is not able to specify a path to FS storage manually.
      CP_CreateSharedFileSystem
  3. The content of files stored in FS data storage could be previewed as well as in regular data storages:
    CP_CreateSharedFileSystem
  4. Since FS isn't an object data storage, it isn't possible to add metadata tags to files in the FS storage.
  5. Use FS storage between cluster nodes. If pipeline Tools contain FS client, FS storage(s) will be mounted automatically.