Skip to content

Transfer Data to/from ZIH Systems via GridFTP

GridFTP endpoints are accessible via gridftp[1,2].hpc.tu-dresden.de. Using GridFTP rather than rsync or sftp commands may improve performance and allows to initiate transfers between two remote systems from a separate device. This way, the full bandwidth of the data link between the sites is available, and a client doesn't need to supervise the transfer. Additionally, GridFTP may prove useful when other means of data transfer are inaccessible due to a site's firewall configuration.

Setup

In order to make use of the service, install the client tooling:

marie@local$ sudo apt install globus-gass-copy-progs

marie@local$ sudo dnf install globus-gass-copy-progs

The examples shown below assume the sshftp protocol and accordingly configured SSH access to ZIH systems (see also: Connecting via Terminal (Linux, Mac, Windows)). This requires an SSH-keypair (here: ~/.ssh/id_ed25519, ~/.ssh/id_ed25519.pub), the public key of which is to be appended to the target system's ~/.ssh/authorized_keys file. Additionally, add entries to ~/.ssh/config as such:

Host gridftp1
    User marie
    Hostname gridftp1.hpc.tu-dresden.de
    Identityfile ~/.ssh/id_ed25519

Host gridftp2
    User marie
    Hostname gridftp2.hpc.tu-dresden.de
    Identityfile ~/.ssh/id_ed25519

Since the service supports other protocols as well, refer to the remote site's documentation for how to interact with GridFTP if data is to be transferred between sites.

Usage

For a full set of available commands, see man globus-url-copy or the documentation available at gridcf.org. A first simple use case of listing a remote directory lends itself to confirming connectivity:

marie@local$ globus-url-copy -list sshftp://marie@gridftp1/home/marie/
sshftp://marie@gridftp/home/marie/
    .cake
    bedroom/
    clutter.tar
    kitchen/

Mind the trailing /, it is obligatory for the command's functionality. Actual file transfers may happen in one of two ways, between the client and a remote machine, or directly between two remote machines involving the client only for orchestration. In the first example, a local file is copied onto the remote system.

marie@local$ globus-url-copy file:///etc/hostname \
    sshftp://marie@gridftp1/home/marie/local_hostname

Confirm the transfer was successful:

marie@login$ cat /home/marie/local_hostname
local

In the second example, a file is copied from remote one remote system to another. For demonstration purposes, this can also be done between the two ZIH GridFTP gateways. Using this methodology, internal tests confirmed transfer rates far exceeding the connectivity of the orchestrating client.

marie@local$ globus-url-copy \
    sshftp://marie@gridftp1/data/horse/ws/marie-number_crunch/10Gifile \
    sshftp://marie@gridftp2/home/marie/
Connecting to sshftp://marie@gridftp1/data/horse/ws/marie-number_crunch/10Gifile ...
Connecting to sshftp://marie@gridftp2/home/marie/10Gifile ...

No other pages link to this page.