// File transfer from Discover to Dirac

Recommended methods to transfer data between Discover and Dirac

  1. In a datamove job use cp to copy from $ARCHIVE
  2. On a login node use cp to copy from $ARCHIVE
  3. On a login node use bbscp
  4. In a datamove job use bbscp through dirac
  5. Using SCP and RSYNC to copy to/from Dirac

File transfer between Discover and Dirac using the datamove partition

One way to prepare data ready for a large compute run on Discover is to first submit a batch job to the datamove partition in order to copy large data files from the archive or an external location to the Discover file systems.

Jobs submitted to the datamove partition run on a “gateway” node. The gateway nodes have external network interfaces, which allow access to the archive (Dirac), and other external systems. They also have access to Discover’s local GPFS cluster-wide file systems. You can submit a job to the datamove partition to migrate data into the Discover environment, making it visible to Discover’s compute nodes. You can then submit a compute job to analyze the data you moved. The results of your compute job can then be transferred back out of Discover via an additional datamove job.

You can submit three jobs all at once (a datamove job to move data into the system, a compute job, and a datamove job to move data back off the system). You can then use Slurm's dependency functions to ensure job execution in the correct order.

Copying Large Files from $ARCHIVE to $NOBACKUP using SLURM

First, ensure that the file you are moving is online and not on tape.
IMPORTANT: Retrieve your files from tape before attempting to use them! USE dmget

Once the state of the files in the archives are dual, they are ready to be copied to your $NOBACKUP where you will be able to use them.
IMPORTANT: Do not untar anything in the archive area!
Untarring files in /archive will create many small files in the archive area, which will then be written to many tapes.

The best way to copy from your $ARCHIVE into your $NOBACKUP is by using a datamove node, either with a batch job or interactively. The following example, assumes $ARCHIVE and $NOBACKUP into some hypothetical pathnames. Here is the Slurm script for the batch job for copying a big file from archives to your nobackup area using the datamove queue.

Batch job datamove transfer example: #!/bin/bash
#SBATCH -J mycopyjob
#SBTACH -t 00:01:00
#SBATCH -p datamove
#SBATCH --account=xxxx
source /usr/share/modules/init/bash
module purge
cp /archive/u/myplace/bigfile /discover/nobackup/myplace/

As a rule of thumb,it is recommended to set the wall time limit to be 2 seconds per gigabyte, and multiply that by 2 for good measure.

Interactive datamove transfer example:

If you are logged onto a discover node, start an interactive shell on a datamove node, and then issue the copy command. The -t flag will set the time limit to 1 hour.

$ salloc -p datamove -t 01:00:00
salloc: Pending job allocation 8164701
salloc: job 8164701 queued and waiting for resources
salloc: Granted job allocation 8164701
srun.slurm: cluster configuration lacks support for cpu binding
$ cp /archive/u/myplace/bigfile /discover/nobackup/myplace/

Moving Large Files to $ARCHIVE using cp on Discover

It is better to transfer few large files than many small ones between the tape cache (/archive) and your nobackup area.

When copying large files from discover to dirac, we recommend using "cp" over "mv." As mv deletes files after copying, there is a risk on losing your files if they system hangs due to storage shortage or any other issues. The following example script can be used to automate the transfer process. #!/bin/bash
# check the status of a command
if cp a b ; then
echo "Copy succeeded"
rm a
echo "Copy failed"

The same script in csh looks like this: #!/bin/csh
# check the status of a command
if { cp a b } then
echo "Copy succeeded"
rm a
echo "Copy failed"


Bbscp is a wrapper tool to the bbftp utility; bbscp generates the appropriate command for the users without requiring them to define flags and/or other options for bbftp. Bbftp is a remote file transfer utility that does NOT encrypt files for transfer. In return, it increases the transfer speed for quicker transfers between the systems. Bbscp and bbftp can only be used for file transfer between Dirac, the front-end to the mass storage system, and the Discover cluster. Bbscp should also be utilized through the Discover cluster. Follow the steps below to appropriately use the bbscp/bbftp utility:

  1. Log onto the Discover cluster, and set up passwordless ssh to Dirac from the Discover cluster.


  2. Once passwordless ssh has been set up, log onto both dirac1 and dirac2 servers (from the Discover cluster) and accept the RSA fingerprint: $ ssh dirac1
    $ ssh dirac2

  3. Bbscp runs the -r flag by default, so users can transfer single files or directories without specifying any flags. To transfer a file from your current working directory on the Discover cluster to your home directory on Dirac, run:$ bbscp test_file dirac:~/
    To transfer an entire directory from your current working directory on the Discover cluster to your home directory on Dirac, run:$ bbscp test_dir/ dirac:~/

  4. To learn more about the bbscp or bbftp, use the -h flag: $ bbscp -h
    $ bbftp -h


A simple file transfer utility, like scp or rsync, can also be utilized for file transfers. The following examples show how to transfer a file from the Discover cluster to your Dirac home directory: $ scp /my/file.txt dirac:~/

If rsync is available on your client, it might be a better alternative to scp because say if there is a network issue, you can still finish a given transfer (and it should work with the standard bastion configuration for ssh). For example: $ rsync -av --progress --partial /my/file.txt dirac:~/

If the file transfer fails and --partial is set, the target file will be named as the actual file name (i.e. FileYouAreSending.txt), even though the file isn't complete. You can later complete the transfer by running rsync again with either --append or --append-verify: $ rsync -av --progress --append /my/file.txt dirac:~/

For more assistance with either of these commands run "man scp" or "man rsync" to see their manual pages.