// File Transfer from NCCS to User Systems

You must streamline your connections for direct SSH first before you can transfer files from the NCCS.

For file transfers from ADAPT, please use the Transfer Nodes.

Command-Line Users

In standard bastion configurations, you can run the following from your terminal: $ scp userid@discover.nccs.nasa.gov:/target/file .

The " . " will place the file in your current working directory; you may specify a /target/location instead.

If rsync is available on your client, it might be a better alternative to scp because you can resume a transfer if there is a network issue. For example: $ rsync -av --progress userid@discover.nccs.nasa.gov:"/target/file" .

For more assistance with either of these commands, run "man scp" or "man rsync" to see their manual pages.

In standard bastion configurations, you can also run sftp from your terminal: $ sftp userid@discover.nccs.nasa.gov

Within the sftp prompt, you will be able to see your remote location and download files to your local current working directory by running the following commands: $ pwd
$ ls -alh
$ get filename

For more assistance with sftp, run "help" in its prompt to see a list of commands.

MobaXterm

MobaXterm is recommended for Windows users because its built in scp/rsync/sftp is similar to Linux/MacOS configurations.

See the instructions for how Windows users can streamline their connections by downloading and configuring MobaXterm.

Once downloaded, you can start a local terminal within the application window.

You may test file transfer by running either: $ scp userid@discover.nccs.nasa.gov:/target/file .
$ rsync -av --progress userid@discover.nccs.nasa.gov:"/target/file" .
$ sftp userid@discover.nccs.nasa.gov

It will ask you to authenticate and then it will display the transfer status. For more assistance with either of these commands run "man scp" or "man rsync" to see their manual pages.

Downloading via HTTP

Hypertext Transfer Protocol (HTTP) is the protocol that drives most web site internet traffic today. A variant of the protocol, called “HTTPS”, “S” for “Secure”, has been chosen to replace FTP.

HTTPS encrypts all transactions between client and server. This makes it extremely difficult for third parties to intercept what is being transferred.

HTTPS downloads will not require an active login. HTTPS downloads can be found at https://portal.nccs.nasa.gov/datashare/ .

Downloading via wget

wget is an open source utility that can download entire directories of files with just one command. The only path that is required is the root directory. wget will automatically traverse the directory and download any files it locates.

wget is free and available for Linux, macOS, and Windows.

Installing wget

Linux

  1. Launch a command-line terminal
  2. $ yum install wget -y

Mac

  1. Install https://brew.sh/ (admin privileges required)
  2. Launch Terminal
  3. $ brew install wget

Windows

  1. Download the latest 32-bit or 64-bit binary (.exe) for your system https://eternallybored.org/misc/wget/
  2. Move it to C:\Windows\System32 (admin privileges will be required)
  3. Launch cmd
  4. In the Windows Command Prompt, type > wget -h to verify the binary is being executed successfully
  5. If any errors appear, the wget.exe binary may be not be located in correct directory or you may need to switch from 32-bit 64-bit

Command-Line/Terminal Usage:

$ wget -r -np -nd -R "index.html*" “https://portal.nccs.nasa.gov/datashare/PATH_TO_DATA_DIRECTORY” -P TARGET_DIRECTORY_ON_YOUR_FILE_SYSTEM

Be sure to replace the following:

  • PATH_TO_DATA_DIRECTORY: location of source directory on datashare site

  • TARGET_DIRECTORY_ON_YOUR_FILE_SYSTEM: Where you would like to download the files. Examples include /Users/jdoe/data for macOS and Linux or C:\Users\jdoe\data for Windows

Using wget to list directory files

This example uses the wget command to make a request for the file listings in a specific directory on our web server at the URL https://portal.nccs.nasa.gov/datashare/PATH_TO_MY_FILE

$ wget -qO- https://portal.nccs.nasa.gov/datashare/PATH_TO_DATA_DIRECTORY/ | sed -e 's/]*>//g;s/^ //g'

Using curl to download

This example uses the curl command to make a request for the file on our web server.

$ curl https://portal.nccs.nasa.gov/datashare/PATH_TO_MY_FILE -o /Users/jdoe/Desktop/file.ext

Be sure to replace the following:

  • PATH_TO_DATA_DIRECTORY: location of source directory on datashare site

  • TARGET_DIRECTORY_ON_YOUR_FILE_SYSTEM: Where you would like to download the files. Examples include /Users/jdoe/data for macOS and Linux or C:\Users\jdoe\data for Windows

  • The “-o” flag may not be applicable to Windows systems, the option species a location to save the file on the local system.